Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline

Dingyi Yang, Chunru Zhan, Ziheng Wang, Biao Wang, Tiezheng Ge, Bo Zheng, Qin Jin

Open Source

• We explore the task of Synchronized Video Storytelling, generating text to narrate the ongoing video scenes and serve as useful voiceovers.
• We release a new benchmark in the advertising domain called E-SyncVidStory.
• We propose an effective framework named VideoNarrator, which simultaneously supports storyline generation and controllable video story generation.

What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation

Dingyi Yang, Qin Jin

Open Source

• We provide an in-depth discussion of the challenges in evaluating various story generation tasks.
• We analyze standardized criteria, addressing issues of inconsistent definitions and vague expressions.
• We systematically review traditional, LLM-based, and collaborative evaluation methods, highlighting their strengths and limitations in the context of story evaluation.
• We suggest potential future research directions, extending from story evaluation to general evaluations.

Visual captioning at will: Describing images and videos guided by a few stylized sentences

Dingyi Yang, Hongyu Chen, Xinglin Hou, Tiezheng Ge, Yuning Jiang, Qin Jin

• We explore Few-Shot Stylized Visual Captioning, which aims to generate captions in any desired style, using only a few examples as guidance during inference, without requiring further training.
• We propose an effective framework that handles multiple styles with a single model. It extracts the style representation from stylized samples, and aligns visual information to generate stylized captions.

Attractive storyteller: Stylized visual storytelling with unpaired text

Dingyi Yang, Qin Jin

• We explore Stylized Visual Storytelling, generating stories with complicated styles for image sequences.
• We propose StyleVSG, which applies style-specific parameters to control text style and incorporates a memory module to maintain context coherence.
• To balance style accuracy and visual relevance, we employ multi-task training using our pseudo {images-stylized story} pairs and {images-factual story} pairs.