Demo #1: "A traditional Japanese townscape featuring a vivid red arched bridge, torii gates, wooden boats, and classic tiled-roof buildings lining the waterfront."
We strongly recommend watching the video to quickly understand our method and see our demo results.
Demo #1: "A traditional Japanese townscape featuring a vivid red arched bridge, torii gates, wooden boats, and classic tiled-roof buildings lining the waterfront."
Demo #2: "An industrial-style warehouse, featuring stacked crates, oil drums, industrial equipment, gas cylinders, a staircase, and a yellow-black construction vehicle, all arranged in a clean and organized manner."
Demo #3: "An ancient temple with a reddish-brown stone structure and dome sits in a lush forested valley, surrounded by greenery and misty mountains, exuding tranquility."
Demo #4: "A busy shipping yard filled with colorful stacked containers, industrial buildings, forklifts, and signs of active cargo operations under a partly cloudy sky."
Demo #5: "A charming coastal village scene featuring brightly painted wooden houses, a lighthouse, and winding stone paths under a partly cloudy sky."
Current interactive long video generation methods lack scene-consistent memory, often resulting in completely different scenes when simply looking left and right within a short time span. This issue stems from the limited utilization of historical context. Below, we introduce our approach to leveraging unlimited historical context to ensure scene-consistent memory in newly generated video frames.
(a) We propose Context-as-Memory, where all historical context frames serve as memory conditions in the generation of predicted frames, with Memory Retrieval extracting relevant information from all context frames. (b) Our proposed Memory Retrieval method is a search algorithm based on camera trajectories. It selects relevant frames by evaluating the overlap between camera views of different frames.
We collected some open-domain images from the internet and attempted to generate long videos using our method to test whether memory capabilities can be achieved in scenes unseen during training. The answer is yes. However, it's worth noting that open-domain capabilities largely depend on the pre-trained model's capabilities. We currently use a relatively small 1B parameter model, which has limited generation capabilities for complex trajectories. A future direction would be to validate our method on larger and more powerful pre-trained models.
Result #1: "Japanese landscape, Mount Fuji, blue sky, red and white pagoda, cherry blossom trees, cultural harmony, natural beauty."
Result #2: "Black Myth Wukong, snow-covered stone pathway, ancient temple, wintry forest, towering trees, solemn statues, sacred ambiance, icicles, frost, mist, solemn tranquility, timeless mystery."
Result #3: "Black Myth Wukong, mist-laden forest, weathered wooden structure, overgrown vegetation, moss-covered stone steps, ornate tiled roof, haunting beauty, serene atmosphere, distant cliffs, abandoned mountain sanctuary."
Result #4: "Black Myth Wukong, a barren landscape of red and grey features a desolate shrine with weathered East Asian architecture, a lone stone statue, and a twisted lifeless tree under churning thunderclouds, casting dynamic shadows and creating a foreboding atmosphere."
Result #5: "A sunlit alpine valley shows logs by a rustic cabin, surrounded by meadows, forests, and towering mountains. The breeze gently stirs the grass, enhancing the scene's tranquil isolation."
Result #6: "The Legend of Zelda, vibrant green fields, wildflowers, sprawling landscape, winding rivers, distant ruins, towering mountains, clear blue sky, colossal mechanical structure, epic scale, boundless exploration."
Result #7: "Genshin Impact, a festival pier glows with lanterns, featuring a poised girl in traditional attire amid dragon-shaped lanterns and swaying pavilions, all under a magical twilight sky."
Result #8: "Genshin Impact, a gentle pan reveals a vibrant countryside village in golden morning light, with a windmill turning above timber-framed houses and drifting dandelion seeds. Castle-like towers in the distance add to the storybook-like tranquility."
Result #9: "A smooth tracking shot sweeps through a sunlit high-rise office, showcasing ergonomic workstations and a stunning cityscape view. The calm, professional space evokes quiet productivity and modern design."
Result #10: "A bright, open-plan office features minimalist desks, ergonomic chairs, and calming greenery, all under natural and LED lighting, emphasizing modern productivity."
Result #11: "A serene bedroom features a neatly made bed with cream bedding, wooden nightstands, and minimalist decor. Soft natural light and framed artwork enhance the calm, refined atmosphere."
Result #12: "A slow glide through a formal garden reveals a circular stone pedestal, symmetrical hedges, and blooming rose bushes. Ornamental urns and stone paths evoke the elegance of a historical estate."
Result #13: "An aerial view glides over a winding path in an alpine meadow, with golden sunlight and mist creating a serene morning scene. Rugged mountain peaks in warm light contrast with the cool sky, evoking natural grandeur."
Result #14: "A slow zoom captures a tranquil alpine forest and a mountain range glowing in the sunset. A still body of water reflects snow-capped peaks and evergreens, creating a serene, awe-inspiring twilight scene."
Result #15: "A sunlit view of a frozen alpine basin surrounded by snow-covered peaks. The camera pans, highlighting the contrast between sunlit ridges and shaded slopes. Pine trees and weathered rocks add to the quiet, majestic winter landscape."
Result #16: "A tranquil lakeside scene reflects a snow-capped mountain, with golden light revealing shoreline buildings amidst vibrant autumn foliage. Fiery red maple leaves frame the view, enhancing the serene harmony of nature and human settlement."
Result #17: "A sunlit mountain path winds through a lush valley with wildflowers and a rustic cabin. Snow-capped peaks rise above, set against a crisp blue sky, creating a serene, untouched atmosphere."
Result #18: "A sunny alpine path curves along a hillside, bordered by evergreens and a stone wall. A fence overlooks a valley with rooftops. Limestone cliffs and snow-draped peaks complete the peaceful scene."
Result #19: "In an arid desert under a burnt-orange sky, a dusty six-wheeled rover stands amid scattered components, evoking a Mars expedition base."