My Test Drive of Runway
Runway has several products on offer, including creating video from both images and prompts. They offer an AI video editor and audio generator. They also offer standard image generation products and 3D design.
Gen-2 is their proprietary AI video generation model. They allow for text input up to 320 characters. You can also include a reference image with your text prompt or use the image alone. When using the image + text to generate video, Gen-2 will match the image directly and use the text to help create the video.
Gen-2 costs 5 cents per second of video created. It is also available on iOS.
You can include prompt modifiers to apply a style to your video. Runway has suggested that masterpiece, classic and cinematic work well and produce good results. They have suggested using additional words to provide for action and movement, think cinematic action or flying. You can also use camera specific words like angles (full shot or close up), lens type (wide angle) movement (zoom or slow pan). Try to stay away from any video production jargon and use language that a toddler would understand.
A basic prompt structure recommended is [base prompt] in the style of [style], [aesthetic].
Once you enter a prompt, you are given a still-frame preview of what the video will look like. If you don’t like it, you can modify your prompt.
Prompt Test 1
I was unable to use the preview function of my prompt because the service was busy when I was test driving it. I used the following prompt to generate a 4 second video blind:
Mountains with a waterfall running through them and the sun shining in the background. The mountains are covered in brightly colored flowers and snow at their peaks in a cinematic style, ultra realistic
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 2
The current AI video generators on the market seem to only excel at adding motion to still frames. The dragon and unicorn playing softball prompt I’ve been using seems to be beyond the scope of their capability, simply because the model cannot generate the action of playing softball in a still frame (my theory anyway). With that in mind, I crafted a new prompt to create a different still frame for motion. I used the following prompt for my second test:
a cardinal and a bluejay perched in a majestic oak tree with deep green leaves. the oak tree is on the edge of a lake with a sun setting in the horizon beyond the lake. zoom on the birds and use a realistic cinematic style
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 3
The result made the birds change and I suspect that might be from the direction to zoom on the birds. I’m going to try again without this and instead have the full image zoom out to see if the birds remain unchanged.
a cardinal and a bluejay perched in a majestic oak tree with deep green leaves. the oak tree is on the edge of a lake with a sun setting in the horizon beyond the lake. Focus on the birds while zooming out
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 4
The birds continue to change in the motion of the video. I’m going to remove all motion control to see if this resolves the birds morphing into weirdness.
a cardinal and a bluejay perched in a majestic oak tree with deep green leaves. the oak tree is on the edge of a lake with a sun setting in the horizon beyond the lake.
For this prompt, I accidentally used it twice. The outputs were very different and I suspect that is from using a different random seed with which to build the image.
Results:
Prompt Test 5
This last one was really nice. The birds didn’t change and although the bluejay looked just a bit weird on its head, the birds looked nice.
My girlfriend has a tattoo of these two birds in a cherry blossom tree on her arm. That served as my inspiration and now I will change the tree to see her tattoo come to life. I used the following prompt:
a cardinal and a bluejay perched in a cherry tree full of blossoms. the cherry tree is on the edge of a lake with a sun setting in the horizon beyond the lake.
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 6
This one is missing the bluejay altogether and the bird still changes. I updated the prompt to add for ultra-realistic and cinematic motion to see if that corrects the problem.
a cardinal and a bluejay perched in a cherry tree full of blossoms. the cherry tree is on the edge of a lake with a sun setting over a mountain on the horizon beyond the lake using an ultra realistic, cinematic style
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 7
This one was better but the birds still morph. I adjusted the prompt again to see if I can prevent this.
a cardinal and a bluejay perched in a cherry tree full of blossoms. the cherry tree is on the edge of a lake with a sun setting over a mountain on the horizon beyond the lake using an ultra realistic, cinematic style. do not change the birds, zoom the camera out
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 8
The birds continue to morph. I tweaked the prompt again to try to prevent this by reversing the order of the sentence so the birds are the active subject in the sentence. I hit the character limit with this one:
a cardinal and a bluejay perched in a cherry tree full of blossoms. the cherry tree is on the edge of a lake with a sun setting over a mountain on the horizon beyond the lake. use an ultra realistic, cinematic style. prevent movement in the bluejay and the cardinal, the only movement the camera zooming out
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 9
Somehow I now have two bluejays and no cardinal. The model really struggles to keep the birds from moving or I just can’t find the right language to help it understand to keep the birds the same throughout the video. I will try to put that explicitly in the prompt.
a cherry tree full of blossoms with a cardinal and a bluejay perched in a branch looking at a lake with a sun setting over a mountain on the horizon beyond the lake. use an ultra realistic, cinematic style. Do not change the birds during the entire video.
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 10
Changing the order of the cherry tree and birds in the prompt took me further from the results I wanted. I will go back to the prompt before and add in the phrase to keep the birds from changing instead of trying to control the motion.
a cardinal and a bluejay perched in a cherry tree full of blossoms. the cherry tree is on the edge of a lake with a sun setting over a mountain on the horizon beyond the lake. use an ultra realistic, cinematic style. Do no change the cardinal or bluejay in the entire scene
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
Prompt Test 11
This one was better. The birds stayed mostly the same, however there were now two bluejays in the scene. I’m not sure how this happened. I’m going to start a new test with a slightly modified prompt to see if I get close.
one cardinal and one bluejay perched in a cherry tree full of blossoms with the wind blowing them around. the cherry tree is on the edge of a lake with a sun setting over a mountain on the horizon beyond the lake. use an ultra realistic, cinematic style. Do no change the cardinal or bluejay in the entire scene
I had to wait in the queue for about a minute and it took another minute or so to generate the video. The video cost me 20 credits.
Results:
This is the best I’ve had yet. There was only slight morphing on the bluejay’s colors and in the face of the cardinal, but overall, it looks really great!
After all that experimenting, I have 57 seconds of video left which I’d like to save for another test later when I might have a new use case to try.
Runway is a fun tool to experiment and I’ve benefited from testing a few tools prior so I’m getting better at understanding how these tools work and how best to instruct them in my prompts. Again, it might not be useful to any of my business projects, but it is an amazing tool to create art as long as you have the credits. Free accounts start with 105 seconds of video to try it and I encourage you to do so!
