I decided to take another step down the rabbit hole of AI generated content. I've been playing around with AI images for a while, and with AI videos more recently, and wanted to try the best tools out there to see what I could make.
This was my very first attempt at creating a "mini film" (or "Splintershort"). Note that I had zero experience with video editing prior to this, so please don't be too harsh when reviewing the final result... :)
Starting off, here's the end result, which took about 3-4 hours to make from start to finish. Further below, I explain the process I went through and what I'm going to be trying next.
Tools
For this project I did some research on the various tools out there (there are so many!) and settled on just going with the "industry-leading" ones, specifically I used:
- ChatGPT for script support and storyboarding (https://chatgpt.com/)
- Midjourney to create consistent high fidelity images (https://www.midjourney.com/)
- Runway to animate the scenes (https://app.runwayml.com/)
- Freesound for free sound effects (https://freesound.org/)
- Pixabay for the music track (https://pixabay.com/)
- Clipchamp to stitch the video together
Base Character & Storyboarding
The first step was to create a story. I picked one of my favorite characters in #Splinterlands, Avina of the Wolf, and started off by getting ChatGPT's help to create a short storyboard and start visualizing the various scenes. This can be easily achieved by giving ChatGPT a premise and clear guidelines (e.g. "12 short scenes for a mini film"), and then asking it to create consistent visual representations of each scene.
After a bit of back and forth, I had the basis for the scenes I wanted to use and it was time to refine the images.
High fidelity images
After doing some research, I concluded that Midjourney would be the best tool for the job as it creates beautiful imagery and has tooling that enables for image consistency as well as editing.
In Midjourney, you can provide images as part of the prompt and specify whether those are character or scene references. By taking the original art from Splinterlands for Avina as the character reference, using the ChatGPT image as scene reference, and providing some more details, I was able to start refining the image of Avina to be closer to what I wanted.
After a few iterations and experiments, I landed on an image baseline and style that I liked.
This took a lot of experimentation and I got some incredible images in the process.
In some of the prompts above you may see some parameters applied. The most interesting ones in my opinion are "--iw", which specifies the weighting the the scene image reference, and "--cw" which specifies the weighting of the character image reference. By using these correctly, you're able to maintain good atmospheric and character consistency required if producing a storyboard. However if you pay any attention at all you'll notice the consistency isn't perfect and both Avina and the Orc are changing their clothing a bit throughout the scene 🤣
With this part done, I just needed to create key images for the remaining scenes.
Image to Video
The next (and scariest) step was to take the storyboard into Runway to start animating the scenes. This is something I was totally unfamiliar with, and frankly I feel like I just scratched the surface and have a LOOOONG way to go before I get any good at this. It's not trivial at all, but it's certainly doable with patience and effort.
On Runway, I used the Generative Video option with their Gen-3 Alpha Turbo model and inserted the scenes one by one.
One very important learning was when doing image-to-video in Runway you do not need to re-describe the image itself. You should only describe the camera style (what shot you want, and how it moves) as well as what motion you expect from the characters in the shot.
I took me ~100 exports to get the videos I wanted. Sometimes Runway would hallucinate and do really weird stuff with the videos (like it kept adding a random wolf into one). Others it just wouldn't look right (getting those orcs to walk properly took 10 tries, and the final one still wasn't great). Other times it just didn't seem to quite do what I wanted so I needed to finetune my instructions and keep experimenting.
Stitching things together
Finally once I had reasonable videos for each scene in my mini-film, it was time to stitch it together. For this part I picked a very simple tool rather than a state-of-the-art one, partly because I was too tired to figure out Davinci and partly because I didn't want to spend a ton of money on Adobe Premiere 😅. Clipchamp did the job as it provides a very simple, no-frills interface for putting together video, image, sound, music, and text assets together.
As I said before, the entire process took me somewhere around 3-4 hours. I could've taken much more but I had some prior experience with several of the tools here, and also had a gameplan that I stuck to. I didn't strive for perfection but rather for completion, as I wanted to get through all the motions once so that I can understand the entire workflow before starting my next project.
I hope you found this helpful and/or entertaining, and I look forward to offering you more (and better) videos in the future.