Boosting Stop Motion Animations to 60 fps using AI

Welcome back, today I got a different sort of Stop-Motion video for you: A video straight from the state-of-the-art of AI research. If you’re a Stop-Motion animator, like me, you probably know the effort it takes to record movies at 30, let alone 60 frames per second. I personally stick to 15 fps in order to not go bananas when recording. But that is all history now. Researchers from Shanghai, California and Google have created a neural network that interpolates frames into your movie seamlessly and without any visible artifacts. That way, you can boost your framerate from 15 to 60 frames per second and beyond. I was curious to see if this also works with Stop-Motion videos and so I applied it on my Apollo 11 movie. On the left, you can see the original footage. And now let me reveal the interpolated version on the right. Wow, this is just stunning! I’ve tried out many approaches for frame interpolation lately, but this result literally blows my mind. For example, look at how the interpolation works both for near and for far objects. The technique the researchers applied to achieve this explicitely models this depth information which is why they named it DAIN, short for Depth-Aware video frame INterpolation. Another side effect is that occlusion – that is objects that hide behind other objects – is not a problem anymore. Here is another snippet of the movie. I’ll play it in slow-motion so that you can see that there are no visible artifacts in the interpolated video frames. In fact, if I wouldn’t know which pictures are the real, recorded ones, I would have a hard time telling them apart from the AI generated ones. But now comes the surprise: The software to do this interpolation is completely free! You don’t need any knowledge in coding and can directly run it on your computer. (given that you have a graphics card that is capable of it) I’ve provided links to the developers Patreon page in the description where you can find the latest version. If you are an AI researcher, like me, you can also play around with the source code, which is openly available on github under a permissive license. I must admit that this is literally a dream coming true for me. I can tell you from my own experience that going from 15 frames per second to 30 means three to four times more work during recording. And now, it’s just a mouse click and your computer can do all that extra work for you. I’ve uploaded the whole Apollo 11 movie in a 60 fps version to YouTube which you can find here or in the description. I’m really curious to know which version you prefer. Please leave a comment down below and tell me if you like the charm of a raw 15 fps stop motion or prefer the eye-charming 60 fps version the neural network generated. I hope you liked this kind of different video, and I’ll see you next time!

7 Replies to “Boosting Stop Motion Animations to 60 fps using AI”

  1. What do you think about the results? Are you as baffled as me by just how good this looks? 🙂

    Was haltet ihr von dem Ergebnis? Seid ihr genau so erstaunt wie ich davon, wie gut das Endergebnis aussieht? 🙂

  2. Eine Frage reicht ne Gtx 1060 mit 6 GB Video Speicher? Ich habe 16g ram und ne amd R 5 2600 sollte reichen oder ?

  3. Spannend zu sehen, wie weit die KI in diesem Bereich ist. Das Ergebnis ist wirklich beeindruckend. Trotzdem gefällt mir persönlich der Look der 15 fps besser. Die 60 fps sind mir zu glattgebügelt, irgendwie nicht authentisch.

Leave a Reply

Your email address will not be published. Required fields are marked *