The Foundry Un-Rolls Your Shutter
Thursday, August 6, 2009 at 9:01AM The Foundry have released a $500 plug-in for Nuke and After Effects that attempts to remove rolling shutter artifacting, AKA “Jell-o cam,” from CMOS footage (RED One, Canon HV40, Canon 5D Mark II, etc.). It’s based on a technology demo that they showed at NAB earlier this year.
I’ve tried it, and it works—sometimes. The Foundry are well known for being leaders in the motion estimation field, and they have harnessed their unique experience in this area to attempt the impossible: a per-pixel reconstruction of every part of the frame, where it “would have been” if the shutter had been global open-close instead of read out a line at a time.
When it works, it’s brilliant. But the more motion you have in the frame, the more likely this plug-in, mighty though it may be, will get confused. Unfortunately, when this happens, parts of the image turn to scrambled eggs.
It’s a very similar problem to re-timing 30p footage to 24p actually—in fact, I wish The Foundry had added an option for frame rate conversion to this plug-in. Although, in fairness, I would only rarely use it.
Folks are always asking me about converting 30p to 24. I responded in a thread on the Rebel Café:
…The more motion you have, the more likely it is that any optical flow re-timing system is going to encounter problems.
Not even getting into the combat shots, here’s the Apple Compressor method referenced in Philip’s tutorial failing on one of the more sedate shots in After The Subway:
Now I’m not saying that you won’t occasionally see results from 30-to-24p conversions that look good. The technology is amazing. But while it can work often, it will fail often. And that’s not a workflow. It’s finger-crossing.
On a more subtle note, I don’t think it’s acceptable that every frame of a film should be a computer’s best guess as to what happened between captured frames. The magic of filmmaking comes in part from capturing and revealing a narrow, selective slice of something resonant that happened in front of the lens. When you do these motion-interpolated frame rate conversions, you invite a clever computer algorithm to replace your artfully crafted sliver of reality with a best-guess. I feel (and feel free to disagree, I won’t bother arguing) that this artificiality accumulates to create a feeling of unphotographic plasticness. Screw that. Didn’t you select the 5D because you wanted emotionally resonant imagery? You’d be better off with a true 24p video camera that works with you rather than against you, even if it doesn’t give you the convenient crutch of shallow DOF.
To be crystal clear, that’s Apple Compresser failing to properly estimate the motion in one particular frame when trying to convert from 30p to 24p. Nothing to do with The Foundry or their new plug-in, which actually tackled that same shot quite well.
The Foundry knows that they’ve made a tool to help us limp along while camera manufacturers sort out this CMOS issue. Like any crutch, I wouldn’t plan on leaning too hard on it—but kudos to The Foundry for attacking this problem head on, and making a product out of a technology demo in record time.
Stu
Of course, there will always be some rolling-shutter artifacts that can never be fixed:
Reader Comments (29)
Interestingly, I wrote a 30p-to-24p tutorial for Vegas users on my blog the other day too.
The ghosting you see there on Compressor is just resampling. Vegas has a way to disable resampling and get rid of the ghosting.
I have found that the best way to convert non-speaking 30p footage to 24p is by slowing it down exactly by 25%. When an exact number is entered to the slow-down menu that converts the footage by a "clean" number, then no ghosting is introduced. And if in some cases it is, then by disabling resampling you can get rid of that too.
I believe that After Effects has such controls too, but I don't know about Compressor/Premiere.
I'm working on a project that has an effect produced by shooting with intentionally two specifically aligned crappy "lipstick" cameras that save video at 30 fps in a motion JPEG variant.
For the final shots I've been converting that footage to 24p to match our Red One source. Even with the low quality footage the conversion (via Apple's Compressor) is sometimes just whacked.
Even though I'd expect the lower quality video to be less easy to convert or estimate motion, it would seem that the quality of the source isn't going to guarantee a good conversion.
Eugenia, this is not a matter of enabling or disabling anything, this is compressor trying to do motion estimation and reverting to simple frame blending when it can't guess the motion vectors.
But I definitely agree that the best thing to do with 30p footage is simply play it back at 24, for a slight slow-mo effect. That's all I do with my 5D.
I think we are talking about the same thing, but with different terminology. You see, what you call "motion estimation" it's called just "resample" on Vegas. And it creates ghosting on Vegas as well as it does on Compressor (check the example picture on my article linked above). Which is why Vegas has a switch to turn it off, and just drop frames rather than try to guess motion. I know AE has a way to turn that off too, I am just not sure if Compressor has such a switch too.
"I feel (and feel free to disagree, I won’t bother arguing) that this artificiality accumulates to create a feeling of unphotographic plasticness. Screw that."
hey stu,
*totally* agree with that!
i have the impression that by optical flow retiming, we're basically converting live action into CGI: have the computer create shapes and have them re-rendered into new positions. and as with so many things CGI, it usually screams "fake". (i mean, i can't be the only one who thinks that the effect work in 2001 looks more pleasing then the CGI in Star Wars III - and that was over 40 years ago!).
on the other hand, rolling shutter is even more annoying then the plastic feel, so it's good to know that the foundry is trying to sort out the problem.
however, i returned my D90 after three days and went back to shooting film for my personal projects, with all the lovely grain texture and skin tones and highlight handling.
++ chris
you can obviously disable it, but what's the point in using motion compensation if you're not using it? the whole point of using compressor in the first place is that it tries to generate frames based on motion. what you're seeing in that frame is what happens when it fails, not what it's supposed to do.
btw i'm experimenting with a virtualdub plugin called deshaker. it removes rolling shutter artifacts globally and it works really well. this doesn't help when the subject moves of course, but it removes the jello from handheld footage and skew from pans and tilts.
http://www.guthspot.se/video/deshaker.htm
DOF overuse is fake too, as is color correction and many other effects we use in post. Image stabilization is just another tool ... it saves my ass all the time.
Rambo
Hi Stu,
I found this today and thought you might be interested in seeing it.
Some pretty intense slow motion. The guy broke the 180 degree shutter rule... he was shooting at 1/800th!... but I think it really made a difference in the in end to really give a 'frozen' look!
http://www.vimeo.com/5565214
Hi Stu,
I was wondering why my Bendy Prop video suddenly jumped to 426 views in one day. It's an honor to be mentioned on your blog!
Cheers,
-Steve
HI, A friend of mine invented this for th rolling shutter problem:
http://www.vimeo.com/5565214
marz
hi Rambo,
i was talking about images that have been modified using optical flow/motion estimate technologies, not traditional stabilisation. i'm pretty sure there are a lot of people who don't mind the artifacts introduced by this, or don't even see them, but personally i just find them very disturbing and try to avoid them whenever possible.
++ chris
Another way to fight rolling shutter
http://www.vimeo.com/5565214
Eugenia, you and Stu are in fact not talking about the same thing. Resampling and motion estimation are two differnent things. Allthoug they are often used to solve the same problems.
Resampling (the way Vegas does it is simply merging frames (effectively laying the next or previous frame ontop of the current with 50% opacity, to smooth out the motion it the clip.)
Motion estimation is far more advanced add can be on a per-pixel level. It is basically drawing a new frame based on the frames before and after the "fake" one that have to fill the gap.
It's why twixtor or furnace gives a far superior slow or quick motion than FCP or Vegas.
Motion estimation is used for speed changes, adding motionblur, warping/morphing, stabilizing and now rolling shutter artifact reduction.
I kinda dig the bendy propeller video. Maybe someone can wrap a clever narrative around the effect and find a win-win for rolling shutter. ;-)
That is the coolest demonstration of rolling shutter I have seen!
That’s very cool. A lot of really complex math to track all those pixels and their intended velocities. Incredible!
Stu,
I work with the people who write Compressor. If you'd like to share your video with us, maybe we can make Compressor work better. (And if you think it's not worth the time and would just like to see your hardware work better, that's fair enough.) Send me an email at dcardani@apple.com if you're interested.
Darrin
Stu,
Firstly, have you tried doing the retime in Shake? Even though it is meant to be the same as compressor these days I find Shake's optical flow retiming far superior. I even get better results out of it than with Kronos half the time.
Also that propellor video is too cool. I would have had to spend hours creating that as an effect if it weren't for rolling shutter!.
Cheers,
Toby
$ 500 for a "finger crossing" app? No way Foundry, no f*** way!!
These CMOS Cameras can be used as special effects cameras to save you time in post as seen in the "propeller" movie.
Anyone who is supporting (means buying) these cameras must be a rich man who has time for goofing around or hopelessly ill in his mind nowadays.
This technology is just ridicoulos and a complete wind up with every piece sold.
PERIOD
Hey Stu,
What are your thoughts on the Panasonic GH-1? Everyone seems to be talking about it lately. I've been impressed with how it can shoot full 1080p and can autofocus during video....and is only $1,500. Have u shot one one yet and if so what are your thoughts?
Do we (humans) have kind of rolling shutter? Take a wand or a pencil with the tumb and finger and move it up and down rithmically. It seems as the pencil bends.
Simon.
you'll see echo and trail, depends on the (over)dose of real life-drug you're contaminated.
What I don't understand is, why are there no consumer HD cams with global shutters (or at least extremely fast read-reset rolling shutters) on the market? I am well aware of the technical hurdles involved in developing one, but people have been complaining about Jell-o footage for several years now. The first manufacturer to release such a camera -- preferably a micro 4/3 or APS-C running at 24p and with HDMI and RAW out -- will grab massive market share.
Tzedekh, there are plenty such cameras. Just look for one with a CCD sensor, such as the HVX200 and some of its ancestors.
Stu, thanks for the heads up, but I'm aware of the HVX200 and the other three-chip CCD cameras. They cost thousands more than the GH1and, unlike the GH1, can't make use of commodity lenses. What I'm referring to is an as-yet nonexistent GH1-like model with a global shutter, better AVCHD (e.g., High Profile@Level 4.1 at 24 Mbps) or even High 4:2:2 Profile, HDMI 1.3b and RAW out, and 24p mode, for about the GH1's price. Not feasible today, but the manufacturer that moves in that direction soonest will garner and probably hold a huge market.
The funny thing about all of this is that someday there will be pricey pluggins to replicate the rolling shutter "look" in much the same way people now try to duplicate super8 and the film look.
Chaywa, actually some ccd HD camcorders have it as expression effect.
I guess the sony hc9, if i right.
What if you could more accurately estimate the motion? I'm thinkingnof the iphone 4 being an example here - you could theoretically hbe as a parallel track to the video and audio, the gyroscope readings. Could you use a gyroscope in my way to better help some CMOS issues?