The Foundry Un-Rolls Your Shutter

The Foundry have released a $500 plug-in for Nuke and After Effects that attempts to remove rolling shutter artifacting, AKA “Jell-o cam,” from CMOS footage (RED One, Canon HV40, Canon 5D Mark II, etc.). It’s based on a technology demo that they showed at NAB earlier this year.

I’ve tried it, and it works—sometimes. The Foundry are well known for being leaders in the motion estimation field, and they have harnessed their unique experience in this area to attempt the impossible: a per-pixel reconstruction of every part of the frame, where it “would have been” if the shutter had been global open-close instead of read out a line at a time.

When it works, it’s brilliant. But the more motion you have in the frame, the more likely this plug-in, mighty though it may be, will get confused. Unfortunately, when this happens, parts of the image turn to scrambled eggs.

It’s a very similar problem to re-timing 30p footage to 24p actually—in fact, I wish The Foundry had added an option for frame rate conversion to this plug-in. Although, in fairness, I would only rarely use it.

Folks are always asking me about converting 30p to 24. I responded in a thread on the Rebel Café:

…The more motion you have, the more likely it is that any optical flow re-timing system is going to encounter problems.

Not even getting into the combat shots, here’s the Apple Compressor method referenced in Philip’s tutorial failing on one of the more sedate shots in After The Subway:

Now I’m not saying that you won’t occasionally see results from 30-to-24p conversions that look good. The technology is amazing. But while it can work often, it will fail often. And that’s not a workflow. It’s finger-crossing.

On a more subtle note, I don’t think it’s acceptable that every frame of a film should be a computer’s best guess as to what happened between captured frames. The magic of filmmaking comes in part from capturing and revealing a narrow, selective slice of something resonant that happened in front of the lens. When you do these motion-interpolated frame rate conversions, you invite a clever computer algorithm to replace your artfully crafted sliver of reality with a best-guess. I feel (and feel free to disagree, I won’t bother arguing) that this artificiality accumulates to create a feeling of unphotographic plasticness. Screw that. Didn’t you select the 5D because you wanted emotionally resonant imagery? You’d be better off with a true 24p video camera that works with you rather than against you, even if it doesn’t give you the convenient crutch of shallow DOF.

To be crystal clear, that’s Apple Compresser failing to properly estimate the motion in one particular frame when trying to convert from 30p to 24p. Nothing to do with The Foundry or their new plug-in, which actually tackled that same shot quite well.

The Foundry knows that they’ve made a tool to help us limp along while camera manufacturers sort out this CMOS issue. Like any crutch, I wouldn’t plan on leaning too hard on it—but kudos to The Foundry for attacking this problem head on, and making a product out of a technology demo in record time.

Update on Thursday, August 6, 2009 at 2:28PM by Stu

Of course, there will always be some rolling-shutter artifacts that can never be fixed:

Flatten Your 5D

Readers of The DV Rebel’s Guide know that I like to set up my cameras to record as much dynamic range as possible, resulting in a low-contrast, low-saturation “digital negative” that allows more flexibility for grading in post. If you’ve seen any video I’ve shot with the Canon 5D Mark II lately, whether in the Red Giant TV tutorial or on fxphd, you may have wondered how I set up the camera to achieve this. The answer comes in the form of some in-camera Picture Style settings.

I posted a while back about using Canon’s PictureStyleEditor software to create these custom Picture Styles. Since then I have found that the controls in the camera are more than sufficient for creating a more post-friendly look for the 5D’s video files.

Starting with the “Neutral” setting, I make the following modifications:

• Sharpness all the way down
• Contrast all the way down
• Saturation two notches down

Save that as on of your User Defined Picture Styles. Then, optionally, hop over to the Custom Function menu, select C.Fn II: Image, and enable Highlight Tone Priority.

UPDATE: Or don’t. See Update 1 below.

Your settings should look like this:

This will remove the contrasty, video-like tone curve from your future recordings, and eek out a little more highlight detail. Here’s a shot made with the default settings:

Video frame made with Standard Picture Mode, Highlight Tone Priority offHere’s that same shot with the Stu settings:

Prolost Settings

You can see the reduced contrast, the increase in shadow and highlight detail. It’s closer to a raw image with a linear tone curve. Zoom in and you can see the difference in highlight handling on the reflections:

Default settings — harsh, clipped highlightsProlost settings — smooth rolloff in the highlights, increased detailYou can also see the difference in sharpening artifacts. My settings reduce, but do not eliminate, the moiré effect endemic to the 5D Mark II’s line-skipping:

Default settingsProlost settings

You’re going to put all that contrast and most of that sharpening back in post of course, but in your own way, and with more control, and after any shot-to-shot evening out or clever power windows. The ability to design a “zeroed out” or CINE_LIKE-esque Picture Style is one of the things that makes the 5D Mark II’s video dangerously close to usable.

I recommend assigning this setting to one of your Custom notches on the mode dial, so that you don’t inflict these settings on your stills. They don’t affect raw files of course (UPDATE: That’s not actually true, see Update below), but they do get baked into the JPEG previews that accompany those raw files. I have the above settings registered as C3 (along with manual control and a 1/60th shutter), so I can quickly pop into my ideal movie capture settings.

Update on Friday, August 21, 2009 at 9:11AM by Stu

If you do set up your 5D as described above, make sure to do the last step with the custom function so that these settings only affect your video shooting. You probably don’t want to have Highlight Tone Priority on for your stills, especially if you shoot raw. What it does is “push” (reduce) your ISO setting by one stop, causing you to capture a full stop of additional highlight information through no more sophisticated means than simply underexposing. For video and JPEGs, that additional highlight range is non-linearly mapped into the image, creating a very film-like “shoulder.” For raw, it simply passes metadata to your raw decoder that may or may not be honored.

Lightroom, for example, interprets HTP as a simple instruction to boost exposure by one stop. Fair enough, except that this won’t match what you saw on the camera LCD at all. Blacks will be too crushed and that extra highlight detail will be missing (until you futz with the Develop controls to bring it back). Trying to bring more detail back into the blacks will bring in nasty noise that was not visible in the JPEG preview.

Underexposing to capture extra highlight info can be a fine idea, but it’s best implemented case-by-case, and in a predictable way.

Highlight Tone Priority hides the underexposure from you and might cause you to think you’re capturing a more robust negative than you really are.

But for video mode, it creates a more film-like low-contrast, high dynamic range image, with just a touch more noise from the one-stop push.

I now roll with two custom function settings, one with HTP on, one with it off. For daylight exteriors with plenty of light, HTP: On. For lower-light situations, or more controlled lighting, HTP: Off.

Update on Monday, August 24, 2009 at 2:09PM by Stu

There’s an interesting discussion of the nuances of HTP and ISO performance over at cinema5D.com. Interpret tests results such as these with caution though — as the OP says, he’s trying to show off the noise, so it’s not so much a real-world test as it is an attempt to learn more about what’s happening in the camera. Turns out there are some surprizing variations in noise preformance at different ISO settings, possibly having to do with some ISO settings being “virtual,” i.e. metadata offsets of native settings.

fxguidetv #061

Can Mike Seymour and Stu talk about cameras for hours? Is Japan the greatest place on Earth? How bad do you want the Redrock Micro “Subling” rig? These and other pressing questions are answered in episode 61 of fxguidetv.

(Plus also lots of great moments from the now-underway DSLR Cinematography course at fxphd)

Subscribe in iTunes.

Nikon D300S

The Nikon D300S features 24p video at 1280x720, with an external microphone input. DX (crop) sensor.

It also features limited autofocus in video mode. The D300S will auto-focus on a fixed point in the frame.

This page hints that it might have some manual exposure control (thanks @tylerginter!).

Nikon also clearly learned from Canon in that they got the camera into some pro shooters’ hands before its release:

Rolling shutter is either greatly improved from the D90 or it’s being expertly hidden in editing.

You can pre-order the D300S now from Amazon. $1,800, body-only.