Converting 30p to 24p
Monday, March 8, 2010 at 9:02PM As the long-awaited 24p firmware update for the Canon 5D Mark II draws near, I joined Mike Seymour on episode 57 of the Red Centre podcast to talk about how excited I am that it marks the end of painful workarounds for the 5D’s no-man’s-land frame rate of 30.0 frames per second.
For as long as I’ve had my 5D Mark II, I’ve avoided using it for any projects that I could not shoot 30-for-24, i.e. slowing down the footage to 23.976 fps, using every frame. My 5D has been a gentle overcrank-only camera. There are plenty of occasions to shoot 30 frames for 24 frame playback—we do it all the time in commercials to give things a little “float,” or to “take the edge off” some motion. I still do this often with my 7D. Whatever frame rate I shoot—24, 30, 50 or 60, I play it back at 24. Just like film.
Folks ask me about 30p conversions often. Twixtor from RE:Vision Effects is a popular tool for this, as is Apple’s Compressor. Adobe After Effects has The Foundry’s well-regarded Kronos retiming technology built-in. All of these solutions are variations on optical flow algorithms, which track areas within the frame, try to identify segments of the image that are traveling discretely (you and I would call these “objects”), and interpolate new frames based on estimating the motion that happened between the existing ones.
This sounds impressive, and it is. Both The Foundry and RE:Vision Effects deservedly won Technical Achievement Academy Awards for their efforts in this area in 2007. And yet, as Mike and I discuss, this science is imperfect.
In August of 2009 I wrote:
I’m not saying that you won’t occasionally see results from 30-to-24p conversions that look good. The technology is amazing. But while it can work often, it will fail often. And that’s not a workflow. It’s finger-crossing.
On a more subtle note, I don’t think it’s acceptable that every frame of a film should be a computer’s best guess. The magic of filmmaking comes in part from capturing and revealing a narrow, selective slice of something resonant that happened in front of the lens. When you use these motion-interpolated frame rate conversions, you invite a clever computer algorithm to replace your artfully crafted sliver of reality with a best-guess. This artificiality accumulates to create a feeling of unphotographic plasticness.
Of course, it’s often much worse than a subtle sense that something’s not right. Quite often, stuff happens in between frames that no algorithm could ever guess. Here’s a sequence of consecutive 30p frames:
Right-click and select View Image to see full-resNothing fancy, just a guy running up some stairs. But his hand is moving fast enough that it looks quite different from one frame to the next.
Here’s that same motion, converted to 24p using The Foundry’s Kronos:
Right-click and select View Image to see full-resBlech.
Again, don’t get me wrong—these technologies are great, and can be extremely useful (seriously, how amazing is it that the rest of the frame looks as good as it does?). But they work best with a lot of hand-holding and artistry, rather than as unattended conversion processes.
(And they can take their sweet time to render too.)
I’m so glad we’re getting the real thing.
Reader Comments (42)
Optical flow is also at the heart of Avid's 'Fluid Motion' retiming method - and I stopped using it a long time ago for this reason. More often than not I found it produced weird and unwanted motion artifacts and distortions. Usually it would look mostly fine in motion, but sometimes some odd distortion would catch the eye, and then if you slowed it down or looked more closely - well then there was lots of weridness.
Fluid Motion offers the ability to view and edit the motion vectors, and I've tried to hold it's hand in the past - sometimes I manage to correct one thing but then manage to introduce a whole lot more weirdness.
Yes. 5 years ago I Twixtored. I said" Well, that is cool and not going to work". 24P is the only way to get 24 frames to capture. The cadence is so burned into our film viewing experience, our amazing minds so good at discerning temporal pace, that NTSC 29.97(30P) looks Waaaaaay different.... and the difference is not appealing to me, and I am taking a guess, Stu's preference as well. The best visual comparison of 24 to 30 is at your local Best Buy, watching Transformers 2 in "120mhz motion flow", quite possibly the worst technology ever adopted by TV manufacturers.
Still, there is a certain kind of editor who reaches for the retime button with very little thought for the implications downstream, when online/confirm time comes. If they get lucky, the retimes can be done 'in the DI' (if the material suits and the DI system in use is up to it). If they get unlucky, there are a new bunch of unexpected vfx cleanup shots to pay for, late in the schedule. A savvy post supervisor will swing by the cutting room before things get out of hand, and enquire about re/speeds!
For the sake of discussion, I'm curious if you caught redcentre 56 with Shane H. and your thoughts/responses. I ask, not to create conflict, but rather i'd really like to know how you'd respond to some of his opinions and methods. As a serious post guy, I cringe at some of the ideas proposed.
I've done a lot of Twixtoring in the past. I was taking a lot of Betacam footage shot at 29.97 interlaced and making it 24p. My workflow was to first use RevisionFX FieldsKit to make a 59.94 progressive movie, and then Twixtor it down to 23.976. The benefit? Twixtor had more data to work with, and I had less work to do on the settings.
Since most (if not all) of my work is video bound, Now I just ask for 30p (29.97) footage and just work in that realm. I just dont have the time for that many submasters anymore.
I keep thinking to myself, "Optical Flow, the most amazing tool you hope you never have to use". Disregarding creative uses (?) and the fact that the technology behind these tools is quite incredible, their use seems almost the post-production equivilant of using a 360 shutter.
Stu did you hear the Shane Hurlubut interview on the last RedCentre (ep 56) Sahne totally insisted on shooting everything at 30 and then re-timing it with twixtor, said it was part of his process, likes the softness.
Or something.
I just about hurled. Thank you for talking sense on RedCentre this time.
I've been using my 5D for music videos for Bay Area artists, and each time I had to slow down 30p to 24p (and speed up the audio during the shooting process). It's a nice effect, that usually compliments music videos, but it gets old. I'm glad we're getting 24p too.
The rencentre guys must've felt guilty after so much BS on the last episode and called you in to make up for it.
I agree with several posts here cringing at Shane Hurlbuts Twixtor "cocktail" comments. His work is quite good...beautiful actually. That said, I get the feeling that he really doesn't understand how the technology works. I doubt he has ever sat down an done the post himself. As Stu relates in this posting Twixtor (or other flow technology) is not a reliable conversion workflow and I find it slightly disturbing that Shane so readily and publicly pushes it.
Likewise, his comments that he can just shoot handheld all the time and stabilize the footage is equally ill advised. Granted, you can get some great shots because of the smaller mass and body style of the DSLR's vs a larger film camera, but I'd be more than happy to sit down and show anyone how trying to stabilize even the smallest little hand induced jiggles in 5/7d footage more than often reveals rolling shutter jello, stretching and distortion that makes the footage virtually unusable. This is especially true in closer shots.
Finally. A "cocktail" of conversion, stabilization, denoising and regraining tends to take forever...even on the fastest of Mac Pro's.
I'm glad Shane is doing great, high end work with these cameras, but he's fairly off the mark with a lot of his commentary and I hope people don't take them to seriously. Everyone should shoot for success, get it in the camera as best they can and at the speed they want and save the great software out there for the instances that they really need to save a desirable shot.
Guys, I love Shane's work. I think it's really gorgeous. He said in his Red Centre interview "Everyone is going to have their own opinion." Here on ProLost, you can read mine.
I've lost count of the number of times that I've had retime shots to cleanup... As a few people have said, optical flow retime tools are great, and very smart, but there are some things that you just can't let a computer try to figure out on its own. Sometimes you just need a human to step in there and get rid of those last artefacts.
I have bought Kronos because everytime I have test it it work that was because I was lucky with the image and realise that it work when you do not have close up.
it's exactly like you said it's finger crossing.
Also the Thing that really piss me of with Kronos I have paid 500 bucks for it and one week later it was on special for 150.
I am not saying it's a bad product either. but like you said it's finger crossing technology, sometime it work sometime it don't. Thanks forsharing
Shake still has the cleanest OFlow retime for me, AND it has scene detection so you can just push the finished graded cut through before delivery. Obviouslt provided you have three to four hours up your sleeve on an 8-core. You still get the odd funky frame but really who cares? You wouldn't do your comps on retimed footage and it in no way effects the story.
To be honest the biggest issue I have had is losing motion blur and getting too crisp a result. As I'm in PAL land I was going to test shooting with the shutter at a 50th as that would make it correct for the finished product but I guess now I don't have to.
Cheers,
Toby
I have to agree with the other Matt on the as well. I can't stand the new 120hz gimmick on tvs. It was meant to be a temporary fix for the latency in response on LCD tvs being they did not have the quick response of plasma tvs, but everyone I know who has it watches everything in 120hz. In my opinion it destroys the original look of the movie / film / show etc.
Sorry in advance about sidetracking Stu,
Optical Flow and retiming, like so many other things we deal with, are just tools. There is a time, place or circumstance for the use of any of these tools, whether hardware, software, or technique. It is our responsibility to understand the implications of the decisions that we make regarding these tools. Thanks to Stu for expressing his opinions and furthering our knowledge.
For the record. I think Shane's work is gorgeous as well...and he did say everyone has their own methods. I ranted and pointed fingers a bit more than I normally do. Having been on both sides of the direction and post production fence it just annoyed me a bit that he was throwing information out there that I thought might cause newer filmakers to make choices that would cause a lot of grief later on.
@Eric Ferguson & Paul Schneider
I sat down with Shane Hurlbut a few months ago, and watched an absolutely astounding fifteen-minute, 1080p24 sizzle reel of material from a low-ish-budget action feature.
The material was truly remarkable - obviously extremely well crafted and lit - of navy seals running around jungles, aircraft carriers, and helicopters. One thing I was completely unable to spot, even on the typically problematic scenes, was the aliasing which - to my eye at least - still plagues 5D footage. When we talked about his process (Twixtor to 24p and the addition of a subtle film grain), I had a bit of an "a-ha" moment: if aliasing artifacts are a side effect of the technical details of the camera, and they vary unpredictably from one frame to the next - unlike the actual subject being filmed - it stands to reason that an optical flow solution would discard them and only synthesize new frames from the real subject.
To my eye at least, this entirely "synthesized" material looked significantly better, with more "film-like" motion, than I'm used to seeing with native frames from the 5D. When the firmware is released, I'd be very interested to see a side-by-side of material shot this way against native 24p material.
@Stu
I've not used either extensively myself, but I've seen much better results from Twixtor than Kronos on close-to-default settings. The terrible, wiggly Kronos artifacts you have above don't seem to occur with Twixtor. I think it might just have smarter defaults.
Ben, I tried both Twixtor and Kronos for this test. I chose to show the one that gave the best results.
I'm curious if anyone has compared Kronos and Twixtor against the DVO Twister tool from Digital Vision?
(full disclaimer, I recently joined Digital Vision as a software engineer)
Congrats Tim, DV is a great company. I miss my Film Master!
I never tried using it to do a 30p to 24p conversion, but it sounds like you might be in a position to do just that.
I will bet you that what you will find will be that it will work wonderfully in some cases, and not so much in others.
Detect a pattern here?
I couldn't believe what Shane was saying (firmly believing in shooting 30p and twixtoring) on the podcast and how serious he was about it. And I couldn't believe his navy spot, shot 30p and twixtored to 24p according to him, looked so flawlessly filmic.
Anyone even remotely involved in this particular work and know what went through in the post? I won't buy that 'apply twixtor to the timeline and come back next morning' was what happened.
Another question is, if this optical flow frame interpolation is such a hard math how do those TVs do 120Hz motion, in realtime? I still hate it but I'm impressed that they could do it in the first place, and I wonder if whatever TVs are using can be mixed with twixtors to improve the result. Say, 30p -> 60p -> 24p.
Those TVs fail all the time, but they fail back to good old 24p. If you look closely you can see artifacts galore.
Hmm, next time when I'm at Costco I'll stick around longer to see the artifacts. It's amazing that people will withstand such thing, and I fear that tasteless technology can actually affect people's preference to go lower on average. Like kids growing up on fast food and snacks having lower standard for food and losing ability to truly enjoy high quality cuisine.
Straight lines appear to be the biggest problem for retiming software. I've used a couple of standalone programs in the past, Slow Motion & Motion Perfect. They both do a brilliant job, but just like your example pics above they have problems dealing with straight lines.
Hey Stu,
Thanks!
Yes, frame-rate conversion utilising motion estimation, does get it wrong some times with interesting consequences. Normally the frame-rate converter will fall-back onto a "safe" mode, so the output will always be as good (or better) than the same sequence put through a field-blend type frame-rate converter.
What would be very interesting would be to shot 60i and then frame-rate convert to 24p, the extra motion might significantly reduce the number of artefacts generated (because the motion estimator has more information for fast moving objects).
IIRC there are some rules of thumb about the speed of motion (of the subject and of the camera) when shooting film (24p). Do you follow the same rules when shooting 30p for frame-rate conversion to 24p?
Cheers,
Tim
Don't overlook the fact that you don't often if ever get decent results out of retiming by just hitting the Retime button.
What Twixtor, Kronos, and others benefit from is getting in there and providing a great deal of additional information for the retiming of a shot -- animated masks in AE, or the equivalent in other packages -- in order to figure out where the edges are, etc. Then you retime the shot, and if it looks like hell, you do more.
It's almost like roto. There's no Roto button. Retiming is VFX work, too, and it's never easy, or fast.
If I'm shooting at 60fps, and playing back at 24fps, is 1/125 shutter speed the best option on the 7D for a film look, or should I stick to 1/50??
For shooting at 30fps playing back at 24fps should I be using 1/60 shutter speed??
Any thoughts appreciated.
Yes, always use a minimum shutter speed of double the frame rate. So for 60fps, use 1/125 (closest available speed to 1/120), and for 30 fps use 1/60.
The 180 degree rule always applies, even when over- or under-cranking. The only exception would be if you knew you were going to skip frames in the transfer for some reason.
And now that I have a 7D and 5DMKII in my hands, I somehow JUST now realized that you CANNOT put the cams in 48th shutter or 120th shutter. Well.... that SUCKS. Furthermore, I have yet to see anyone asking for Canon to do this!? My HV20 does 24@48th, and guess what, with a JAG35 DOF adapter, it look really close to 16mm film. If the 7d/5D would JUST allow me to select/force 48th, we would all be saying how much it looks like Super35 film. 2fps is a REALLY big deal! So I am going on a shoot next week, seriously considering just doing 25P/50th the whole time, even on the 7D... and slowing that to a 23.978 timeline/encode.. since I am throwing out all the sound. When the sensor grabs data at the 48th "shutter", the fluidity and softness and motion blur is as close as you will get to film IMHO. A great lens, like PL mount cinema lens + 48th shutter and 24P should look nearly indistinguishable from 35mm movie film.
Which brings up a point, can a RED camera do 48th shutter or 120th shutter?
Yes, of course it can.
Regarding the 5D and 7D shutter speed limitations, I would suggest that it's not that big of a deal. I'm a total shutter Nazi, and 1/50 is close enough to 1/48 for me. Same with 1/125 and 1/120. In both cases, the shutter speeds afforded by the Canon cameras are ever so slightly faster than what would be ideal—which is a far better direction to err than slower.
It would be lovely if Canon offered the true 180º shutter speeds for each frame rate, but there are bigger problems that I'd rather see them address first.
cool Stu, thx for your opinion. I just shot a small series of backplates... just repeated versions using 5D 25fps/50th and 7D 24fps/50th... just to have a grouping using slightly diff settings. I am liking that I can 2D track the crazy DOF in Mocha! Opens up some easy integrations on closeups with CG chars
Hey Stu, of course you're right about not thinking "it’s acceptable that every frame of a film should be a computer’s best guess" but purely from a technical point of view isn't that already happening with all the compression algorithyms these days?
"part of his process, likes the softness" reads like 'he likes other people to paint his failed oflow frames for him instead". Extremely selfish and irresponsible.
Hi Stu, just finished listening to the RedCentre and had a quick sound question. You said going from 30fps to 29.97 should help sync the sound a lot better. What about 24(23.976)??? Is there a particular frequency the sound should be recorded in to make sure it stays in sync? I recently bought a Zoom H4n and was just curious. didn't know if 24 had no problems with sound or not. Thanks!
Since the 7D runs at 23.976, you won't have any problems with sound sync.
Can you put raw 30 fps file about 2-3 sec on ftp or dropbox?
the firmware is up on cannon's site!
General Q: Why choose 24 p not 25 p.
Isn't 25p the right choice if you are in europe.
Pro and cons - 24p vs 25p ?
Yes, use 25p in Europe. Although some European 35mm film productions do use 24 fps—I guess it depends on whether TV or theatrical is your primary distribution? Maybe a reader from a PAL country can weigh in with the rules of thumb here?
there is a certain kind of editor who reaches for the retime button with very little thought for the implications downstream, when online/confirm time comes. i think iIt would be lovely if Canon offered the true 180º shutter speeds for each frame rate..what do you think so?