Space Monkeys, Raw Video, and Giving Us All You've Got
Wednesday, May 15, 2013 at 10:23AM The team at Magic Lantern have managed to hack the Canon 5D Mark III to record 14-bit raw 1080p video at 24 frames per second. The results are stunning—the highest-quality video we’ve seen from a DSLR yet, comparing favorably to images from cameras costing much more.
This is a big deal. But maybe not as big a deal as some have made it out to be. Like Ham, the chimpanzee that was launched into space on a Mercury rocket, the Magic Lantern raw hack is less notable for its discrete accomplishment than for what it portends.
How it Works
I was skeptical about the announcement of raw video at 1080p. This wasn’t a sensor crop, this was a full-frame image, somehow downsampled to 1920x1080—yet still being touted as “raw.”
How is this Magic Lantern thing “raw” if it’s scaled down from sensor res to 1080p? Scaling requires debayering. Debayered is not raw.
— Stu Maschwitz (@5tu) May 13, 2013
The answer came back from Magic Lantern themselves:
@5tu it’s not debayered, it’s 14-bit RGGB.
— Magic Lantern (@autoexec_bin) May 13, 2013
@5tu no idea, only Canon knows how the downsampling works. On 5D3 there’s no moire, on 5D2 there is.
— Magic Lantern (@autoexec_bin) May 13, 2013
Put this way, it makes sense—and matches what Magic Lantern said in their original post:
Key ingredients:
- canon has an internal buffer that contains the RAW data
Of course. Canon sparsely samples the sensor (the popular theory is that they skip lines and bin rows) to create their own 1080p bayer image, which they rapidly debayer to create 1080p video frames. It looks like Magic Lantern have found a way to grab this raw buffer and save it directly to the CF card.
Whatever magic Canon worked to eliminate the moiré in the 5D Mark III’s video now benefits Magic Lantern users as well. And whatever downsides there are to this sparse sampling will also affect the raw hack.
What it Takes
Right now, the hack requires quite a bit of work to get up and running, and even more work to derive useable results.
But those usable results are compelling. There are tremendous opportunities afforded by the raw video hack over the compressed H.264 recordings the camera natively makes.
- Post-processing debayering can be much better than the hasty in-camera processing. The much-lamented softness of 5D Mark III video is improved noticeably by handling the debayering (including noise reduction) after the fact. Although I can’t help but wonder if we’d get even better results from a demosaicing algorithm tuned to the specifics of the 5D’s subsampling pattern.
- The 14-bit raw frames contain a great deal of dynamic range that 8-bit, heavily compressed video does not.
- No compression.
But there’s a price to pay for recording 14-bit uncompressed raw. It requires crazy-fast CF cards, and you’ll only get 15 minutes of video on that 128 GB card, according to Cinema 5D, who graciously posted their workflow and samples, “after struggling for a day” to get it all working.
What it Means
Even once the setup process is streamlined, and the raw-to-DNG conversion process is streamlined (or eliminated), uncompressed raw video is probably not the best option for most DSLR shooters. Better would be something like ProRes or DNxHD—but that would require high-quality debayering in the camera, which would seem to require more processing power than the 5D Mark III has to offer. More processing means more heat, which means a very different camera design than an SLR. It’s easy to see why digital cinema cameras start to get expensive.
The 5D Mark III raw hack is cool. It’s important. It’s something we’ll use, and like, and get good results from. But as exciting as it is today, it’s even more significant for what it means for the future of low-cost digital cinema cameras.
Mr. Ham risked his furry life to prove that a primate could flip switches in space. Three months later, a human being took the same trip. By proving that raw video is possible from the 5D Mark III, Magic Lantern have joined the forces pushing the industry in an important new direction.
Five years ago I wrote that large-sensor video had shown “that it’s no longer OK for manufacturers to make a video camera that doesn’t excite us emotionally.” Since the industry did such a good job of heeding that advice, my new mandate for the future is this:
It’s no longer OK for cameras not to give us everything they’ve got.
What I love about the new generation of cameras, such as those from Blackmagic, Red, and even GoPro, is that they all give you everything they’ve got. They’ll give you the highest image quality they can, in the smallest package possible. They’ll compress images as much or as little as you want. They’ll max out their resolution at the expense of frame rate, or vise versa—whichever you like. And they’ll pack their best dynamic range into any format they can record.
Compared to this, Canon and Sony’s digital cinema product lines seem cruelly restricted at every tier. The result was palpable at NAB. As I said on stage at the SuperMeet, the show seemed to belong to camera upstarts. To cameras from small booths. From the Phantom Flex4K with its Super35 4K at 1,000 fps, to the Digital Bolex, the exciting cameras at every price point were the ones not charging by the button, feature, or codec, but simply giving you all they could do.
What Magic Lantern have done is show camera makers that if they won’t give us all they’ve got, we’ll just take it anyway. The smart manufacturers will try to beat us there out of the box.
Reader Comments (22)
Fantastic post Stu. I'm pretty sure Canon never anticipated THIS. Let's hope they AT LEAST respond by providing ambitious and aggressive firmware updates for their entire EOS line.
Just a preemptive note for those demanding to know why Canon didn't enable this at the factory: writing large Raw files requires 1000X cards, which didn't exist when they were engineering the firmware for the 5D II and 5D III. Even though it's now (barely) possible to write the data, it's not 100% reliable, so they would be flooded with angry calls about dropped frames and card speeds.
Then again, you have to wonder why the "C" series cameras don't have Raw recording built-in, when we now know it's possible. My guess is that Magic Lantern may have forced Canon's hand to enable Raw on the C series.
OK, 1000x cards might not exist back then, but fast SSD drives did. For a small team of Canon engineers, it would take a couple of weeks to create a version of the 5D3 that records RAW to an SSD drive and which could be sold for $5K and still be very profitable. But they preferred to bring out a $15K camera that records to an 8-bit codec. Great for some, irrelevant for most.
What I dont't get is why Nikon is not doing this. Can you imagine a D5200 (14 stops of DR on an S35 sensor) recording uncompressed RAW to SSD drives?
As well as the complexities of writing data at this rate to some form of storage, there's also the problem of playing it back; the Magic Lantern hack can't playback the RAW video you capture.
"What I dont't get is why Nikon is not doing this. Can you imagine a D5200 (14 stops of DR on an S35 sensor) recording uncompressed RAW to SSD drives?"
Truthfully, I don't think it's a matter of capability. I think it's will. DSLRs have turned out to be massively useful for filmmakers... but I suspect our numbers are dwarfed by people who use DSLRs for stills only. I'm betting that the needs of video shooters are pretty low on Canon and Nikon's priority list.
What I don't get is why these companies limit their stills only customers to raw 8 bit images (what?)... HDR capture should be relatively simple to do from a stills perspective. Second, I'm wondering why the video divisions of Canon, Sony, Panasonic, et al aren't getting the message that digital cinema shooters want lightly compressed, high dynamic range images in a small, 'inexpensive' form factor... seems like that should have been obvious years ago... but they're still giving us $10k 8 bit MPEG 4 cameras... why!
Shawn
The raw video on the 5d3 has more moire to my eyes.
download the file here and check the buildings.
https://vimeo.com/66083408
Much better image for sure, but I'd wager they're still line skipping. You see it run along straight lines.
It seems pretty cool but I'm not sure how practical it is.
Shawn, 5D Mark III stills are 14-bit raw (if you shoot raw). They are, in fact, HDR—because they have a good 2–3 stops of overexposure latitude built-in. So, mission accomplished.
Edward, I agree with you. But, in fairness, the aliasing in that sample is mostly visible on bright horizontal edges like the lit trim on that building. Even a perfect Digital Cinema camera will alias if you aim it at something like that.
But that sample shows other artifacts that corroborate an assumption that the 1080p raw is achieved by skipping whole lines.
I would already be very happy if the MKIII would be as sharp as it should/could be. Compared to a C300 I had to downscale a MKIII Shot about 50% to get a similar pixel sharpness, that can't be called FullHD. RAW is a wonderful option for some selected Projects. But a clean, high-res compressed image in such a compact form factor would be awesome. That 5D softness bothers me every time.
I totally agree, It's not ok to hide any special skills.
"Shawn, 5D Mark III stills are 14-bit raw (if you shoot raw). They are, in fact, HDR—because they have a good 2–3 stops of overexposure latitude built-in. So, mission accomplished."
Fair enough, thanks for the correction Stu. :-)
Shawn
Aaaand we have a new paradigm "it’s no longer ok for cameras not to give us everything they’ve got" Thanks Stu
Thank you sir. I think it fits nicely with yours, which I also love.
It would be technically possible to make an adapter that acts as a CF card and passes through the image signal to an SATA port where you could attach an much cheaper, bigger and faster SSD. With 300MB/s+ 3.5k should be possible without frame loss, unless thats a limitation of the camera.
Maybe such an adapter already exists, haven't found one though.
I read on your twitter that you're playing with CineForm. If that's as a way to make the 5D3-RAW workflow a bit less painful, I'd like to hear your thoughts about it.
Personally, I've found it to be very convenient (playback at 15 fps instead of 1.5 fps), but also a big blow to image quality, not because of the compression (which is great) but because the debayering method in Adobe Camera RAW is a lot better:
Optimal image quality with the BMC
Have you found a way to work with CineForm without this hit to IQ?
Something like this but way faster
USB 1.1 Host Adapter CF Card
Lexar and Sandisk should write the Magic Lantern team a big check. CF spec is limited to 167MB/s so it doesn't seem like the larger size 3.5k frames would be viable even if there was a way to connect faster recording media. Lots of missing pieces still - beter camera controls, spanning to allow > 40 second recording times, some kind of scheme for audio syncing, an RGGB to cinemaDNG converter etc.
Canon may respond by improving the functionality of their cinema products but more likely they will take additional steps to lock down the hardware in the future to prevent upstarts like ML from exposing the limitations they impose.
Seems like there is a middle ground here for the 5d that would allow better image quality with higher bit rates and audio if Canon cared to implement it. Canon bring us the 5Dc.
"it’s no longer ok for cameras not to give us everything they’ve got"."
I think Panasonic has learnt this with the GH3 vs GH2. Who knows if with next models we have a nice surprise.
The ideal DSLR camera (for filmmaking) Canon or Nikon could make would be an 8 megapixels one with raw direct to card and/or external drive. Imagine no line skipping and full 14 bit raw at 4K with big dot CMOS.
@ Roger Smith, I've been thinking the same thing! This popped up, albeit far from ideal ... http://www.eoshd.com/comments/topic/2765-attaching-ssd-to-5d-mark-iii/
@Dan:
Much more serious discussion in the Magic Lantern forum:
http://www.magiclantern.fm/forum/index.php?topic=5470.0
(I thought I was being smart when I ordered some of these parts and adapters last week... I'll be playing with my soldering iron this weekend, but I guess that, if this can be done, the ML guys will develop the thing a lot faster than myself)
I´m playing around with RAW hack on my 5D2 and so far I love it.. for me its fantastic to be able to get RAW images that is much sharper and so much more gradeable.
My problem is After Effects since I need to convert my DNG... I want to export an 32bit float EXR Linear so I can grade them even further i Fusion or Nuke the 2 compositning program I use daily in my work.
It seems like After effects clamps my EXR color to 0-1 so even though I render out 32-bit EXR´s...
I have setup the project setting to use 32-bit and sRGB with Linear and in Intepret footage the color managment is grayed out so I import and make some adjustment in the RAW conversion and then render ou at EXR with 32bit and the same here the Color magamneg is grayed out but it says Srgb and Linear...
When I open the exr sequence in Fusion its clamped to 0-1.. Shouldn´t I be able to store the "RAW" 14-bit data in a 32bit EXR? If I push my RAW conversion so its burned out I can´t retrieve any info from my EXR´s in Fusion.. The white just gets grey...
Am I missing somethig vital in AE? Is the RAW conversion 32bit if I set my Project setting to 32bit in AE?
Sorry if my question is a little bit off topic...
Cheers and thanks for a great Blog
Daniel, your problem is that the Camera Raw module does not return HDR pixels. I wrote about how to get linear floating-point HDR from raw many years ago:
http://prolost.com/blog/2006/3/16/linear-color-workflow-in-ae7-part-6.html
But be warned: the settings I use in that post don't apply to the new Camera Raw processing version. You'd have to manually choose PV2010.
http://help.adobe.com/en_US/lightroom/using/WS2bacbdf8d487e58240e1c02a1341ed8e630-8000.html
With regard to the last two comments, I'm experimenting right now with a similar workflow, and I'd apprecaite any comments. I am processing day-to-night timelapses shot in Canon CR2, where you often need to keyframe grading parameters as lighting conditions change. Other people use Bridge and custom JSX scripts to drive Camera Raw, ramping its parameters (exposure, WB, etc.) through the sequence, but I prefer AE's interface (keyframes, ease in/out...).
Thus, I am following Stu's linear workflow guide in order to import the full dynamic range of the RAW files into AE in 32 bits, and I then apply Colorista there. I'm not doing any compositing, only grading, so I don't check the "linearize working space" option. Am I on the right track with this workflow, or am I making a mess of things and don't even know it? Also, is there something in Camera Raw's new PV2012 that might make it worth it to do the grading there, even if it means giving up AE's advantages?
Paul, using Colorista II to grade time lapse is a good idea, but you needn't be concerned about linear color space. Just get a nice, flat image from the raw importer, making sure not to clip any highlights. Then color grade to taste in Colorista, animating whichever adjustments are needed.