Passing the Linear Torch
Wednesday, September 30, 2009 at 1:38AM 
I used to show you weird crap like this all the time
Back in the day I blogged a lot about how compositing and rendering computer graphics in “linear light.” a color space in which pixel values equate to light intensities, can produce more realistic results, cure some artifacts, and eliminate the need for clever hacks to emulate natural phenomena. Along with Brendan Bolles, who worked with me at The Orphanage at the time, I created eLin, a system of plug-ins that allowed linear-light compositing in Adobe After Effects 6 (at the mild expense of your sanity). I also created macros for using the eLin color model in Shake and Fusion. Along the way I evangelized an end to the use of the term linear to describe images with a baked-in gamma correction.
Then Adobe released After Effects 7.0, which for the first time featured a 32-bit floating point mode, along with the beginnings of ICC color management, which could be used to semi-automate a linear-light workflow. The process was not exactly self-explanatory though, so I wrote a series of articles (1, 2, 3, 4, 5, 6) on how one might go about it.
Then I rambled endlessly on fxguide about this, and in the processes managed to cast a geek spell on Mike Seymour and John Montgomery, who republished my articles on their fine site with my blessing.
This week Mike interviewed Håkan “MasterZap” Andersson of Mental Images about the state of linear workflows today on that same fxguide podcast.
Which is so very awesome, because I don’t want to talk about it anymore.
It’s just no fun going around telling people “Oh, so you put one layer over another in After Effects? Yeah, you’re doing it wrong.” Or “Oh, you launched your 3D application and rendered a teapot? Yeah, you’re totally doing it wrong.”
You are doing it wrong. And I spent a good few years trying to explain why. But now I don’t have to, because Mike and MasterZap had a great conversation about it, and nailed it, and despite the nice things they said about ProLost you should listen to their chat instead of reading my crusty old posts on the subject.
Because it has gotten much, much simpler since then.
For example, there’s Nuke. Nuke makes it hard to do anything but work in linear color space. Brings a tear to my eye.
And the color management stuff in After Effects has matured to the point that its nearly usable by mortal men.
Since I’ve seen a lot of recent traffic on those crusty old posts, here’s my linear-light swan song: a super brief update on how to do it in AE CS4:
In your Project Settings, select 32 bpc mode, choose sRGB as your color space, and check the Linearize Working Space option:
When importing video footage, use the Color Management tab in Interpret Footage > Main to assign a color profile of sRGB to your footage:
Composite your brains out (not pictured).
When rendering to a video-space file, use the Color Management tab in your Output Module Settings to convert to sRGB on render:
That’s the how. For the why, well, those crusty old articles could possibly help with that, especially this one on color correction in linear float, and this one on when not to use linear color space. Part 6 is still pretty much accurate in describing how to extract a linear HDR image from a single raw file using Adobe Camera Raw importer in After Effects, and this article covers the basics fairly well, although you should ignore all the specifics about the Cineon Emulation mode, which never should have existed. This little bit of evangelism is still a good read.
But the ultimate why answer is, and has been for a while now, within the pages of a book you should have anyway: Adobe After Effects CS4 Visual Effects and Compositing Studio Techniques (deep breath). Brendan Bolles guest-authored a chapter on linear light workflow, and not only does he explain it well, he gives many visual examples. And unlike me, Mark keeps his book up-to-date, so Brendan’s evergreen concepts are linked directly to the recent innovations in After Effects’s color manglement.
OK, that’s it. Let us never speak of this again.
Stu
I can’t believe I wrote all that without once linking to the official Adobe whitepaper on Color Managed workflows in Adobe After Effects CS4. It’s a great write-up, super detailed, and accurate in every way. Further proof that my work here is done.
Reader Comments (43)
Sorry Stu, I but I have to ask a why.
Why interpret footage as sRGB? Shouldn't HD sources be marked as REC709 and PAL and NTSC in their respective color spaces?
speak of what?
Man, I've been doing it by using the Linear sRGB color profile that was posted on the fxguide articles a long time ago, then checking Preserve RGB on every single piece of interpreted footage ( full 32bit EXRs and jpegs alike ) then setting up a custom Color Output Simulation that would Convert to Output Profile: Working Space and reinterpret as Preserve RGB (checked) Simulation Profile: sRGB 2.1 Convert to Monitor profile: sRGB 2.1 - then when I would ever render anything, on the output module Color Management tab, I'd always have to check Preserve RGB. The results actually look pretty good, but the process was very wrong. I just wonder if the math was completely wrong behind the scenes :/
Follow up - So i switched all of my footage interpretations from Preserve RGB to sRGB and the Working Space from Linear sRGb to just regular sRGB. Now with Display Color Magnagement off, everything is too dark. With DCM on, but set to say, HDTV Rec 709, it is way too washed out. I'm not sure which Dispay Color Management setup to use w/in AE ...
This is insane. Totally complicated... AND you never know what is doing WHAT to your image? It's not US doing anything wrong...The software and the hardware people need to get it right and STANDARDIZE the pipeline. Even if I give someone the exact specs of software and hardware I use in production, even the experts on this cannot tell me what to do..... so WTF? I keep working so hard to make pretty images, and something, somewhere f#@$'s it up anyway. What I get is some of my stuff looks great out there, some sucks... I keep good looking stuff for the reel... and we just wait til THEY get it right.
DaNi, the answer is not simple. There are some caveats about Rec709, for example — in short, its transfer function is one thing whereas its assumed display function is another. sRGB is actually a good color profile to use for Rec709.
The larger point though is that this is workflow is not trying to address the vagaries of mixing and blending different color spaces in a "correct" way. This workflow is about getting more photographic results from basic operations like over, add and multiply. And for that, doing it the above way is plenty good for any video source.
Hey Stu,
Been following your linear workflow articles for awhile now. Great work and I have implemented a pretty strong workflow in our facility. I was wondering if you had an answer for this. I use both Nuke and AE. Nuke is great as you said and is my predominate tool. I'm trying to figure out how to emulate how Nuke converts Cineon LOG files in After Effects using Color Profiles. All the FUJI and KODAK profiles don't seem to do exactly what I want. I basically want the ELIN LOG to LIN cineon converter plugin but in a color profile. Does that make sense?
Jason
Colin,
You're doing it wrong. :)
When you select "Preserve RGB," you're saying "don't convert my pixels, leave them as-is," which is exactly the opposite of what you want to do. You want to convert your gamma-encoded sources to the linear working space.
You can select "Preserve RGB" for sources that are already linear, as your EXRs should be. But you won't have to because AE will assume EXRs are linear.
Leave Display Color Management on. This whole operation hinges on a display LUT or correction that allows you to view your linear images on your non-linear monitor.
Try following the workflow outlined in the post. The first test of whether it's working is, can you round-trip your source footage safely? I.e. can you render your footage back to disk and have it match the original source exactly?
Then the second test is, do basic operations like a soft-edged Over look different than when you don't use any color management? The whole point of this is to see some benefit of blending colors in linear light. That benefit should come in the form of better-looking motion blur, edges, add-mode mixing, etc.
What you were doing was setting a linear Project Working Space and then completely sidestepping it by checking all those "Preserve RGB" boxes.
Matt,
It gets so much worse.
If you do all this stuff correctly, everything you ever knew about compositing will break. You'll have to stop using some of your favorite blend modes. You'll have to convert footage back to vid or log to do basic things like key or track. You'll constantly be turning your view LUT on and off for various purposes. You'll make mistakes and accidentally send your client a take that's missing it's LUT, or double-LUTed.
You can't have it both ways. You can't trust the software to just do all this for you and at the same time feel that you're an informed participant in the process. Some systems err on the side of forcing you to do all this manually. The upside of that is that you always know exactly what's happening. Some systems, like Nuke and AE, have some gamma management assumptions built-in to their import, export, and viewer. The downside of that is that it's a bit hidden from the user.
Take you pick, but yes, math is hard Barbie.
I should add though that if you don't want to go full-bore with this, but you still want to see some of the benefits of using linear-light blending in After Effects, just leave Color Management off altogether and check "Blend Colors Using 1.0 Gamma" in Project Settings.
Hi Stu.. a few questions:
1- In "Simulate Output" in this workflow. you let it off (with display management on), right?
2- What do you think of "Blend Colors Using 1.0 Gamma" option in AE CS4?
It linearize everything and transform it back? Is it right to use it?
3- Cineon files: I cannot make it work with cineon files yet with this new approach of LWF. What do I need to do to make a cineon look right inside this linear workflow?
I put a cineon converter (Log to lin.. highlith rolloff in 0) but it looks washout?
Which is your method if you import a cineon file to make it look right in linear AE?
Stu.. you make difficult stuff so simple!
Thanks for your hard work, you are my "obi-wan-kenobi"
Javier
Hi Javier,
1. That's correct.
2. This option does not convert your pixels, so all your pixel values remain as they were in the source files, but it does invisibly convert them to linear and back for each blending operation. You get some, but not all, of the advantages of linear this way. Your Over, Add and Multiply operations will benefit from it, but things like motion blur, lens blur, and other pixel operations will not be performed in linear color space.
3. This post does not cover Cineon, For a Cineon workflow, you need to make a decision on both how you will convert your log DPX files to and from linear, and how you will view the results. After Effects does support all this very well, but it's more complicated than the video-focused workflow I describe here.
For an absolutely terrific description of all this stuff, including a Cineon workflow, see the official Adobe Color Management whitepaper:
http://www.adobe.com/devnet/aftereffects/articles/color_management_workflow.html
I can't believe I forgot to link to that in the main post — I'm going to fix that now.
I'm following the workflow in the article ( thanks again! ) - and I passed the round trip test, but with Use Display Color Management on ( w/ no output simulation ) - all of the blending looks worse ( flat out wrong ) to me. As if its being double corrected... If I turn it off the over exposed bleeds, blooms look much more natural. It could very well be that my eye is just broken after all of these years in the wrong world :/ I'll just keep turning knobs I guess.
Hard to help you troubleshoot that one without seeing what you;re seeing.
Thanks Stu for your answers!
This post of yours come to me as from heaven. Because I spent tonight a lot of hours researching about this topic. There's a lot of contradiction in the subject. So this post helped me a lot!
Thanks again. You are like a mentor to me and I guess to everyone that follow this cool blog.
You still are the linear knight for me me Stu. Hope you don't mind fill my knowledge gaps.
1.
I just wonder what your Fusion eLin tools did? Something Fusion wasn't capable back then? Are they redundant with the Color Gamut tools in Fusion or do they have some extra magic.
2.
Our eyes don't see the world in linear terms right? But this can be ignored for the fact of the profile and output device / screen? Simply because of the relative low contrast of screens or is it that the screen outputs something compensated for our perception?
3.
Also I wonder how to know which image formats are linear, which have gamma applied by definition and which are arbitrary or store the gamma (independent if it is wrong or right)?
Has JPEG always the profile applied?
Is EXR always linear?
What up with PNGs Gamma Tag or Quicktime?
Is there something like a database for various workflows?
I hope Master Zap will continue with the website.
Cheers
Blazej
Speaking as one of the folks who has been looking at older posts, it's great to see this topic revisited.
I still don't quite understand it fully, and probably won't until I've read Adobe's white paper back and forth four times. I'm pretty sure we will be speaking of this again, if only after CS5 comes along and affects this workflow.
Blazej,
1. The Fusion eLin tools converted images between vid, log, and lin spaces. They were designed before the Gamut tool was created. I think the Gamut tool may do everything you need for a video workflow, but you'd need to use Fusion's Cineon log/lin tools for a film workflow.
2. Correct. We perceive a greater difference between similar dark values than between equally similar bright values. This is why an 18% gray card looks like 50% to our eye.
Our video screens corroborate this. If we display a 50% value on a gamma 2,2 display, the screen actually displays it to us at 18%, and to our non-linear eyes it appears as 50% (approximate values).
3. Of the formats you listed, all are vid except EXR, which should always be linear. Generally speaking, floating-point file formats are linear, integer file formats are vid or log.
Well, I kept playing with my current project, and eventually came to the conclusion that after adopting your workflow everything is right, its just i need to get used to compositing slightly differently. It was the weird things like a lot of my fast blurred layers did not have a smooth falloff, it was a very harsh, abrupt edge... that and a few other random unexpected results. When I'd put an adjustment layer w/ a gamma reduction of .45 it felt "right" to me. I just couldn't help but feel like something was technically wrong at first, even though nothing was.
So I spent some time to change most of the composite so it looked as close as I could get to the way it was before, and all those nuances that bothered me at first, ended up making for a much nicer image overall. So thanks again for setting me straight!
One last question though, when using Magic Bullet Looks w/ a full linear workflow like this, I need to set the input and output gamma to 1.0 and turn off sRGB and Setup correct? There doesn't seem to be a preview lut, so I temporarily set the output to 2.2, make my adjustments, and then set it back to 1.0 before exiting the plug-in... is that correct? If I left it at 2.2 or whatever the default is, it would completely blow out my scene and create all sorts of artifacts. Anyway, thanks again!
Thanks so much for clarifying all this, even though it would actually be Adobe's job to do so. However, I have got a bunch of questions as well:
1) Based on your response to Matt's post, am I right in assuming that keying and tracking should always be done in vid/log space? My experience is that converting heavily compressed formats to lin and then performing a key can in some instances magically eliminate edge fringing problems that would otherwise be very hard to resolve. I usually use whatever works better, but I'm just wondering what is correct from a mathemathical point of view.
2) What's the deal with Photoshop? In the color management settings, there is an option to "Blend colors using gamma …". Will a gamma of 1.0 result in correct blending? Is it equivalent to "Linearize Working Space" in AE? Is that setting saved with the document and what happens to files that were created with different settings/on different computers? How does it relate to the "Color Matching" and "Compositing" settings under "Preferences > Performance > GPU Settings > Advanced …" in CS4? What happens if I import such a document into a film compositing application with layers intact? Sorry for asking so many questions at once …
Stu, 1024 raised to N:th power thank you for this post, I am humbled, but as you see, you ended up with all the questions anyway, poor guy; LOL :)
It's quite common that the 1st step into linear is "everything looks wrong".
The 1st step is to remove all the hacks you did to make it "pretty". Any "screen" blend mode has got to go at all cost. Frankly, you should be comping layers in "over" or "add" 99.9% of the time. Generally, you need to turn stuff down a lot. A CG light, for example, that looks "halfdark" in a classic workflow (i.e. an intensity of "0.2) can look quite bright in linear, where you find yourself counting "0.02" as dark instead, for example.
The 2nd more longterm step is to readjust your eyes. (After a while in linear, you will be saying exactly the reverse, and "see" a nonlinear rendering a mile away.)
We are so used to seeing simple things like a lambertian shaded sphere displayed wrong for 25-30 years, so when you see it linear, you say "hey wait the falloff looks harsh and the lighting flat". I invite people who say that to go into a closet with a pingpong ball and a flashlite. They either come out "enlightened" (pun intended) with a changed mind... and if not... at least you know you have them safely locked away in the closet :)
/Z
Actually I don't think we're out of the woods yet. First of all that "sRGB" color profile desaturates all my images. So I don't want it anywhere near footage with a 10 foot pole.
Secondly AE still assumes alpha channels are gamma corrected when loading a gamma corrected TIF.
So if you have a white plane with 50% opacity and enable gamma correct in say... 3ds max and output a tif you'll get RGBA values of: 0.72, 0.72, 0.72, 0.5 respectively. Which is correct. The alpha channel is a data channel it shouldn't be gamma corrected. Now load that same tif into AE and lo and behold your image which should be 0.72 is now 0.5.
So if you load a linear image you get screwed by the wonky sRGB color profile which desaturates everything. If you gamma correct your footage it assumes your alpha channels are also gamma corrected.
I don't think we've really made much progress except when working with 32bit exrs.
Gavin, the sRGB color profile should not "desaturate" anything. If you work in linear light and use sRGB as the vid-lin-vid transform, everything should look normal to your eye while working. If not, guess what? You're doing it wrong.
Peter,
First of all, Adobe has clarified all this very well. Did you see the link to their color management whitepaper? It's wonderful. Also, I think they should get a ton of credit for the informative descriptions of each of the options in the color management areas of After Effects.
Now, on to your questions:
1. Most keyers are designed to generate their mattes from vid or log sources. But to your point, the composite will usually look better performed in linear. So extract mattes from log or vid sources, reunite them with linear RGB, and comp in linear for best results.
2. I do not know what the deal with Photoshop is, but here are a few things I do know. That linear blending checkbox is weird and unreliable. But you can work with a linear profile in Photoshop. And if you use Photoshop's 32bpc (float) mode, you are forced to use a linear profile. Any profile you assign in float mode will have its gamma set to 1.0 for you without warning.
Stu...
WOW.. the adobe white paper was great and very easy to understand also!
They use HDTV profile instead sRGB.. but I will use sRGB as you said.
I trust more in your experience than in Adobe.
For Cineon workflow they use the Universal Camera Film Printing Density (linear)..
But I did this based on your Linear Color Workflow AE7, part 6:
1- Working space: srgb (linear)
2- Cineon files imported as Kodak 5218
3- For Simulate how it will look in theatre: I put in simulate out: 5218 to Kodak 2383
4- Render everything to cineon (output workspace: Kodak 5218..)
Then I imported what I renderd and was perfect! film pipeline! Very happy.
In the white paper they use "Universal Camera Film Printing.." as workspace. I tried it also.. changing the workspace and the imported cineon file. The output was the same! It only changes a little how you see the cineon files in the monitor. but just a little.
I realized that After Effects CS4 made so easy everything now! I can change the output profile without adjustments layers on top of everything! (as you had done in AE7 tutorials)
Something funny are the color pickers.. they change in linear space.. but if I use de default system color picker in AE.. it remains not linear. So I guess I will have to get used to the AE color picker to be able to see colors as they are! :)
Stu.. your are my mentor! Thanks for everything you do all the time to help.
Really appreciate!
Hey Javier,
I'd follow the whitepaper instead of my crusty old post. The Universal Camera Film Printing Density profile was developed by Adobe in direct response to use testing at The Orphanage. It's what I use now.
WOW. Thanks for your ultrafast reply! I will use it too!
Stu, thanks so much for your answers, that was incredibly helpful.
Sorry, my bad, I probably should have refreshed my browser cache … That whitepaper is really excellent. And now I finally have official proof to back me up when I tell people not to use ProPhoto RGB or Wide Gamut in 8-Bit.
Gavin:
> So if you have a white plane with 50% opacity and enable gamma correct in say... 3ds max and output a tif you'll get RGBA values of: 0.72, 0.72, 0.72, 0.5 respectively. Which is correct. <
Actually, that's not correct, assuming we're talking about a premultiplied image. It sounds like 3DSMax is applying a gamma correction the image after premultiplication. Which is wrong. (A premultipled "white" image in any colorspace should have rgb = a)
also, have a look at Steve Wright's article on how Nuke handles all this stuff:
http://www.swdfx.com/PDF/Nuke_Color_Management_Wright.pdf
Actually, you are wrong. First of all, premultiplication is not something that is "performed", it is the natural outcome of a renderer. The alpha channel is information about how much of the background should be seen in the pixel, nothing else.
RGB=A is not a requirement for *real* premultipled alpha files at all. Not only should the gamma *never* be applied to alpha (thats insane) but a color with RGB data and zero alpha is completely legal (it will be composited additively). This is why premultiplied is the "one true way" when it comes to Alpha, and non-premultiplied (aka "straight") is an abomination. :)
/Z
Zap is correct, and this is a major downside of After Effects: it does not properly support premultiplied images where RGB > A.
After Effects does not apply gamma correction to alpha, not sure where that info is coming from. It does make an assumption that premultiplication has happened in the working space. I was also not trying to imply that c > a is illegal.
In AE all colorspace conversions (such as from source to PWS) are performed on unmatted color. This is basically a requirement for any transformation where the color primaries don't match. I'll leave this part as an exercise for the reader.
Zap, I'd think it would be interesting to discuss further in some less space-constrained place. As a counter example: would you apply a (subjective) color correction to a premultiplied image? What about a applying a gamma correction to a transparent white value composited over a solid white value? (Shouldn't that stay white in any colorspace, because gamma isn't supposed to modify the white values?) etc.
If you're interested in the math, I suggest taking a look at the Adobe transparency model formulation here:
http://www.adobe.com/devnet/pdf/pdf_reference_archive.html
(aside: c > a does work as you would expect in AE in floating point, except for the obvious case when a = 0)
well, i am having some sort of brain burp tonight -- i misinterpreted the c > a result above as being a gamma correction applied to the result of premul over premul, instead of a linear-light composite of premul over premul. apologies for the confusion.
hello,
thank you for being so open about the post process.
this article is very interesting, i have been looking for the reason, and a fix for the color in difference in FCP and AE, up until now it was Premiere.
moving on, if i do follow all of these instructions, what should my monitor settings be? (i am on a mac). insofar as should i be running my monitor at 1.8 or 2.2 ?
thank you
good day.
Quick question y'all... I'm rocking the new workflow, but now, I've noticed that w/ Display Color Management on, I tend to get pretty bad banding during ram previews. Stop the preview, everything is great. It can make things difficult to finalize, tweak. Is there anyway to prevent this or is it just me?
Colin, here's something I do with complex projects (read: all of them): separate what I'm critiquing. RAM preview is a quick and convenient way to check yourself, but as a final indicator of your result, you can't beat a render. Since I can't rerender the full comp every time I want to check a dozen pixels, I try to compartmentalize what I'm checking on.
If I'm checking the timing and motion of elements in a comp, I set resolution to half or lower if necessary, hit RAM preview, and try to concentrate on how things are moving. If I want to check how elements look, I usually will slog through three or four still frames at full resolution, or render out a still in the render queue. Finally, to see a combination of the two, I trim the working area to a narrow range of critical frames and render a sequence at half-res, or full res if I have the time.
I'd love to hear how anyone else handles it, though.
What's funny is that I still use some of your old posts as reference when I'm trying to contextualize linear workflow for junior artists in Fusion. It's written coherently and provides a good base before we start having to have *that* conversation.
Stu, looking at YUV to RGB a little wider than how AE handles it. What do you say about the way some video conversion software handles YUV to RGB by scaling 16-235 to 0-255. I have your DV Rebel book and love it and refer to your reclaiming the 10%.
Bottom line should the YUV to RGB conversion and then to linear be done without the scaling?
I'm on Linux and when you look at the open source video apps they all seem to use FFMPEG as a base and I can't get a non scaled YUV-RGB out of FFMPEG infact after it's scaled I can't adjust the levels and reclaim anything it's all been clipped and thrown away.
Any thoughts? Maybe I should ask Euginea> ;-)
re last post.
In AE's Colour Management Workflow pdf page 27 discusses Limited Range Video profiles that AE provides:
"In many cases, recorded analog video limits the range of tone in a captured image
to “video safe” levels of luminance (normally IRE values from 7.5 to 100 in the US). However, it may be the case that luminance levels are recorded below 7.5 IRE (super black) or above 100 IRE level (superwhite or overbright). This presents a problem."
"Since After Effects works with RGB values, there will be an additional translation from YCbCr to RGB before work can be done in After Effects. The translation of colors from YCbCr to RGB is normally done by the codec used for a specific video format. This codec may translate the luma values to limited range RGB (16-235) or the codec may translate luma values to full range RGB (0-255). In some cases, a codec used in the workflow may have controls for how it will interpret luma values. In other cases, this translation is hidden from the user. You may be working with footage from several different formats that treat luma levels in different ways. This could add additional time and cost to your project as you manually tweak each piece of footage to comply with the requirements of your output."
"The Adobe-supplied “16-235” profiles address this issue and provide additional flexibility in workflows where control of luma expansion/compression is critical."
And on...
For linearized compositing and CC should these profiles be ignored or adopted?
Really informative post. And comments had raise and solved many other aspects of the tutorial given here. I really found it informative and interesting..
VFX course
I'm working with the linear workflow and it's really awesome. I'm having a small problem pop up though. It is the same problem that Colin was experiencing and was wondering if anyone has experienced it as well.
My workspace looks great, as do my final output rendered tiff sequences, but the ram previews I do while I'm working look like garbage. There is a terrible amount of color banding occuring. Has anyone had this happen to them?
So please tell me if this is correct, because I think Im confusing myself at render time. I rendered out linear exr's from Toxik/Composite and imported then into AE with the necessary project settings and color management. Then do all my gamma correction, adjustments, and ready to final output. (Which is DVD, and yes, it will be encoded in something other then AE). Right before its time to render the image, you then change the linear color profile into a regular profile. Wouldn't that then be double gamma'd, since I adjust that in the color correction process?
@Colin and @Andrew
I was having the same issues and have found a solution:
Under your comp windows click on icon for Adaptive Resolution, OpenGL and so forth. Click the last option "Fast preview preferences". In there set your Viewer Quality on both options to 'More Accurate'. BAM