YCbCr --> RGB conversion tips

[Redsandro]

I haven't used zebra 70, but I was going to test that also. Using it as a guide should theoretically leave a lot of headroom. But maybe it's TOO low?

I think, theoretically and ideally in consumer-type situations, skin-tones are about 70% up on the scale of brightness. You would just have to adjust the shutter 'till you get zebra's for people. Just a thought, though.


About your work:
Is there actually an option for choosing RGB 16 as black (and 235 for white) while rendering cg (for compositing)? I occasionally do 3d rendering and compositing, too. I always had to brightness-adjust the cg in post, and actually I didn't even realise why (ignorant me). (I only have experience with 3d studio max though)

[lordtangent]

About your work:
Is there actually an option for choosing RGB 16 as black (and 235 for white) while rendering cg (for compositing)? I occasionally do 3d rendering and compositing, too. I always had to brightness-adjust the cg in post, and actually I didn't even realise why (ignorant me). (I only have experience with 3d studio max though)

No package I've ever used seems to know about it in the native renderer at least. There are "video legal" filters some packages will let you run as a post process though. Most packages seem to know about gamma at least. What I've always done is deal with the issue in the comp because the issue really has more to do with how the plates were acquired/ingested rather than how you generate the 3D stuff. Now that I work in a departmentalized studio, I don't capture the stuff myself. It's delivered to me from some technician or post house that grabs it from either film or video. Sometimes black is at 0,0,0, some times not. It's a total case-by-case issue and I use my eye balls and a color picker to get things right. Very low tech really...

But that doesn't excuse me from knowing the technical stuff regarding where the images came from. As we've already seen, the "video range" issue is a potential trip-up. And it gets even worse with film scans, which are usually delivered as LOG encoded files. You can really screw yourself if you handle the color in those incorrectly.

[Redsandro]

I was trying to figure some stuff out today, but I cannot seem to get frames stored as YUV (like a weird looking TIFF or whatever).

Does anyone know a method of converting a MPEG stream to an image sequence in YUV, exactly as it is inside the MPEG?

[Redsandro]

@lordtangent,

About the ranges, it looks like you're right when I use avisynth, but not when I use Avid. I want to do some 1:1 comparisons, but avisynth messes too much with the chroma to make the comparison possible. I use LoadPlugin("MPEGDecoder.dll") with an .MPG captured with Adobe Premiere. See the attachment, the green in the top and the purple along the lines of my 'test stone' are most visibly 'off'.

What plugin do you use to read mpeg without messy conversion? Does DirectShowSource() keep the stream in YUV space?

[Redsandro]

DGDecode_mpeg2source() does the trick! MPeg is parsed fine now.
Still looking for conversion to true greyscale YUV images.

[lordtangent]

I was trying to figure some stuff out today, but I cannot seem to get frames stored as YUV (like a weird looking TIFF or whatever).

Does anyone know a method of converting a MPEG stream to an image sequence in YUV, exactly as it is inside the MPEG?

ffmpeg can save out a raw YUV stream. I think MEncoder can do it also. Another thing you might try is writing an AVISynth script that just moves each channel (Y, Cb, and Cr) to it's own channel and then saves it out to it's own file. (one file for each channel) I'm pretty sure it can do that but I haven't tryed it myself. Check out the AVISynth Wiki. Someone has probably already done a script for it.

It might also be possible to split out each channel and then transform/combine them all into one big greyscale page. (then you could view it in a normal graphics program) That should work with 4:2:0 since each chroma channel is exactly 1/4 the res of the luma.

But if you are go straight out Lagarith 4:2:0 there is no RGB conversion so I doubt you'll see much difference from one decoder to the next.

UPDATE:

I couldn't stop thinking about this since it's something I wanted to do in the past myself to help visualize YCbCr. So... I went into AviSynth and figured out some code to do it. I'm presenting it below. The code snippit is configured as a DGindex template (What HV20pulldown.exe uses) If you use this code along with vDubMod you can save the output movie as a PNG sequence. It's possible to save a single frame also. Or you can use one of the HV20pulldown.exe output templates and save to a movie. The chroma channels are padded with solid nothing so even though the channels are technically "full" it should still compress pretty well in Lagarith.

### Lordtangents HV20 pull-up and tile YCbCr template
### v0.1 October 4 2007
### 
### Requires: DGDecode.dll, TIVTC.dll
### http://neuron2.net/dgmpgdec/dgmpgdec.html
### http://web.missouri.edu/~kes25c

##### MAIN -- Do not edit unless you really know what you are doing!

v=MPEG2Source("__vid__")
a=MPASource("__aud__")
audiodub(v,a)
tfm(d2v="__vid__")

YUV=tdecimate()

Y=Tweak(YUV, sat=0)
U=UToY(YUV)
V=VToY(YUV)

StackVertical(Y, StackHorizontal(U, V))

YCbCr_tiles.png
xvid avi http://hv20.info/yopu/tiled_YCbCr.avi
png codec mov http://hv20.info/yopu/tiled_YCbCr.mov

[lordtangent]

@lordtangent,

What plugin do you use to read mpeg without messy conversion? Does DirectShowSource() keep the stream in YUV space?

A workflow I'm testing now involves keeping everything in YCbCr as long as possible. (Instead of converting to RGB early) When removing the 3:2 pulldown, I transcode to Lagarith (lossless codec), which supports 4:2:0 YCbCr natively. The codec is lossless so there is no quality hit in that operation. It doesn't save me from having to go though RGB when I do my edit in Premiere (Premiere only does RGB), but at least it keeps everything in YCbCr on disk and differs the color space conversion until the last possible moment. It also saves a alot of disk space compared to storing the transcoded stuff in RGB.

Considering that I almost always apply some sort of color correction anyway, going though RGB is usually inevitable. I'm mostly just concerned with keeping all the dynamic range in the conversion. Something I'm still testing is how Premiere / Direct Show handle the conversion. I may end up returning to my "Convert to RGB early" methedology if I find the delayed conversion to RGB is messing up my color more.

[Redsandro]

Thanks very much for figuring this out! I wanted a way to see EXACTLY what is in the source, not with some conversion interpreting stuff for me. This is very usefull.

I will continue figuring stuff out as I just want to know details that will help me make quality improving decisions. When I have the time.

For now I'm wondering about storage. CbCr seems thin, if you know what I mean. I'd guess its average range is 5 bits. Ofcourse the formula's are chosen wisely to keep chroma information as thin as possible, making entropy encoding really 'efficient'. So how many bits can chroma channels actually hold (I'm assuming 8) and how many bits will it hold in reality? I've not come across footage with the chroma channels having a spread range.

Wikipedia tells us some stuff we already know, including the RGB range after conversion. But I'm having trouble finding information about the actual common chroma channel range, and therefore the theoretical range of resulting RGB channels. You've already proven the theoretical range is bigger than the common range.

Just a simple thought.. although the exact formula's are too steep for me, luma is what makes the chroma simple (thin). If the chroma is not simple, possible RGB values increase. Where does it end?

I think no more information (RGB range) will secretly come out of the HV20. Or will there? If not, it's nice to know how it works.

[dbx]

i capture with hdv, and open it up in after effects. if after effects or other nle progs dont support 16-255, whats the point? i see no practicality in this if ur end product is always going to be in a restricted color space.

[Nfect]

So, if i understand good, the problem is

input 16-235

but the regular capture maps the 16 to the 0, and the 235 to 255 ?

So now, what if I open after effect (premiere,shake,speedgrade,...) and simply add it a "level" control with those entry

input 0-255 (default)
output 16-235
(then save it as a template, nammed like "real input" or something))
then work on it or do whatever i want.

this will fix the problem no (well logicaly)?

0-255


16-235


VS



If you want to print out chroma du monde, and focus here's the link, A3 format, CMYK at 300 dpi

If i made an error about the cam-aligns, please tell me !!

[lordtangent]

@Nfect
Your solution does not fix the actual issue. (Round tripping full range YCbCr though RGB) In that respect, your solution is not logical at all. It ignores the part of the pipeline actually leading to problems...In your example you accept the fact that you've lost the outside 16-235 YCbCr data. OK. But why? Why would you allow that data to just be clipped off and thrown away? (It's not required that it be thrown away) In your example, you still haven't actually taken control of the conversion process. As a result, you don't know for sure how the RGB is going be be converted back to YCbCr anyway. Depending on how stuff is converted BACK, your re-normalization of the RGB data might just make things even worse than they already are. (given your assumptions, you've already lost the full range data... it's long gone. You've made it an issue of damage control ) You are barking up the wrong tree.

From the start what I proposed was a way to KEEP the full-range YCbCr data through the color space conversion process. The solution you should be working on is: How do I round trip the full-range data?

The answer isn't in how you normalize stuff once it's already RGB. The magic, in this case, happens at the YCbCr to RGB conversion stage. Unfortunately, you don't always even have the option of controlling it since not all software exposes those controls. (part of the point of my original post) But once you are aware of the issue you might be able to detect if your software is misbehaving. (Vegas has problems here for sure. And I would not be surprised to learn Premiere is also stupid about this issue.) It's not a small issue either, which is why the fact so many programs get it wrong amazes me. Anyway, there are work arounds (as I mention in my first post) but they are not easy for some people.

[lordtangent]

i capture with hdv, and open it up in after effects. if after effects or other nle progs dont support 16-255, whats the point? i see no practicality in this if ur end product is always going to be in a restricted color space.

Huh?? You must not have read the post, or you mis-understood it. The point is, they DO support it. But the color space conversion is often WRONG (which when it is) throws away color that is present in the HVD. And to assert that AE or other NLEs don't support the full range is just silly. It's just 8 bit data. Nothing magic there.

Re-read the initial post and look at the photos. If the issue isn't clear then ask questions rather than throwing rocks. Just because you don't understand something doesn't mean it has no "practicality". You should pick your words more carefully.

[dbx]

no one is throwing rocks here, im simply stating that if i cant get the right formula calculated in after effects or any mainstream program, what does this do for people? i appreciated your results and findings, but if it cant be applied to say AE, what is there to do? <- that is all honest questioning, im not trying to be an asshole like you probably think i am heh

[lordtangent]

For now I'm wondering about storage. CbCr seems thin, if you know what I mean. I'd guess its average range is 5 bits. Ofcourse the formula's are chosen wisely to keep chroma information as thin as possible, making entropy encoding really 'efficient'.

Well, it might look "thin" but in reality, the bulk of the data is probably actually in the "high" bits. Meaning the channels may actually have more precision than it looks. As you mentioned, the reason the channels aren't normalized better is a result of the formula used to generate them. They are calculated as the DIFFERENCE from luma, which means they will probably never be full range themselves. (and if you think about it, part of their "precision" is encoded in the Luma as a result) It's a really clever system when you think about it. (Basically descended from the invention of color analog TV, so it's much older than digital!)

[Nfect]

Hi Lordtangent,

I'm trying in no way to break your idea to get everything right, I'm just having some questions to understand it right, and how to get something right? by improving it.

So I kind of experimented what you explained, took your pictures in photoshop and it seems like your right, if I scale down from 0-255 to 16-235, i can see the whites (very bright) becomming grayish, but the blacks stays the same with a verry small loss of information in details. I also noticed a loss of saturation... But in the other hand, it has something that came out "weirdly", it's that your brightest white on the 16-235 picture is way over the 235 (253,5) but the black still upper than 16.

So what you are meaning, is that even if the normal range is from 16-235, it can still contain information over the boundaries ? Well, anyway, thanks for those informations because YES it's verry important and usefull to get an headroom in editing. Please clarify about the "extra range" !!

Thanks
Nick

Oh ok, I did reread the whole thread from the begining, and found those "extra range to 255", well, thanks, great work, and yeah, you get my vote for a sticky too !! Good job

[lordtangent]

no one is throwing rocks here, im simply stating that if i cant get the right formula calculated in after effects or any mainstream program, what does this do for people? i appreciated your results and findings, but if it cant be applied to say AE, what is there to do? <- that is all honest questioning, im not trying to be an asshole like you probably think i am heh

And, not trying to be a wise ass in answer... not all mainstream packages are created equally. This is very much a mainstream issue though. It's not unreasonable to expect proper results from any mainstream package. But you need to know if your package is behaving correctly to begin with. If you are unaware of the potential issue, how can you even know if there is a problem? (Part of the point of my initial post) Maybe your mainstream package is doing the right thing or maybe not. This isn't something the vendors advertise. (Considering how many of them seem to get it wrong, why would they?) So, you need to know the potential problem and evaluate for yourself.

Almost all video is YCbCr and many NLEs are still RGB and all compositors are RGB. So, the trick is getting any potentially brain dead software to do what you want it to do. ( I learned as a pro, you can't make excuses when your software doesn't do a particular thing. You have to figure out how. Clients/bosses don't care about "software" they want results ) Sometimes that means actually knowing the technical stuff, even if the programmers that made the software didn't.

The workflow I outlined can be applied to any package that can read 8 bit images. You might just have to convert the video externally if said package is too lame to do the conversion correctly itself. I haven't tested AE so I don't know if it's one of those packages that does it wrong. I do know it is RGB only, so we know for a fact the conversion must be happening somewhere. Not sure which mpeg decoder they are using. Maybe it's wrong, maybe not.

So, you need to test. Usually the color space conversion happens in the codec, which means even a well regarded mainstream package may be at the mercy of a badly written codec.

Part of the reason I put my post in this section was because the tools I used the take control of the YCbCr to RGB conversion are the same tools used by the free 3:2 pull down removal pipeline. The other reason is (at present) you need to process all the video to remove the pulldown before editing ANYWAY. So, the two work hand in hand really well together without throwing much extra complexity into the mix. Actually, there is no difference in workflow at all, just a slighting tweaked AVISynth script.

Nothing at all changes down stream of that so the approach works with any NLE or compositor that can load the AVIs from the pull down removal system. I encourage you to test your software and see if it interprets the YCbCr stuff correctly before bothering. Maybe it does. (though, most likely it doesn't )

[lordtangent]

@Nfect

Yes, there is good info in the range outside 16-235 of the HDV file. To get at it, you need to make sure the YCbCr is converted to RGB "full range". That's it!

Then when/if you go back to YCbCr, you do the same thing. Full range conversion. It's still a little lossy because the color space conversion has some rounding error no matter what. But at least you aren't literally clipping the ends off. Ouch!

[Nfect]

Definitly good advice, already testing !

Thanks again
Nick

[dbx]

after spending about 4 hours playing in after effects with footage from the hv20, ive discovered the AE has a plethora of color profiles which are much much better than the standard RGB translation made in 1998. For example, http://www.luminous-landscape.com/tu...hoto-rgb.shtml explains how prophoto RGB designed by kodak is a much much larger color space to work in, and from my tests, EXTEMELY better.

[lordtangent]

after spending about 4 hours playing in after effects with footage from the hv20, ive discovered the AE has a plethora of color profiles which are much much better than the standard RGB translation made in 1998.

I understand what you are getting at. And yes, the color profiles can provide good color managed working space. But... color profiles are a mostly separate issue from what we are talking about here. (It's a whole different conversation outside the scope of this thread really). You seem to mis-understand the context of that article. I've read it, and it focuses on a 16 bit work flow, a color depth that can actually get some benefit from such large color spaces. Also, they start from a Camera RAW, which is a deep source with it's own pretty big color space (Not really RGB until is reconstructed from the Bayer pattern) that needs a working color space the computer can use assigned to it. I'm quite familiar with the workflow myself, because it's what I use with the RAWs from my dSLR. And it's a similar workflow for most of the high end single chip d-cinema cameras right now, which I am also studying.

First a quick glossary:

1. Color models: The model of date chosen to represent color data. RGB, YCbCr, HSV, LAB, CMYK, etc.
2. Color Space: Where a color falls in the CIE color chart. It's absolute color. (Meaning, an absolute color in the "REAL WORLD")
3. Color management: Correlating digital data (#1) to absolute (CIE) color values (#2).

I can't totally go into the scope of color spaces and management here. It's a big topic and one where a lot of people get confused. (The casual naming used colloquially to label certain things doesn't help understanding in this area, so I'm going to use the totally technically correct labels for stuff.) I want to make it clear to people who are just picking up the thread here, color management (i.e. correlating RGB data to absolute CIE color) is a TOTALLY separate issue from YCbCr -> RGB color model conversion. Neither color model, RGB nor YCbCr has an explicit CIE color space... and color management is a totally separate issue from their relationship to EACH OTHER. Both color models are simple containers for data. They have a simple mathematical realtionship to each other. (In this case what we have been assuming to be rec709)

If you can't control the conversion from YCbCr to RGB (with full range data) in the first place, you've already gone astray. The point is, without full range conversion you don't have all the data to begin with. And color profiles beyond basic sRGB or monitor profiles aren't going to do you any extra good if you don't have the full range data. In fact, they will probably hurt you even when you DO. (For example, using too large color space like ProPhoto in only 8 bit just "spreads" the color over the limited bits available (same bits, bigger CIE color space) leading to reduced precision and potential artifacts. Doh!... I'm getting into nuts and bolts when I said I wouldn't.... Argh... must resist....

[Ian-T]

lordtangent, I'm not too hip with scripting etc. How do I apply the script you made into my HV20Pulldown.exe program? I just copied and pasted but when I run the program I get a "parse error." Thanks.

Also, have you ever heard of or used JPEG2000? Someone mentioned this codec the other day so i decided to give it a try. I ran one of my M2t files through with this codec to an intermediate file and I could not believe the results. When I blew up a frame of the video I seen none of he usual HDV square blocks. It was as if the picture smoothed out. Also, I don't know if my eyes were deceiving me but the dynamic range of the picture lookd better than the original M2t fle. Is this possible? I believe they call this a wavelet codec (??) Anyways...it supports 4:2:0, 4:2:2 and 4:4:4. i used an application called Morgan JPEG2000. I bring this up because I am wondering if this does the same thing that you have been discussing here. (of course your method is free and this you gotta pay for....but ...).

[dbx]

ah i see what u mean now tangent. hrm, i wish i could see if ae infact does import footage and converts it properly by doing the color value thing your talking about (255 instead of 235 for whites etc) but i wouldnt know where to start.

[Redsandro]

if after effects or other nle progs dont support 16-255, whats the point?

If you want to manipulate the colors, contrast, brightness, stuff like that, you will have improved quality if you do all that in a workflow that keeps the extra data present. At the very end, you will scale it back to 16-235.

Well, it might look "thin" but in reality, the bulk of the data is probably actually in the "high" bits. Meaning the channels may actually have more precision than it looks.

I think appearances are not as deceiving as one might think. Humans are a fool for color, but when shown in black and white we're feeling smart again.



The Cb channel (I keep calling this V) is clearly using 1/4th of it's potential spectrum. I know the calculation is a difference from luma, another smart way to compress the data. Still, theoretically, a lot of the chroma bits could be used to store more luma. But that is getting beside the point. This is about improving quality of the footage we have. I just thought it's interesting to know.

ive discovered the AE has a plethora of color profiles which are much much better than the standard RGB translation made in 1998. For example, http://www.luminous-landscape.com/tu...hoto-rgb.shtml explains how prophoto RGB designed by kodak is a much much larger color space to work in, and from my tests, EXTEMELY better.

I'm not sure, but isn't the color space you are talking about just the way colors are presented on screen? The RGB or YUV values are still the same internally, right?

-EDIT-

I opened this page before I left, and now hours later posted my reply.
I did not notice that people replied already. My mistake! :P

[dbx]

i may be wrong but im convinced AE imports properly

[Redsandro]

i wish i could see if ae infact does import footage and converts it properly by doing the color value thing your talking about (255 instead of 235 for whites etc) but i wouldnt know where to start.

I wanted to check that myself anyway so here you can see that in fact AE does NOT convert properly.





Important note

It is unfair to speak of improper conversion. The world decided that valid color range is 16-235 and I guess the vast majority of all camera's stais within that limit. If all NLE would include head and toe space there would be a smaller gamut available in the end result in cases where footage is in fact 16-235.

However, just ASSUMING we won't ever need to go beyond those values could be concidered ignorant in my opinion. We are lucky the HV20 stores more data than is to be expected.


-EDIT-

If you are watching on LCD, it helps to look onto the screen from down below (because bright colors become more dark and contrasty).