Lossless (or low-loss) trimming

[plnelson]

So I shoot 10 minutes worth of stuff and capture it, full-res, to my PC, where it soaks up about 1.6 Gigabytes on my disk!

But after reviewing it I see there's only a couple of minutes-worth I really want to keep. How do I trim it down to just the good parts without altering the quality? (FWIW I have A.P.Elements 3.0)

MPEG is lossy. So if I read it into A.P. Elements, edit out the parts I don't want and write out a new MPEG, it's like transcoding it and I will introduce loss. (think of it like reading in a JPEG still photo in Photoshop, editing it, and writing out a new JPEG ) But at least in the case of JPEG I can set a high quality factor to minimize the problem. What's the right strategy with MPEG?

(or is it smart enough to write out the unaffected keyframes bit-for-bit?)

[SenorKaffee]

There are several tools that can cut segments from MPEG files.
ProjectX for instance can work with M2TS.

[plnelson]

There are several tools that can cut segments from MPEG files.
ProjectX for instance can work with M2TS.

No doubt, but my question has to do with the fact that MPEG is a lossy format. How losslessly do these tools trim MPEGs?

[SenorKaffee]

ProjectX and similar tools are lossless, they write the original video data to a new file. The only things they (can) change are container format and header information.

Tools that don´t recompress have the limitation that you can only cut GOPs, so you might end up with some extra seconds of video or an incomplete GOP at the beginning and the end.

Some NLEs can also "smart-render" HDV, so they just recompress parts with changed image (fades, color-correction) and the frames that would produce an incomplete GOP. Since I don´t think you should use an NLE for rough cuts, I can´t tell you much about applications that are capable of smart-rendering HDV.

[plnelson]

ProjectX and similar tools are lossless, they write the original video data to a new file. The only things they (can) change are container format and header information.
I went to the ProjectX website and it said it's "not meant for a regular use but for test and learning conditions".

Also it's only supplied as source code (Java). I don't even have a Java compiler and I don't program in Java (I use C++/C#) so I don't want to mess with that.

Doesn't EVERYBODY have to solve this problem (trimming captured content down to just the parts they want to keep)? What's the mainstream solution?

[bluegrass]

I went to the ProjectX website and it said it's "not meant for a regular use but for test and learning conditions".

Also it's only supplied as source code (Java). I don't even have a Java compiler and I don't program in Java (I use C++/C#) so I don't want to mess with that.

Doesn't EVERYBODY have to solve this problem (trimming captured content down to just the parts they want to keep)? What's the mainstream solution?

I can't speak about Adobe Elements 3, but most of us I believe are capturing the mt2 file from our HDV tape in our camera with or RLE and than opening that file with our RLE for trimming, adding titles, and whatever else we want to edit than render the file to whatever format our RLE supports. Hopefully our RLE will support rendering the format that we need. I'm not sure what is lossless and what is lossy, I just go by the resolution I want my output file to be and choose one compatible with what I want to do with it. If you want to get techinical, I guess data written to the tape as an HDV file has already lossed a bunch of data. To my knowledge the only lossless file would be one captured to a raid array or something really smokin' fast via HDMI from the camera in real time.

[Lunchbox]

The "lossiness" here is referring to the quality loss in generation. Say like, you capture the a.m2t file, export to b.m2t, import b.m2t to NLE and export to c.m2t and so on.

Since the HDV uses MPEG2 which is a lossy format, you loss image quality in each generation. This applies to DV-AVI too. I tried importing and exporting the same DV-AVI file for 10 times (with re-compression checked). Then the footgae shows noticeably lots of squareness like those really bad JPEG artifacts. However, during the first or second generation, there isn't any noticably difference.

So the thing is to avoid exporting and importing the same clip espcially in Premiere Pro 2 that it will re-render each frame even if there is no effect applied to the clip.

If you intend to do just trimming frame, it makes sense to use some render-less NLE such as Womble MPEG Wizard.

[plnelson]

I can't speak about Adobe Elements 3, but most of us I believe are capturing the mt2 file from our HDV tape in our camera with or RLE and than opening that file with our RLE for trimming, adding titles, and whatever else we want to edit than render the file to whatever format our RLE supports.

Do you mean NLE?

Anyway most editors do nondestructive editing. For example, I can point A P Elements at an MPEG I just captured from my HV2, and edit and split it all over the place and end up with a 2 minute clip from my 10 minute MPEG, and my original MPEG is still untouched.

So I'm not sure how your response addressed my question.

[plnelson]

If you intend to do just trimming frame, it makes sense to use some render-less NLE such as Womble MPEG Wizard.

I still feel like I'm not any closer to understandinmg how most people handle this issue.

If you capture 10 minutes of video and after reviewing it and considering what you want to do with it, you decise you only want to keep 2 minutes of it to use in future projects, how do you, or how do most videographers deal with this?

Do they just keep the whole thing aound just so they have access to the parts they like?

[Lunchbox]

Check out Womble MPEG Wizard. It's a lossless trimming program you can cut your 10 minutes video down to 2 (to a new file of course). Then you can delete the old file.

[bigjohn1961]

VideoReDo is the program i'm using, it can cut out a part (or parts) and sow them together in a new file whitout altering the file, and that in just seconds.
If you like more editing control, use Ulead Video Studio 10 or something similar that will perform Smartrender.

[bluegrass]

I still feel like I'm not any closer to understandinmg how most people handle this issue.

If you capture 10 minutes of video and after reviewing it and considering what you want to do with it, you decise you only want to keep 2 minutes of it to use in future projects, how do you, or how do most videographers deal with this?

Do they just keep the whole thing aound just so they have access to the parts they like?

I think I'm better understanding your question now. I think I speak for the majority in that most of us never destroy the master tape. I may have one or two tapes that I use over for just screwing around and doing some tests. I'm sure some of us including myself will write off the "10 minute capture" file to a DVD so we don't need to recapture it again. Yes, there is a lot of wasted tape sitting on my shelves, but I think that is the nature of the game here. For one reason or another, I might go back and on a second look, find something that I can use and would regret if I had scrubbed it away.

As to just keeping that "2 minute trimmed" beauty for doing some editing in the future to it, I guess maybe it will not be the equivilent in quality of the same 2 minutes that is contained in the "10 minute master". I guess your question is really "how can I economize my storage and still keep the same quality as the master?"

[plnelson]

I think I'm better understanding your question now. I think I speak for the majority in that most of us never destroy the master tape.

My question is really about the captured files. These tapes are digital so I assume that what's on the tape is bit-for-bit the same as what gets captured. I don't see any reason to keep a tape around after I do a capture any more than I would keep flash memory cards for my DSLR of a photoshoot after I've uploaded the images to a PC. After a photoshoot, once I've uploaded the raw files to a PC and run a backup I reformat the card for the next shoot.

Also the problem with relying in tape as an archive is that tapes don't get backed-up. If you had a fire or theft all those tapes and their content would be lost. Also if you ever switched to a new type of camcorder you wouln't have any way to read the tapes. And magnetic tape itself only has a limited shelf life - one study by 3M found that data tape loses half its gauss in 10 years.

I'm a backup fanatic. I keep an external USB HD attached to my PC at all times and run regular backups. I have two other external HD's - one at my work and one in a safe deposit box and I rotate through these so I always have two offsite backups in addition to the onsite backup attached to my PC in my studio.


I guess your question is really "how can I economize my storage and still keep the same quality as the master?" Right. So in practice how do most videographers achieve that?

[Ian-T]

plnelson...if your hard drive fails then you are hosed. Your right about tape lasting 10 years but to me it is a safer archiving solution to a hard drive (IMHO and to each his own I guess). As far as clipping M2T files and saving ...I don't know if that is widely practiced...but it seems the software mentioned about shoud do the trick. I have never used them......but I am in the same mindframe of what blugrass said about "I might go back and on a second look, find something that I can use and would regret if I had scrubbed it away". I think most people probably keep "all" of their raw footage so that if they have to go back they have some more to play with. I might be wrong but so far I don't see a lot of takers on this topic.

[plnelson]

plnelson...if your hard drive fails then you are hosed.
No I'm not - that's why I do backups. If my HD fails my last backup will be within the last 24 hours. The only people who are hosed when their HD fails are the people who haven't done a recent backup.

(and it's AMAZING how many people don't do regular backups!)


but I am in the same mindframe of what blugrass said about "I might go back and on a second look, find something that I can use and would regret if I had scrubbed it away". All I can say to that is that I'm pretty serious photographer ( pnArt.com (http://www.pnart.com) )with pretty high standards, and in a typical 5-hour model shoot I'll take about 200 frames, and in my first pass I'll select maybe half or LESS of them to read in from the card, and then after reviewing those I'll toss out half of those, etc. In the end maybe 5 shots will make it into my, or my model's portfolio, and another 10 or 15 will be kept for their "potential", and I've NEVER regretted reformatting my CF cards! I think it's just a matter of having good judgement about what has some potential.

[Lunchbox]

Shit happens. It's important to do regular backup. I setup a second computer (a Dell C521 low profile computer.. costs $3xx) in my living room with these functions

- Backup server
- Media server
- PVR recording TV programs
- Encoding station
- web/FTP server
- File server

It's on all the time connecting to my plasma TV running Windows Media Center. It's also a FTP and web server. Then running backup program (Second Copy) to backup my main machine every few hours.

I have Canopus Procoder installed to that machine using the "Watch Folder" feature. Whenever I need to encode MP3, WMV, MPEG, DIVX..... i just need to drag a short cut to those watch folders, then check the output folder later to get the resulting files.

[plnelson]

Shit happens. It's important to do regular backup.

Agreed.

Even though this thread has gone on forever and I've gotten lots of useful suggestions for software (e.g. Womble) , I still don't feel like I know what most videographers' workflow actually is WRT content they don't choose to use in their final product. Would someone please answer this:

You go to an event - a family picnic, your kid's little-league-game, etc, and shoot an hour of footage.

After reviewing it on the HV20 you see about a half-hour of material with "potential" and in the end you boil down the best bits into a 10 minute highlights video.

What do most videographers actually do with the stuff that didn't make the final cut? Dp they only capture the stuff with "potential"? Do they do ALL their review on the HV20 and only capture the parts that end up in the final 10 minute video? Do they leave everything, used and unused , on their PC intact; do they trim it down to the essentials using one of the programs already mentioned; do they throw out the captured MPEGs after making the final video; do they keep a bookshelf of un-backed-up tapes? Since I'm a newbie at this I want to know how experienced videographers who've shot hundreds or even thousands of hours of video do.

[bluegrass]

No I'm not - that's why I do backups. If my HD fails my last backup will be within the last 24 hours. The only people who are hosed when their HD fails are the people who haven't done a recent backup.

(and it's AMAZING how many people don't do regular backups!)

All I can say to that is that I'm pretty serious photographer ( pnArt.com (http://www.pnart.com) )with pretty high standards, and in a typical 5-hour model shoot I'll take about 200 frames, and in my first pass I'll select maybe half or LESS of them to read in from the card, and then after reviewing those I'll toss out half of those, etc. In the end maybe 5 shots will make it into my, or my model's portfolio, and another 10 or 15 will be kept for their "potential", and I've NEVER regretted reformatting my CF cards! I think it's just a matter of having good judgement about what has some potential.

I don't doubt your seriousness in what you're doing but there is a huge difference in video and still photos. I'm sure you can identify what photos you would never use and not worry about destroying them. Think of video as hundreds of thousands of photos and the ability to discern what you would never use again is a more difficult task.

As comparing memory chips to tapes, it's like apples to oranges. I think we all copy our memory chips to disk and than archive the photos to a CD or DVD but tapes only cost $3 to $5 so it's not a valid issue like it is with memory chips. As for as longevity, I can't say much about mini dv tapes but I can say that I have reel to reel audio tapes that I personally taped over 50 years ago and I'm still in the process of sucessfully capturing the audio to my computer and saving them to CD's.

I've been doing computer service for 38 years now and I agree with you on backups. We all must decide on an individual basis what data on our computers is important. We should ask ourselves each day, what would it do to my life if my hard disk crashed. All of the stuff on our disks has varying degrees of importance. I try and back up what is vital and important to me to DVD's and no, I don't put a copy in a safe. I'm a risk taker and I see that you aren't. I doubt that there are one in 50,000 users who would place hard disk backups in a safe but that doesn't mean you're wrong, your're just not typical. It just means that you place more importance on your data then most people do. Do you do image backups so you can recover everything including programs? We all must look at what risks we take and assess whether it is worth the risk.

The bottom line is that you won't find many people who try and scrub video they don't think they want anymore to try and save the cost of tapes.

[bluegrass]

Shit happens. It's important to do regular backup. I setup a second computer (a Dell C521 low profile computer.. costs $3xx) in my living room with these functions

- Backup server
- Media server
- PVR recording TV programs
- Encoding station
- web/FTP server
- File server

It's on all the time connecting to my plasma TV running Windows Media Center. It's also a FTP and web server. Then running backup program (Second Copy) to backup my main machine every few hours.

I have Canopus Procoder installed to that machine using the "Watch Folder" feature. Whenever I need to encode MP3, WMV, MPEG, DIVX..... i just need to drag a short cut to those watch folders, then check the output folder later to get the resulting files.

Taky__________that sound like a cool operation you got going there. I would like to get some more details and feedback on how well things work out with it. Under $400. I'd set that up in a heartbeat if I could.

[plnelson]

I doubt that there are one in 50,000 users who would place hard disk backups in a safe but that doesn't mean you're wrong, your're just not typical. It just means that you place more importance on your data then most people do.
It's not a safe per se, my safe deposit box is just an alternative off-site storage. EVERYBODY should keep their backups in off-site storage; else what happens if you have a fire?


Do you do image backups so you can recover everything including programs?

No, image backups aren't necessary because I can always buy new software, or just replace things like Photoshop CS, etc, from my product key. I also back up install CD's of expensive sw tools like Visual Studio (and yes, I've tested it - it DOES install off a backup) I can't replace my photos (and now videos), financial records, etc. I only back up stuff that can't easily be replaced.

The other reason why images, e.g., Ghosting, etc, are not very useful is that they have to be installed on systems with very similar hw configurations to what they were made from, since all the Registry settings and drivers, etc are specific to the hardware and SP's. So Ghosting is good for a HD crash, but not very useful if you have a fire or theft of your PC and have to replace the whole box with something different. If I get a new PC I'd rather install the apps fresh.

But I still don't know the answer to my question: How much stuff do serious videographers keep on their HD and how do they back up their video content?

[Lunchbox]

plnelson,

I'm a professional wedding videographer. Currently I'm still on SD and slowly starting HD. I use 2 Canon GL2 camcorders. Here's what I will do (again, this is SD)

1. Since I use 2 camcorders simultenously, for a full day wedding, I will have about 14 to 18 tapes. I captured them using my laptop so I can still work on my main computer.

2. I edit all AVIs in my main computer with a Matrox RT.X100 card. Export to final AVI

3. Convert final AVI to MPEG-2 NTSC, then author DVD

4. When the DVDs are delivered and approved by clients, I just deleted all the raw footage AVI files (I also sell client an option to have all the original unedited raw footage for $400. So far, there was only 1 client paid for that option).

5. Record the edited AVI back to the same video tapes I recorded from the event. It usually ended up using 3 tapes. I recycled the rest of the tapes once. Those tapes are Panasonic AMQ tape costs $6 a pop.

7. Burn the edited AVI to data DVD for archiving

8. Burn the MPEG file to data DVD for archiving

9. Burn the Premiere project files and all other clip art, stock photos, sound tracks uses to data DVD for archiving. I won't be able to re-edit the project file anymore because I deleted all the raw AVI files.

It's not a suggested workflow, I have been doing this for 5 years. There is also an alternate step I did was to convert the raw AVI to Microsoft Video1 AVI so I can edit the rough cut in my laptop. Sometimes I just like to sit out in starbucks instead of sitting at home all day.

I think you main concern was about the original raw footage. Let's be honest, no one wants so sit through all the boring stuff. After you edited the final movie, that's the piece you want to show people. However, for some personal events, I would suggest you saving all the raw tapes. They aren't expensive (usually about $2 to $3 a tape). I am sure you won't go back again but it's just cheap to save all the tapes.

For my workflow, although I re-use all the mini-DV tapes, I have the final edited AVI archived to both mini-DV tape as well as DVD data disc.

Sorry for the lengthy post.

[Lunchbox]

Bluegrass, that $400 is just for the Dell C521 machine. There are additinal costs (500GB harddrive, TV tuner card, backup software). What do you need to know about my setup?

[bluegrass]

Bluegrass, that $400 is just for the Dell C521 machine. There are additinal costs (500GB harddrive, TV tuner card, backup software). What do you need to know about my setup?

Sorry, this is getting into a discussion that should be done in private I guess. Let me study your post some more and maybe I'll get with you on the private email. I had been thinking about something like what you are doing and thought it might be worth exchanging some ideas.

Thanks

[Sherb]

plnelson,

I believe this will answer your question.

I capture the entire tape as an m2t file to save as a master copy - it is an exact copy of what is on the tape. I perform the original capture with DVHSCap (FREE!), part of an Apple SDK. On PC side I believe you can use a program called HDVSplit (FREE - I think?). This file is saved as "read-only" so I don't accidentally destroy it. I edit the favorite bits out with MPEG Streamclip (FREE!) and save to a new file - it saves an EXACT copy of the m2t stream except for a few frames around cuts - which must be generated because of the long-gop format of HDV. MPEG Streamclip saves the edited portions as a .ts file - transport stream. This file is very similar to m2t and as far as the video data goes, it is identical. You keep the full quality of the tape, just as you mentioned in your original post.

For any more serious editing than just cuts, you will need a native-hdv editor to retain the full quality of your m2t file.

If you transcode to another format for editing, you can use most any editor.

Hope that helps!

[Sherb]

PS - I really mean no offense by this, please don't take it the wrong way. But I really think you will need to change your perspective on storage space - 1.6GB is miniscule! You can get a 500GB external hard drive for around $100 if you look around a little - that is less expensive than tape!! For a $1000 camera, $100 to back up 40 hours of video is darn cheap.

[Lunchbox]

Sorry, this is getting into a discussion that should be done in private I guess. Let me study your post some more and maybe I'll get with you on the private email. I had been thinking about something like what you are doing and thought it might be worth exchanging some ideas.

Thanks

Bluegrass, it isn't that private... but it's better start a new thread for such discussion as I'm sure many people are also interested in setting up a media PC so they can enjoy their HV20 video on their big screen TV.

[bluegrass]

Bluegrass, it isn't that private... but it's better start a new thread for such discussion as I'm sure many people are also interested in setting up a media PC so they can enjoy their HV20 video on their big screen TV.

Great idea. Should I ask how to setup a Media Center PC for multiuse including viewing or HDV videos on our TV (HD and other). Lots of good discussions can be had about this topic.

I've said more than once, we don't seem to be talking about how to view and use all the great HD we are shooting. My son uses it and even has his 360 Xbox in there somewhere.

Lets do it. I'm headin' home now.

[Lunchbox]

Bluegrass, I have posted a new thread there

http://www.hv20.com/showthread.php?p=7112#post7112

[Kroot]

If you want trouble free automatic backups - just setup headless PC with Windows Home Server. It will automatically backups all PCs.

I'm using it since Beta1 at home (have 5 PCs) and it works perfectly for me :)

[Lunchbox]

I have a media PC attached to my plasma TV running an automatic backup service there. Been working fine.