PC: 24p capture and pulldown removal step-by-step instructions by Farnsworth

[Farnsworth]

UPDATE: This post is several months old now, and the process has been documented much more completely by Eugenia. If you're just starting out with this pulldown removal stuff, by far the best place to start is here:

http://eugenia.queru.com/2007/07/13/...-24p-pulldown/
.
.

ORIGINAL POST:
Well, heck, then I'll post the details. It's really not that complicated. It just FELT complicated while I was doing it.

And it may LOOK lengthy and involved because this post is so wordy. But trust me, after some initial set-up, it's an incredibly simple process and works very fast.

Stuff you need to install:

AVISynth 2.5
the TIVTC plugin for AVISynth (excellent inverse telecine plugin)
dgmpgdec plugin for AVISynth (mpeg2 indexer/decoder)
VirtualDubMod
HDVSplit

Once you've got all that, here's what you do...

1) Capture your clips with HDVSplit. Let it do the scene detection for you. And capture them all into one folder. Again, I'll call it C:\clips
2) Copy my AVISynth template (attached) into that same folder. It's called "template.avs". It's a simple set of instructions for de-interlacing the m2t files, and it'll be run on every clip.
3) Run my batch file. I call it go.bat, and it's a simple one-liner that looks like this:

FOR %%f IN (%1\*.m2t) DO c:\dgindex\dgindex -if=[%%f] -FO=0 -OF=[%%f_] -AT=[%1\template.avs] -EXIT

So you'd just type "go.bat c:\clips" (without the quotes).

It's extremely inelegant so far, but it works. You'll need to adjust the path in this batch file so it points to the folder where you installed dgindex. All this batch does is run DGIndex (included with the dgmpgdec package above) on every clip in the folder. Each clip will get its own .d2v and .avs file, as well as a separate file containing the audio track. I must admit, I haven't been too concerned with audio during this procedure, so I'll have to play around with that aspect a bit.

4) At this stage you could theoretically import the .avs files directly into Premiere as clips using a third-party plug-in. But they'll edit VERRRY SLOWLY. So don't do this.

Instead, run VirtualDubMod. You can load each of the .avs files as if it were a video clip. They'll open as full 24fps 1080P video clips, and they'll look fantastic. And if you want to do them all at once, you can use VirtualDubMod's "Process directory" option. Everybody will have different needs at this point, and VirtualDubMod can probably do what you want. Resize them, make them Quicktime, Windows Media, whatever you need. It'll batch process them all. As I said, I use it to produce compressed low-resolution proxy clips for editing AND full resolution lossless clips to be substituted in after editing, all in one batch process.

Any questions, I can certainly try to help. And if there are any experts out there who want to help make this more elegant, I'd love some help!

[SSzudzik]

Works really well and I'll have to try messing with the audio too! Don't want you to have to do all of the work!

The only things I had to do additionally was to install the DivX package. I chose to install Xvid instead from :http://www.koepi.org/xvid.shtml

Also had to modify the template.avs to include the line:

loadplugin("c:\tools\dgindex\DGDecode.dll")

Without that, VirtualDubMod kept giving me some errors (probably just a configuration issue on my side). The produced AVI doesn't play well in in Windows Media Player, but Vegas seems ok with it and rendering out a new M2T with it worked just fine.

Thanks for all the hard work and time you put into this.

Steve

[Farnsworth]

The only things I had to do additionally was to install the DivX package. I chose to install Xvid instead from :http://www.koepi.org/xvid.shtml

Also had to modify the template.avs to include the line:

loadplugin("c:\tools\dgindex\DGDecode.dll")

I didn't realize DivX was needed as well. I already had it installed, so I guess I never noticed.

And you won't need to add that extra line to the template if you put DGDecode.dll into your AVISynth plugins folder.

I'm glad it's working for everyone!

[Mal]

Okay I separated these posts out from the other thread and made this a sticky.

.....

Unfortunately this does not work on my system; I think it has to do with the fact that the clips NEED to be captured with HDVsplit, and that won't work here. Gives error and quits. Win XP 64.

[Farnsworth]

I don't think there's any reason why you NEED to use HDVSplit, as long as you're capturing m2t clips. What are you capturing with?

[Mal]

Clips had been captured with VEGAS 7. So they are already m2t files.
Since I am rather "command prompt challenged", it may well be something I am doing wrong.
I do get the three files though. What starts out as a 43MB test m2t clip, gets the following added:
xxxx.avs (about 89B)
xxxx.mpa (about 600KB...this is fine, and plays audio perfectly)
xxxx.d2v (about 2kB)


Opening the AVS file above in VirtualDubMod gives following error:
"there is no function named MPEG2Source"

[Farnsworth]

Ah, okay, you're mostly there. It looks like the dgmpgdec plugin just isn't installed correctly. You need to make sure DGDecode.dll (which comes in the same zip file as DGIndex) is in your avisynth/plugins folder. Then it should work perfectly!

[SSzudzik]

Yup that was the same error I received until I added the following into my template:

loadplugin("c:\tools\dgindex\DGDecode.dll")

I moved the DLL into the AVISynth plugin folder and then I no longer was required to have this line in the template. Either way seems to work just fine though.

Still working up the audio though. Always seems just a little bit out of sync.

--Steve

[Farnsworth]

Sadly, I'm not equipped to experiment with sound at the moment. But if I was, the first thing I'd try is making the sound 104.2709% of its original length. You can do this in After Effects with the Time Stretch function, or in Sound Forge, or Audition, or whatever.

Maybe try that and let me know if it works!

[Farnsworth]

Here's an updated version of the template file for this process. Using this new template, when you open the files in VirtualDubMod they will include the original audio, hopefully in sync.

You also need to add the attached plugin to your avisynth\plugins folder. It allows AVISynth to handle mpa audio.

Give it a try and let me know if you have any synch issues! My initial tests looked very positive.

EDIT: I have attached an updated template that holds better synch. It should get your audio synched within about plus or minus 1/100th of a second.

[SSzudzik]

I must have messed something up on my box because I can't get DG to work with my audio anymore. The resulting avs file for me looks like:

v=MPEG2Source("c:\clips\test\Live24p-2007_05_02-16_55_52.m2t_.d2v")
a=MPASource("")
audiodub(v,a)
TFM(d2v="c:\clips\test\Live24p-2007_05_02-16_55_52.m2t_.d2v")
tdecimate()

No MPASource and VDub won't open the file as a result. Have no idea what I did to cause this

--Steve

[Farnsworth]

Is it possible you disabled audio from the Audio-->Output Method menu option within DGIndex? That would be saved as a setting and applied to all the files when you run it at command line. Try setting it back to "Demux all tracks".

And make sure to download the updated template above! I changed it as of about 2:15 EST today.

[SSzudzik]

Yup, you were correct. While I was off trying to get it working on my own, I had changed that setting. I've got my audio back.

It's very close. I've got a slight sync issue, very slight. Excellent progress though! Would be a lot easier with flags, or if Vegas would do what Cinerform does and just analyze the frames instead of expecting flags.

Steve

[Farnsworth]

Did you use the updated template, which introduces a 222ms delay into the audio? That fixed the small synch issue I was having.

[duzzit_madder]

Thanks for the update Farnsworth.

[SSzudzik]

Had the newer template in the parent folder.
Very very close! I'm going to have to take some more 24p video and play around with it, but I think you've got it. Great work!

Steve

[sillyman]

Steve,

Are you able to make it sync without putting a delay in?

My version of this process has to add in the delay as part of MPASource.
vid=MPEG2Source("d:\capture\Auntshirley\\AuntShirl ey.m2t_.d2v")
aud=MPASource("AuntShirley.m2t_ MPA PID 814 DELAY -237ms.mpa").delayaudio(-0.237)
audiodub(vid,aud)
TFM(d2v="d:\capture\Auntshirley\\AuntShirley.m2t_. d2v")
tdecimate()

I'm doing it manually right now but I'm planning to write code to do it automagically.



UPDATE with proposed changes for Farnsworth:
The code below is for fixavsdelay.pl. You need to install ActivePERL 5.8 which is free. Save it to the same directory as go.bat. In go.bat you can create the template.avs as noted in the comment lines. Also modify go.bat with the comment lines below. Once it runs, two avs files are generated. One is the original made by dgindex. The other has a similar name but ends with _fixedaudio.avs which adds the delay value from the filename. It's messy yes, I didn't want to overwrite the original AVS incase you don't like how this works.


#!c:\perl\bin\perl.exe
# This file is fixavsdelay.pl
# You can install ActivePERL 5.8 to parse this for you
# Parse AVS files in directory with proper delay

# My template.AVS:
#vid=MPEG2Source("__vid__")
#aud=MPASource("__aud__").delayaudio()
#audiodub(vid,aud)
#TFM(d2v="__vid__")
#tdecimate()

# My go.bat
#COPY template.avs %1
#FOR %%f IN (%1\*.m2t) DO dgindex\dgindex.exe -if=[%%f] -FO=0 -OF=[%%f_] -AT=[%1\template.avs] -EXIT
#FOR %%f IN (%1\*.m2t) DO perl fixavsdelay.pl %%f


sub mstos {
my $ms = $_[0];
#preserve the "-" sign if present
my $sign = "";
$sign = '-' if ($ms=~/^\-/);
$ms =~ s/^\-//;
my $secs = sprintf("%.3f", ($ms/1000));
return $sign.$secs;
}

my $filename = $ARGV[0];
open(AVSFILE,"<$filename\_\.avs");
open(OUTFILE,">$filename\_fixedaudio\.avs");

foreach my $line (<AVSFILE>) {
if ($line =~ /DELAY /) {
my $subline = $'; # part of string after word DELAY
$subline =~ /ms\.mpa/; # part of string between delay and ms.mpa
my $delaystring = $`;
my $seconds=mstos($delaystring);
$line =~ s/delayaudio\(\)/delayaudio\($seconds\)/;
print OUTFILE $line;
} else {
print OUTFILE $line;
}
}
close(aVSFILE);
close(OUTFILE);

__END__


sharing the joy at another forum ...
http://www.dvinfo.net/conf/showthread.php?t=92893

[SSzudzik]

With the delay in Farnsworths template, it seems pretty dead on. I thought I might have noticed a little sync issue when viewing in Vegas, but when rendering it out to WMV, it looks spot on.

[sillyman]

Put together my version of the scripts at http://www.sendspace.com/file/71jk3n
Also some detail blurb at http://yousillyman.blogspot.com/2007...-hv20-m2t.html

If someone here knows how to make install packages and VB it would be nice if you could help package it up.

[therager]

......As I said, I use it to produce compressed low-resolution proxy clips for editing AND full resolution lossless clips to be substituted in after editing, all in one batch process.

First off thanks for all the effort in putting this together!
Question: I work in PP2.0 and id like to know how to achieve this process you are describing where you edit in low res and then substitute after editing...

[Farnsworth]

With the delay in Farnsworths template, it seems pretty dead on. I thought I might have noticed a little sync issue when viewing in Vegas, but when rendering it out to WMV, it looks spot on.

The delay for HV20 clips seems to consistently average 222ms. (Some clips go down to about 217 and some up around 230, but in general I calculated 222 as the average.) So for simplicity, that's what I put into my template. The variance is less than one frame at 24fps, so it should be adequate for now.

Sillyman, I appreciate you trying to work the accurate delay in. I'm just a little reluctant to add ActivePERL into the mix, since I'm almost positive the same thing can be achieved within the original batch file, with a bit of work. I just haven't figured out how to do it yet.

[Farnsworth]

First off thanks for all the effort in putting this together!
Question: I work in PP2.0 and id like to know how to achieve this process you are describing where you edit in low res and then substitute after editing...

I actually do my online in After Effects, so that's where I do my substitution. I edit with low-res clips in PP2.0, then open the project in After Effects. Once in AE I import all the high-res version of the clips, and then alt-drag them over the low-res versions. It automatically replaces every instance of that clip on the timeline with the full-res version. Then I just change the composition settings to match the proper size.

There may be an easier way to do this! I'm relatively new to it.

[marshallarts]

Is there any way to change the title of this so it says "not for Macs"?

I was so excited only to read and realize this won't work for me. And when did everyone decide to go Microsoft???!!!

[Mal]

Is there any way to change the title of this so it says "not for Macs"?

That's what the "PC:" means in the title.

I think there's some Mac workflow around here somewhere though....not sure.

[marshallarts]

Man.... I am a complete idiot.... I should go back and edit that but I'm so embarrassed I need to leave that up for shame..

Yeah, there is a Mac workflow but haven't found anywhere near the same good input as I have from reading PC users replies for some reason. Thought there would be more Mac people.