Batch 3:2 Pulldown Remover

[Khaver]

I've created a dedicated utility for 3:2 pulldown removal of your PF24 (24p in 60i) clips.

AVS Batch 3:2 Pulldown Remover

MOD EDIT: see Khaver's .sig at the bottom of this post for the latest version

Docs enclosed and below:

AVS 3:2 Pulldown Remover v1.0
(Freeware, use at your own risk)

This utility removes 3:2 pulldown from PF24 (24p in 60i) footage and creates compressed intermediate AVI video files, separate audio WAV files and optionally proxy AVI files for use in video editing applications. It uses your installed Video for Windows codecs for compression.
(Inspired by Steve Szudzik’s HV20Pulldown utility. Thanks Steve.)

Requirements:
The freeware directshow video and audio decoder FFDSHOW.
The freeware video and audio frameserver AVISYNTH.
Donald Graft’s Decomb.dll AVISYNTH filter.
Install these per their instructions. (see below for links)

Place AVS-32PDR.exe and avs32pd.exe and the included post processing script into a folder of your choice.

Run AVS-32PDR.exe



1: Enter the folder for the location of the created 24p intermediate AVI video files. (this path will be saved to the registry and used the next time you run the utility)

2: Click this button to select the folder for the intermediate video files.

3: Check this box to create the audio WAV files of the sound track.

**Note: The created intermediate and proxy files will not contain an audio track. Make sure this box is checked if you need the audio.

4: Enter the folder for the location of the created intermediate audio WAV files. If this is left blank, the WAV files will be put in the same folder as the intermediate files. (this path will be saved to the registry and used the next time you run the utility)

5: Click this button to select the folder for the intermediate audio files.

6: Check this box to create proxy AVI video files.

7: Enter the folder for the location of the created proxy AVI video files. (this path will be saved to the registry and used the next time you run the utility)

**Note: This location can not be the same as the intermediate folder.

8: Click this button to select the folder for the proxy video files.

9: Check this box to apply post processing to the video using the functions in the AVISYNTH script

10: This information box shows the AVS script file that will be used for post processing.

***Note: This utility creates an AVISYNTH script file on-the-fly and uses it to do the pulldown removal. Enabling post processing will append the contents of the post processing AVISYNTH script file to this on-the-fly created file.

11: Click this button to select the AVS template file.

The included "24p-PP.avs" script will simply resize the video to 1920x1080. You can add to this file or use your own custom scripts.

**Note: AVISYNTH is a powerful video processing tool with many 3rd party plugins available for it. Take the time to learn its script language.

12: Enter the suffix to add to the end of the name of the intermediate and proxy AVI files. (default is -INT)

13: Check the box if you want to remove the temporary files the utility creates for each created AVI file.

14: Click this button to edit the AVISYNTH post processing script.

15: Click this button to configure FFDSHOW. (This utility requires FFDSHOW to feed AVISYNTH interlaced video)

In the FFDSHOW decoder configuration dialog, go to the Codecs list and set the desired decoders for all formats you want FFDSHOW to decode.

Required:
H.264/AVC- libavcodec or ffmpeg-mt (for AVCHD files)
MPEG2 - libavcodec or libmpeg2 (for HDV)
Suggestions:
Huffyuv - libavcodec (playback of huffyuv and ffvhuff)
MJPEG - libavcodec (playback of MJPEG and LJPEG)

Make sure the "Deinterlacing" check box is unchecked.

16: Click the "Choose Codec" button to select and configure the codec for the 24p intermediate AVI file compression.

17: This information box shows which codec will be used for intermediate file compression.

18: This information box shows the FourCC of the intermediate compression.

19: Check the box to normalize the audio in the created audio WAV file.

**Note: Normalize will double the time required for WAV file creation.

20: Click the "Choose Codec" button to select and configure the codec for the proxy AVI file compression.

21: This information box shows which codec will be used for proxy file compression.

22: This information box shows the FourCC of the proxy compression.

23: Enter the width and height of the proxy AVI files if you want it different than the intermediate width and height. Enter as WxH. This will add a corresponding resize command at the end of the temporary proxy AVS file. (640x360 is good for 16:9 content) Leave blank for proxy size to be the same size as the intermediates.

24: Click the "Add"... button to add your source files to the file list.

25: Click "Remove" to remove selected files from the file list.

26: Click "Clear" to clear the file list.

27: This is a list of the source files you will be batch converting. The green and red boxes indicate the status of the conversions.

29: Click "Start" to begin the pulldown removal process. The utility will analyze a 1 second portion of the frame looking for the pulldown cadence. If it can’t find it in this portion, it will try another (if it can’t find it in this second portion the conversion will abort and a red mark will be added next to the file in the file list). When pulldown removal is successful and each conversion completes, a green box is added to the source file list in front of the file name.

**Note: If the encoding to your intermediate format fails and avs32pd.exe crashes, a red box will also be added to the file list. It usually means the chosen codec is not compatible with the output from AVISYNTH or there is a syntax error in the post processing script. Change the configuration for the codec or choose a different codec or fix the syntax error in the AVS file.

30: Click "Abort" to stop the analysis process and intermediate and proxy file creation.

31: Individual file creation progress is displayed.

32: Check "Close" if you want AVS-32PDR to close after completing the creation of all intermediate video, audio, and proxy video files.

**Note: If you abort the job or one or more conversions fail, AVS-32PDR will not close.

33: Total job progress is displayed.

I hope you find this utility useful.

Send suggestions, comments and bug reports to: khaver(at)netzero.net

Links:
FFdshow - http://www.videohelp.com/tools/ffdshow
Avisynth - http://avisynth.org
Decomb.dll - http://neuron2.net/decomb/decomb523.zip

04/13/10) v1.0 Initial release.

[Daniel Rutter]

It looks great mate, thanks for supplying it to us!

I have one request though: Can you allow us to use whatever codec we want? I personally HATE AVI footage (it takes up unnecessary amounts of space)...

Cheers.

[Khaver]

Any Video for Windows codec you have installed you can use, though these codecs will create only AVI files. Though not the best colorspace wise, ffdshow's MJPEG codec at it's highest quality setting is good for general use with not unreasonable sizes. Remember, usually the lesser the compression the better the visual quality of your output.

[Erik Bien]

I'm unquestionably the least post-production-literate HV20.com moderator, but this seems pretty worthy of becoming an 'instant sticky,' either here or in the '24p workflow' sub-forum (any other mods wanna say 'Amen'? 'Coz, see above) ...

Nice one, Khaver!

[racer-x]

I'm unquestionably the least post-production-literate HV20.com moderator, but this seems pretty worthy of becoming an 'instant sticky,' either here or in the '24p workflow' sub-forum (any other mods wanna say 'Amen'? 'Coz, see above) ...

Nice one, Khaver!

Yeah....that boy is a genius. It's a pity I never shoot 24p, but if I did, I would certainly use this cool tool.

Nice job!

[Khaver]

Thanks for the kudos guys. It's been slow at work so I have to keep busy somehow, so it's either work on my little utilities or clean my office.

[bassman]

This looks great!! I can't wait to try it. Normally I use After Effects to do the pull down removal but this looks so much easier.

Thanks for putting this together!!

[Khaver]

I forgot to mention in the docs that each or your shots you load into the utility needs to have been captured as separate files like HDVSplit does. In other words, the cadence of the pulldown needs to be continuous throughout each clip. Capturing as one long file and running it through AVS-32PDR won't work.

[Daniel Rutter]

In other words, scene detection is vital... so it splits your files into nice solid 'scene' chunks

[Khaver]

Just for the info, there's some PF24 HV20 clips on the ftp.hv20.info site called "24p_cinform_breakers_XXX.m2t" that I'm assuming NeoScene couldn't remove pulldown correctly. I downloaded those and my utility correctly removed the pulldown.

[racer-x]

No doubt it did.........with the power of Avisynth + Decomb filter, what the heck can stop it????

That's assuming user intelligence of course.......

[Khaver]

Here's how the utility works.
It first creates an analysis avisynth script.

directshowsource("File",fps=29.97003,audio=false). trim(20,80)
assumeTFF()
separatefields()
doubleweave()
selectevery(5,0,3).assumeframebased()
writefile("test file","IsCombed(20)",False)

It then runs this script through avs32pd.exe (a modified version of avs2avi.exe) with a null output. As it's running, the avs script is writing out the test file where the decomb.dll function "IsCombed" outputs a "true" or "false" depending on if the current frame is combed or not. A "true" means the frame is combed and the selectevery function didn't select the correct pulldown cadence.

When this script completes its run, the utility opens the test file and looks for a "true" in it. If it finds one, it goes back and writes a new script where the selectevery function selects a new cadence and the process repeats a maximum of 4 more times (there are 5 possible cadences to check) until the utility does not find a "true" in the test file. This means the correct cadence was found. If the process goes through all 5 times but still hasn't found the correct cadence it will try 5 more times from a different place in the clip (the ".trim(20,80)" at the end of the directshowsource function is changed to ".trim(90,150)"). If that doesn't work, that clip aborts, the file is marked with red in the utilities file list, and it moves to the next file.

Upon successfully finding the cadence, the utility then re-writes a new script with the correct selectevery function and without the writefile function and optionally adds the post processing script to the end of it. Avs32pd.exe then runs this and encodes the video to the chosen codec.

The whole key to this utility is Donald Graft's decomb.dll plugin for avisynth and it's ability to find combed (interlaced) frames.

[zagnut]

Interesting. Any idea how it may work on AVCHD from an HF series cam? Wait, I know the answer, try it and find out.

ffdshow's MJPEG codec at it's highest quality setting is good for general use with not unreasonable sizes.

While space is saved, from my experience this codec just plain stinks in quality compared to lossless like Huffyuv or Lagarith.

[Khaver]

Yes, it will work with AVCHD clips as long as you set ffdshow to decode them.

You're right about MJPEG not being the best codec but lossless AVIs usually take up more space than MJPEG AVIs, and Daniel was looking for space savings.

[kiwipwnage31]

i just tried removing the pulldown on an AVCHD file from my hf s100. worked perfectly.

is there any possible way of removing pulldown without separately extracting the audio?

[Khaver]

I just updated the AVS-32PDR.exe file in the zip package. The wav files it created had a .444 second delay in them compared to the start of the created AVI video file. This meant you would have to move the start of the audio clip ahead of the start of the video clip by 10 or so frames for proper sync. The utility now compensates for this so you can place the start of the audio at the start of the video.

Please download again and overwrite the AVS-32PDR.exe file.

kiwipwnage31, would you rather have the option to simply remux the created wav file with the created avi video file, or a completely different version that uses ffmpeg to encode the video and audio to one of the codecs available in ffmpeg?

[kiwipwnage31]

i think an option to remux the audio and video would be nice. do you think there is a way to add a line to the avisynth script to make it do just that?

[Khaver]

i think an option to remux the audio and video would be nice. do you think there is a way to add a line to the avisynth script to make it do just that?

I'll have to look into this because the program I use (avs32pd.exe) to encode the video can only output the video and audio separately . I'm looking into either modifying avs32pd.exe to optionally encode audio or finding a new command line utility that will do it. Just modifying the avisynth script won't do it.

[Dainis]

I always got error - "Please enter path to AVS template."
Did not understand where I need to type it.

[Khaver]

I always got error - "Please enter path to AVS template."
Did not understand where I need to type it.

Look at the picture in the first post and select the button at "11" and select the avs template that you unzipped along with the rest of the programs.

[Dainis]

Does it work on Win7 64 bit ?? I got error again

[Khaver]

I don't have Win 7 64bit so I couldn't test it. Uncheck the "Remove temporary files" box and run it again. Then try running the generated .avs file in a program like VirtualDub to make sure the avisynth script is running without errors.

[Khaver]

I've added a new version of the Batch 32 Pulldown Remover utility.

This one has the ability to optionally remux the audio back into the generated intermediate and proxy AVI files and optionally delete the WAV files after remuxing. You now select which files in the list will get converted. The audio delay has again been tweaked a little. The "Close" checkbox should work now.

Let me know if you find bugs or have ideas for improvement.

[Khaver]

Just uploaded Batch 32 Pulldown Remover version 1.2

I combined the audio remuxer into avs32pd.exe (avimuser.exe is no longer needed). Added an "Invert" selection button for the source file list. Misc bug fixes.

Enjoy!

[Khaver]

For power ffdshow users I've created a special version of my batch 3:2 pulldown remover. You can unzip it into the same directory os the "regular" version. What it does is allow you to have ffdshow add some processing filters to your video without affecting the audio WAV file creation or analysis pass (filters slow down processing and can interfere with the analysis).

First, unzip the AVS-32PDR-S.exe into the same directory as the normal AVS-32PDR.exe. Then make a copy of avs32pd.exe in the same directory and name it avs32pp.exe.

Now run AVS-32PDR-S.exe and open the ffdshow configuration dialog.


On the left, select "Profiles / Preset settings", and at "Presets" click the "New" button. A preset will be added to the list. Rename it something like "analysis". Click the "New" button again and rename the added preset to something like "postprocess". Now, select the "analysis" preset and at the bottom check the "Automatic preset loading" box and click the "Preset autoload conditions..." button.


Select "On one of the conditions match (OR)" and check the "on application exe file name match" box and input "avs32pd.exe" into the edit box. Click "Close".

Select the "postprocess" preset and, again, click the "Preset autoload conditions..." button.


This time add "avs32pp.exe" to the input box and click "Close".


Select the "analysis" preset and on the left make sure nothing is checked.


Now select the "postprocess" preset and add some filters as needed (do not add deinterlacing). Be aware that some filters may affect how well Avisynth can re-weave some of the fields. Experiment.

Now, when you start the pulldown removal process, the audio and analysis passes will use the "analysis" preset in ffdshow because these passes use avs32pd.exe. The pulldown removal pass will use the "postprocess" preset because avs32pp.exe is used.