So, I think I've figured it out, and it has to do with overrunning/exceeding the buffer.
What you need to do is, you need to download the source files for MPlayer/MEncoder. Once you have them, extract the file. (Something like Mplayer-1.0rc2.tar.bz2) Open up Terminal and navigate to the directory into which the contents of the file was extracted. (For instance, if you extracted it to "MPlayer-1.0rc2" on your Desktop, you can just type in:
...and it should take you there. (This is working from default settings -- if you've modified Bash in any way, I assume you already know how to change directories.)
Then, navigate to the directory in Finder and open up the file mencoder.c with a text editor. (Or, if you're familiar with the command line, use emacs or vi or whatever.) NOT MICROSOFT WORD. I can't possibly stress this enough.
TextEdit should work fine, as long as you make sure you don't accidentally save the file, then, in RichText. I'd strongly recommend TextWrangler, a free text editor, since then you can likely rest assured that you're not inadvertently adding a bunch of stupid extra stuff that word processing applications insert.
Anyway, change this line (just do a "find" for the term "buffer_size=0x" and it should bring you to the line):
mux_v->buffer_size=0x200000; // 2MB
mux_v->buffer_size=0x400000; // 4MB
Then, in Terminal (you should still be in the MPlayer directory that you extracted earlier), type in:
It'll run a script that configures the build you're about to do. It should take maybe thirty seconds or a minute. Text will flood in your Terminal for a bit, but don't worry, and just let it do its job.
Now, that should work just fine, and you'll see a message near the end about "You should now be able to type 'make' and 'make install' to build and install MPlayer."
This will take a little longer, and text will again flood the Terminal. Just let it do what it's going to do.
Now, I ran into an error with mine, about a few undefined libraries (iconv stuff). If this happens to you as well, it's not that big a deal. The easiest way to fix this is to just delete the MPlayer-1.0rc2 directory and start again from the top, only when you get to the part about running the configure script, this time type in:
and you should be all right. If you're missing other libraries, though, there's something else wrong and you should probably google for whatever the problem is that you're getting to see what you can find out about it. (Also, before you do 'make' again, make sure you've remembered to edit the mencoder.c file to change the buffer limit to 40000, as detailed above, since you ran into the problem and had to delete it the first time.)
Anyway, at this point, after you've done 'make', you can either do:
(though you'll probably get permissions errors, so you'll have to type in
sudo make install
and it'll ask you for your password which will give you the "authority" necessary to do what you're trying to do.
However, you've probably already installed the mencoder per the instructions in the original post, so what you should do is:
sudo rm /bin/mencoder
which will get rid of it. And then you could just do:
sudo cp mencoder /bin
which will move the new version of mencoder in to where the old one was.
The next time you run mencoder as on Eugenia's site or in the original post here, you shouldn't get any buffer errors, because you've built a modified version from source with a larger buffer size set.
It's been working fine for me since.