¶Why Avisynth can't be used with 64-bit VirtualDub
One problem with attempting to port software from 32-bit Windows to 64-bit Windows is that 64-bit processes cannot load 32-bit DLLs, even on the AMD64 (x64) platform. This is a major annoyance because it means that 32-bit video codecs, audio codecs, filters, and file format drivers cannot be loaded into 64-bit VirtualDub. Unfortunately, because Avisynth is a 32-bit DLL, it cannot be loaded either, and thus Avisynth scripts cannot be used.
The solution for a problem like this is to create a 32-bit process that hosts the 32-bit DLLs, and then use interprocess communication (IPC) to transfer data between the 32-bit and 64-bit processes. This is suboptimal because of the delays and inefficiencies in switching processes, but it would work at least and maintain compatibility. There are a number of IPC mechanisms available on Windows, but most of them have major deficiencies, such as not being securable, not having support for waiting or non-blocking I/O, or requiring Windows NT. One with relatively less suckage is COM marshaling, in which the Windows COM system automatically transfers a function call and assocated data across the process boundary.
It turns out that COM marshalling between 32-bit and 64-bit processes is explicitly allowed by Win64. The AVIFile interfaces implemented by Avisynth are COM interfaces, and thus should be usable from a 64-bit processes. Trying to do so, however, just causes AVIFile import errors. So what's going on?
I'm not a COM expert by any means, but digging around a bit in MSDN Library revealed that some Registry setup is necessary for interprocess COM marshalling that the Avisynth installer doesn't do, namely:
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Classes\CLSID\{E6D6B700-124D-11D4-86F3-DB80AFD98778}]
"AppId"="{E6D6B700-124D-11D4-86F3-DB80AFD98778}"
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Classes\AppID\{E6D6B700-124D-11D4-86F3-DB80AFD98778}]
@="Avisynth""DllSurrogate"=""This directs DCOM to launch the default dllhost.exe application to host Avisynth. After doing this, the base Avisynth class that implements IAVIFile can be created in a 64-bit process using CoCreateInstance(CLSCTX_LOCAL_SERVER), and then can be queried to IPersistFile to open an Avisynth script.Unfortunately, that's as far as I got.
The COM marshaller needs to know the makeup of interfaces and types used in an interface to know how to "remote" a call across the process or network boundary. It seems that Microsoft never created a type library for the IAVIFile or IAVIStream interfaces, and so although you can create Avisynth in a process host, the IAVIFile interface you get back is useless as it returns 8001000B (RPC_E_CLIENT_CANTMARSHAL_DATA) for all calls. Nuts.
I tried creating an interface description language (IDL) file for the interfaces and compiling it with the MIDL compiler, but it seems the IAVIStream interface isn't remotable. The reason is that it has methods that return LONG instead of HRESULT, making it impossible for the marshaller to return remoting errors. This is borne out by the disassembly of AVIFileOpen() in the OS, which uses CLSCTX_INPROC_SERVER | CLSCTX_INPROC_HANDLER for the creation context, ruling out use of an out-of-process server. As I understand it this basically means that IAVIFile and IAVIStream methods would have to be reflected to a different, remote-compatible COM interface, which would require a custom 64-bit Avisynth module to handle the translation. At that point you might as well create a custom marshalling mechanism too in order to avoid the overhead of copying all of the audio and video data.
So, sadly, it looks like the only way to use Avisynth in 64-bit processes is to actually write Avisynth64.