¶File extension madness

Ever hit the problem in Windows where you tell a program you want it to handle some file types, and only some of them "take"?

I've got a pretty good idea now why this happens, and it's a mess.

The MSDN Library has a section on how to associate file types in code. There are some omissions -- I knew what the positive and negative numbers meant on icon references, although that doesn't actually seem to be documented -- but in all it's actually not bad. If you follow the documentation you can indeed add a UI to your program to associate file types, and if you spend another hour in the Vista UAC section you can even get your program to properly relaunch itself as elevated as needed. Then you send it out to your users, and you find out it inexplicably doesn't work on some users' systems.

Well, long story short, the culprit turns out to be yet another overlay on the file type mechanism located at HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts. This is a bit odd, because there is already a user-local overlay over the main class database at HKCU\Software\Classes. Within the key for each extension here there can be a subkey called UserChoice, which is created when the user specifically overrides the current association with the Open With... option and has the "always use this type" checked. This then overrides the ProgID selected through in HKCR. Frustratingly, I wasn't able to find any documentation on this in MSDN, but it wasn't until I tried to delete the subkey for testing in Registry Editor and got an Access Denied error that I had a clue why it wasn't documented. Let's take a look at the security settings:

A Deny entry for the current user, for only Special permissions? That's strange. What exactly are they?

Mmmhmm, the shell explicitly set a Deny entry for the Set Value permission on the key. That makes this Microsoft's latest salvo in the War Against the Coolest Most Amazing Program Ever Written, especially since bypassing this requires mucking around in security APIs that are mazes of SIDs and ACEs. While I appreciate the intent -- having had to manually clean up the mess from programs that thought it funny to claim .ZIP and .PDB as media types -- I'm not sure that the resultant game of Open With whack-a-mole that results is any better, where first I have to manually switch AVI, then MPG, then MP4, and every other extension still bound to the ugly-as-sin Windows Media Player. It would be nice if the file type documentation actually mentioned this user override too, even if it didn't include storage details, because this appears to make the documented HKCR method useless for any sort of custom association UI.

Digging around the MSDN links a bit reveals an IApplicationAssociationRegistration interface added with Vista which might work, but I haven't tried that yet. Annoyingly, it requires you to register your application in HKLM, which seems backward given the moves toward storing file associations per-user. That it offers a programmatic interface to override associations also doesn't make me hopeful since that also goes against the bunker Microsoft's been trying to build around the file type settings.

Comments

This blog was originally open for comments when this entry was first posted, but was later closed and then removed due to spam and after a migration away from the original blog software. Unfortunately, it would have been a lot of work to reformat the comments to republish them. The author thanks everyone who posted comments and added to the discussion.