One of the rites of passage when converting a Visual Studio 2005 or 2008 project to VS2010 is learning how to fix custom build rules. The MSBuild system uses completely different custom build rule files than VS2005/2008, and the converter is dicey at best. Here I'll outline some of the issues I've encountered and ways to fix them.

First, some background: a custom build rule file (.rules) in VS2005/2008 tells the project system how to invoke an external tool to do a build step. It contains not only the instructions for how to run the build step, but also which properties to expose in the UI for setting or overriding at project and file level, and dependencies to integrate into the incremental build. They're manipulated from the Custom Build Rules menu option on the project context menu. Visual Studio comes with a few custom build rules, including one for the Microsoft Macro Assembler (MASM).

A foo.rules file in VS2005/2008 is converted to three separate files in VS2010 for MSBuild consumption:

• foo.xml: This file lists the properties that show up in the UI and that can be overridden from the .vcxproj property files. Usually this is not a source of problems and can be left alone.
• foo.props: This file contains the default properties for items subjected to the custom build rule, as well as a few derived properties, such as the command line. A number of issues arise from expansions not working correctly on the derived properties.
• foo.targets: This is the meat of the custom build rule and is the one that actually contains the MSBuild-based logic for invoking the custom build tool. This is the file on which you'll usually be doing heavy surgery.

There are two additional complications with fixing custom build rules in VS2010. The first is that the UI for custom build rules has been removed, so fixing the rules involves wading into raw XML. The second problem is that the MSBuild documentation is quite poor, not describing basic concepts adequately and frequently mixing API and build file concepts. The information I provide below has mostly been gleaned through trial and error, so those of you familiar with MSBuild can probably find many places to correct terminology or provide better solutions.

Oh, one more thing: we'll need a sample project to begin. This is a VS2005 project with a correctly working build rule: http://www.virtualdub.org/downloads/CustomBuildRule-VS2005.zip

If you have VS2010 or VC++ 2010 Express, convert the project to 2010 format and follow along.