One of the rites of passage when converting a Visual Studio 2005 or 2008 project to VS2010 is learning how to fix custom build rules. The MSBuild system uses completely different custom build rule files than VS2005/2008, and the converter is dicey at best. Here I'll outline some of the issues I've encountered and ways to fix them.

First, some background: a custom build rule file (.rules) in VS2005/2008 tells the project system how to invoke an external tool to do a build step. It contains not only the instructions for how to run the build step, but also which properties to expose in the UI for setting or overriding at project and file level, and dependencies to integrate into the incremental build. They're manipulated from the Custom Build Rules menu option on the project context menu. Visual Studio comes with a few custom build rules, including one for the Microsoft Macro Assembler (MASM).

A foo.rules file in VS2005/2008 is converted to three separate files in VS2010 for MSBuild consumption:

• foo.xml: This file lists the properties that show up in the UI and that can be overridden from the .vcxproj property files. Usually this is not a source of problems and can be left alone.
• foo.props: This file contains the default properties for items subjected to the custom build rule, as well as a few derived properties, such as the command line. A number of issues arise from expansions not working correctly on the derived properties.
• foo.targets: This is the meat of the custom build rule and is the one that actually contains the MSBuild-based logic for invoking the custom build tool. This is the file on which you'll usually be doing heavy surgery.

There are two additional complications with fixing custom build rules in VS2010. The first is that the UI for custom build rules has been removed, so fixing the rules involves wading into raw XML. The second problem is that the MSBuild documentation is quite poor, not describing basic concepts adequately and frequently mixing API and build file concepts. The information I provide below has mostly been gleaned through trial and error, so those of you familiar with MSBuild can probably find many places to correct terminology or provide better solutions.

Oh, one more thing: we'll need a sample project to begin. This is a VS2005 project with a correctly working build rule: http://www.virtualdub.org/downloads/CustomBuildRule-VS2005.zip

If you have VS2010 or VC++ 2010 Express, convert the project to 2010 format and follow along.

Starting out

Building the VS2005 project produces the following happy output:

1>------ Build started: Project: VS2005-custombuildrule, Configuration: Debug Win32 ------
1>Preprocessing: hello.foo -> hello.cpp (Include name: main)
1>Copying: d:\projwin\VS2005-custombuildrule\main.bar 
1>Adding: d:\projwin\VS2005-custombuildrule\hello.foo 
1>Compiling...
1>hello.cpp
1>Linking...
1>Embedding manifest...
1>VS2005-custombuildrule - 0 error(s), 0 warning(s)
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

Building the converted VS2010 version produces less happy output:

1>------ Build started: Project: VS2005-custombuildrule, Configuration: Debug Win32 ------
1> Preprocessing: hello.foo -> hello.cpp (Include name: [IncludeName])
1> hello.cpp
1> VS2005-custombuildrule.vcxproj -> E:\t\Debug\VS2005-custombuildrule.exe
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

There are a few issues immediately apparent in this output and more lurking under the surface. Let's take them one by one.

Missing output

The first thing you might notice when building the VS2010 version of the project is that it doesn't print the Copying and Adding lines like it does in VS2005. Custom build rules converted in VS2010 don't output anything to the log at normal logging levels. Zero, zilch, nada. This is a pretty lame oversight, and it makes it a bit difficult to figure out why a custom build tool failed.

It turns out that this output is actually captured, but is normally output at too low of a level to appear. The execution description does appear, though, and the Message task has an Importance="High" property. We can therefore apply something similar to the actual build task in the targets file:

<Foobar_builder
  Condition="'@(Foobar_builder)' != '' and '%(Foobar_builder.ExcludedFromBuild)' != 'true'"
  CommandLineTemplate="%(Foobar_builder.CommandLineTemplate)"
  IncludeName="%(Foobar_builder.IncludeName)"
  AdditionalOptions="%(Foobar_builder.AdditionalOptions)"
  StandardOutputImportance="High"
  StandardErrorImportance="High"
  EchoOff="true"
  Inputs="%(Foobar_builder.Identity)" />

The EchoOff property is also needed to prevent the command line from being echoed along with the build step output.

Unexpanded tokens

Next problem: the execution description of the tool looks a bit weird.

Preprocessing: hello.foo -> hello.cpp (Include name: [IncludeName])

Looks like the [IncludeName] property reference is not getting expanded. The reason is that the expansion is only performed by the command line task, and not by other tasks such as the Message task. Therefore, we will need to perform the expansion manually:

<Message
  Importance="High"
  Text="Preprocessing: %(Foobar_builder.Filename).foo -> %(Foobar_builder.Filename).cpp
 (Include name: %(Foobar_builder.IncludeName))" />

The ExecutionDescription property is no longer needed and can be deleted from the .props file afterward, if desired.

Additional dependencies

The way the build tool was originally set up, the hello.foo file also depended upon main.bar, and optionally main.blah if it existed. Does this dependency checking still work? To review, we are using a custom build rule to process (hello.foo; main.bar; main.blah) -> (hello.cpp), which is processed by the C++ compiler: (hello.cpp) -> (hello.obj). Therefore, as you should have learned in Make 101, the following should hold:

• If hello.cpp is missing or older than hello.foo, main.bar, or main.blah, then the custom build rule is invoked to build it.
• If hello.obj is missing or older than hello.cpp, then the C++ compiler is invoked to build it.
• If neither of these hold, then neither the custom build rule nor the C++ compiler should be invoked.

Let's modify hello.foo... builds once... doesn't build again. Good. Now let's modify main.bar and hit Build....

Drat, nothing built.

As you might have guessed, the main.bar dependency isn't actually getting tracked. However, if you delete the executable to force the project to rebuild, you'll notice something else that's screwy, which is that the hello.foo file does rebuild. There are actually two build systems involved here, one being the dependency check in the IDE, and the other being MSBuild itself. They don't use the same mechanisms, and therefore you can get into four different situations:

• VS2010 sees outdated files, and invokes MSBuild, which builds the outdated files. This is how we should start.
• VS2010 and MSBuild both see project as up to date. This is how we should end up.
• VS2010 sees outdated files, but MSBuild doesn't. When this happens, you get projects in the to-build list on a Run command, but all MSBuild does is run through them and determine everything is up to date. Annoying, but livable.
• VS2010 sees the project as up to date, but MSBuild doesn't. This is the broken case we have now. You'll know you're running into this case if you try debugging it by raising the verbosity and you don't get any MSBuild output at all. There is a way to enable logging on the VS project side via the devenv config file, but I've never found it useful.

The reason why this happens is a bit obscure: VS2010 doesn't actually run through the MSBuild scripts, but instead processes the .tlog files in the intermediate directory and checks timestamps on files. In this case, the converted custom build rule has produced a VS2005-custombuildrule.write.1.tlog file indicating what hello.foo produces, but not a VS2005-custombuildrule.read.1.tlog file indicating what it consumes. Therefore VS2010 doesn't see the dependency from hello.foo to main.bar, and it doesn't even bother to invoke the MSBuild engine.

What we need to do is tell Visual Studio about the additional dependencies. We can do this by generating a new .tlog file based on the AdditionalDependencies property:

<ItemGroup>
  <Foobar_builder_inputs Include="%(Foobar_builder.AdditionalDependencies)"/>
</ItemGroup>
<ItemGroup>
  <Foobar_builder_tlog
    Include="%(Foobar_builder.Outputs)"
    Condition="'%(Foobar_builder.Outputs)' != '' and '%(Foobar_builder.ExcludedFromBuild)' != 'true'">
    <Source>@(Foobar_builder, '|')</Source>
    <Inputs>@(Foobar_builder_inputs, ';')</Inputs>
  </Foobar_builder_tlog>
</ItemGroup>
<Message
  Importance="High"
  Text="Preprocessing: %(Foobar_builder.Filename).foo ->
 %(Foobar_builder.Filename).cpp (Include name: %(Foobar_builder.IncludeName))" />
<WriteLinesToFile
  Condition="'@(Foobar_builder_tlog)' != '' and '%(Foobar_builder_tlog.ExcludedFromBuild)' != 'true'"
  File="$(IntDir)$(ProjectName).write.1.tlog"
  Lines="^%(Foobar_builder_tlog.Source);@(Foobar_builder_tlog->'%(Fullpath)')" />
<WriteLinesToFile
  Condition="'@(Foobar_builder_tlog)' != '' and '%(Foobar_builder_tlog.ExcludedFromBuild)' != 'true'"
  File="$(IntDir)$(ProjectName).read.1.tlog"
  Lines="^%(Foobar_builder_tlog.FullPath);%(Foobar_builder_tlog.Inputs)" />

What this does is build an item list called @(Foobar_builder_inputs) out of the additional dependencies, combine it into a string with semicolons as separators, and then pass that to a WriteLinesToFile task which then splits it into lines as semicolons. This is a bit roundabout, but I couldn't get it to work either by referencing %(AdditionalDependencies) directly or by doing the merging directly in the task attributes.

If you actually try this, you may run into another weakness of the dual dependency check setup, which is that the .tlog files themselves are written incrementally, and therefore there may be old dependencies in the files that screw up the dependency. When verifying a change to a build setup, a rebuild is prudent to ensure that the .tlog files are actually being generated correctly.

Conditional dependencies

At this point, VS2010 and MSBuild now agree on what needs to be built... both incorrectly. The hello.foo file is now building all of the time. The best way to diagnose this that I know of is to go to Tools | Options | Projects and Solutions | Build and Run and to set the MSBuild output verbosity to Diagnostic level. Unfortunately the output at this level is rather ridiculous and badly formatted, and mostly full of useless spew -- and when I say spew, I mean spew, because for a real project you may get over 50,000 lines of it! However, it's not too long on this test project, and a bit of searching reveals this line:

Input file "D:\projwin\VS2010-custombuildrule\[IncludeName].bar" does not exist.

Ugh. As with ExecutionDescription, the property tags are not being expanded in the AdditionalDependencies property either. Also, for some bass-ackward reason, MSBuild believes that a missing source warrants a rebuild of the target. (Making a build tool that writes to sources is usually a floggable offense.) We can try embedding the property expansion into the target:

<Target
  Name="_Foobar_builder"
  BeforeTargets="$(Foobar_builderBeforeTargets)"
  AfterTargets="$(Foobar_builderAfterTargets)"
  Condition="'@(Foobar_builder)' != ''"
  DependsOnTargets="$(Foobar_builderDependsOn);ComputeFoobar_builderOutput"
  Outputs="%(Foobar_builder.Outputs)"
  Inputs="%(Foobar_builder.Identity);
%(Foobar_builder.RootDir)%(Foobar_builder.Directory)%(Foobar_builder.IncludeName).bar;
%(Foobar_builder.RootDir)%(Foobar_builder.Directory)%(Foobar_builder.IncludeName).blah;
$(MSBuildProjectFile)"
>

This solves the problem, but introduces a new one:

Input file "D:\projwin\VS2010-custombuildrule\main.blah" does not exist.

The main.blah file doesn't exist, and unlike VS2005/2008, this trips a rebuild. What we need to do is inject this dependency only if that file actually exists. We can't do the existence check in the Inputs property of the target, so we'll have to back that change out, and instead dynamically generate the %(AdditionalDependencies) property from an earlier target:

<Target
  Name="ComputeFoobar_builderOutput"
  Condition="'@(Foobar_builder)' != ''">
<ItemGroup>
  <Foobar_builder>
    <AdditionalDependencies>%(RootDir)%(Directory)%(IncludeName).bar</AdditionalDependencies>
  </Foobar_builder>
  <Foobar_builder Condition="Exists(@(Foobar_builder->'%(RootDir)%(Directory)%(IncludeName).blah'))">
    <AdditionalDependencies>%(AdditionalDependencies);%(RootDir)%(Directory)%(IncludeName).blah</AdditionalDependencies>
  </Foobar_builder>
</ItemGroup>

Check the VS2010-custombuildrule.read.1.tlog file, and the dependencies should now be getting generated correctly whether the main.blah file is present or not. (Remember to rebuild after adding or removing that file, so that the .tlog files are regenerated.)

The final result

At this point, the VS2010 project should be building cleanly with working dependency checking. For a copy of the fixed version of the project: http://www.virtualdub.org/downloads/CustomBuildRule-VS2010.zip.