¶Watch out for number bases in Visual C++ visualizers
I just got burnt by an old bug -- err, feature -- in the Visual C++ debugger.
In the VC6 days, there was a nasty "feature" where the debugger evaluated unprefixed integers in expressions used the current number base setting. This meant that array[10] in the Watch window showed element 10 if you were in decimal mode and element 16 if you were in hexadecimal mode. You had to use the nonstandard "0n" prefix to force decimal, i.e. array[0n10]. Thankfully, this has been fixed and in the last few versions unprefixed numbers work as expected.
However, it turns out that this problem still occurs in autoexp.dat visualizers. My deque visualizer looks like this:
vdfastdeque<*>{
preview(
#(
"size=",
#if ($e.m.mapEnd == $e.m.mapStart)
(
[0]
)
#else
(
[($e.m.mapEnd-$e.m.mapStart-1)*32+$e.mTails.endIndex+1-$e.mTails.startIndex]
)
)
)
children(
#array(
expr: $c.m.mapStart[($i+$c.mTails.startIndex) >> 5]->data[($i+$c.mTails.startIndex) & 31],
size: ($c.m.mapEnd-$c.m.mapStart-1)*32+$c.mTails.endIndex+1-$c.mTails.startIndex
)
)
}vdfastdeque, like most STL deque implementations, stores elements in a sequence of fixed-size blocks with some portions of the first and last blocks possibly unused. Well, it took me a while to figure out that the elements I was seeing were bogus, but only when hexadecimal mode was enabled. The problem is the "& 31" in the array expr: field, which was turning into "& 0x31" in hex mode... argh!!
As expected, changing the visualizer to use 0n31 or 0x1f fixes the problem. I did some experimentation, and this only seems to affect cases where expressions are evaluated for display. In particular, it doesn't affect the size: field of #array, and probably not any conditions or values used by #if or #switch, which is why none of Microsoft's visualizers show this problem. The problem also reproduces in VC2010 Express, so it hasn't been fixed yet. That's not too surprising, since very little has changed in the visualizer engine except a couple of small bug fixes.
Anyway, today's conclusion: make sure you test your visualizers in hex mode, and prefix constants to avoid such problems.