Converting Win32 API results to std::wstring (or std::string)

Hmm. Just realized that this is a bit out of order and should have been published before the previous post .

It turns out that there are a significant number of Win32 APIs that have a similar calling pattern – you call the API once to find the size of the buffer needed for the result, allocate a buffer then fill in the resulting buffer. Examples of APIs like this are GetCurrentDirectory, GetEnvironmentVariable, ExpandEnvironmentString, MultiByteToWideChar, etc. The pattern shows up over and over again in the Win32 API surface. It would be really useful if it was possible to define a fairly standard pattern that results in the creation of a std::wstring from one of these APIs (it’s probably not possible to create a template to handle all the possible APIs but that’s ok, defining a working pattern is probably OK). One of the key aspects of such a pattern is that I only ever want to perform one allocation during the call. Otherwise I could simply use malloc to allocate a buffer, fill in the buffer and then construct a new std::wstring for that pattern.

I did what every developer does in this situation and searched the internet and I found a solution that seemed to work correctly, but I wasn’t really happy with what I came up with (NOTE: DO NOT USE THIS CODE):

 std::wstring UnicodeStringFromAnsiString(_In_ const std::string &ansiString)
{
    std::wstring unicodeString;

    auto wideCharSize = MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, ansiString.c_str(), -1, nullptr, 0);
    if (wideCharSize == 0)
    {
        return L"";
    }
    unicodeString.reserve(wideCharSize);
    unicodeString.resize(wideCharSize-1);

    wideCharSize = MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, ansiString.c_str(), -1, &unicodeString[0], wideCharSize);

    return unicodeString;
}

There are a couple of issues with this code (ignore the fact that it doesn’t handle errors particularly well). The first is that it is inconsistent in its handling of the failure case, I’m also not happy with the “reserve”/”resize” aspect of the result.

So I asked the owners of the C++ runtime library on the VC team what they would suggest and they pointed out a huge issue with my solution.

The code I found above worked but it ignored at least one really important aspect of the implementation of the std::wstring type (and the std::string type). It turns out that the null terminating character in the std::wstring is owned by the STL – are code outside the STL is not allowed to mess with that null terminating character, and that’s exactly what happens when I wrote .reserve (to allocate storage) and .resize (to resize the buffer so it doesn’t contain the trailing null). There’s a more significant problem with this code however. By returning both a named value AND a temporary, it has the side effect of disabling the NRV optimization, which can result in significant performance degradation.

Instead of my example, they suggested instead that I change the code as follows (follow this pattern instead):

 std::wstring UnicodeStringFromAnsiString(_In_ const std::string &ansiString)
{
    std::wstring returnValue;
    auto wideCharSize = MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, ansiString.c_str(), -1, nullptr, 0);
    if (wideCharSize == 0)
    {
        return returnValue;
    }
    returnValue.resize(wideCharSize);
    wideCharSize = MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, ansiString.c_str(), -1, &returnValue[0], wideCharSize);
    if (wideCharSize == 0)
    {
        returnValue.resize(0);
        return returnValue;
    }
    returnValue.resize(wideCharSize-1);
    return returnValue;
}

The key difference between this version and the previous is that we only allocate a new string once we’ve calculated the size and we initialize the string to the needed buffer size. We then fill the buffer (using the array operator [] to gain access to a non const buffer for the string) and finally resize the string to the actual size of the string. By performing the operations in this order, we ensure that we never overwrite the STL’s null character – it always lies one character beyond the end of the buffer that we’re filling in. And because a resize that decreases the size of a buffer never allocates a new buffer, we preserve the desired behavior that we only ever allocate one buffer. And finally, this version only returns the same local variable (thus enabling NRVO).

One final note: This technique only works when the output of the Win32 call is relatively static. If you’re retrieving data whose size can change on the fly, it’s probably best to add a loop around the API call to ensure that the buffer is large enough.