Debug Assestion failure when using _setmode(_fileno(stdout), _O_U16TEXT);

Ioann 61

I want to enable console to print Unicode Characters so i used _setmode(_fileno(stdout), _O_U16TEXT);
But once i do that an exception is thrown where this function is:
SetWindowsHookExW(WH_KEYBOARD_LL, hookproc_keyboard, NULL, 0);

"Unhandled exception at 0x00007FFEB1641208 (ucrtbase.dll) in testprogram.exe:
An invalid parameter was passed to a function that considers invalid parameters fatal."

and a window pops up showing this

"File: minkernel\crts\ucrt\src\appcrt\stdio\fputc.cpp
Line: 49

Expression: ( (_Stream.is_string_backed()) || (fn = _fileno(_Stream.public_stream()), ((_textmode_safe(fn) == __crt_lowio_text_mode::ansi) && !_tm_unicode_safe(fn))))

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts."

Can someone explain to me what this error is ?
Thanks in advance

David Lowndes 4,711 Reputation points

2022-01-25T10:55:43.457+00:00

Are you outputting using the ANSI functions?
A while back I modified some of my console programs to get Unicode output and came to the finding that what worked best for me was to set the mode to _O_U8TEXT, set the console to code page 65001, and use the wide-character console functions such as _putws.
RLWA32 40,856 Reputation points

2022-01-25T11:49:50.947+00:00

"File: minkernel\crts\ucrt\src\appcrt\stdio\fputc.cpp

Line: 49

Expression: ( (_Stream.is_string_backed()) || (fn = _fileno(_Stream.public_stream()), ((textmode_safe(fn) == _crt_lowio_text_mode::ansi) && !_tm_unicode_safe(fn))))

From the documentation at fputc, fputwc -- "Each of these functions writes the single character c to a file at the position indicated by the associated file position indicator (if defined) and advances the indicator as appropriate. In the case of fputc and fputwc, the file is associated with stream. If the file cannot support positioning requests or was opened in append mode, the character is appended to the end of the stream.

The two functions behave identically if the stream is opened in ANSI mode. fputc does not currently support output into a UNICODE stream." Emphasis added.

It's not clear where the call to fputc is originating, but the above explains the assertion and related unhandled exception.
Ioann 61 Reputation points

2022-01-25T23:13:40.303+00:00

Does SetWindowsHookExW() call fputc? Cause In my code I don't use this function at all
David Lowndes 4,711 Reputation points

2022-01-25T23:56:29.21+00:00

No, but the assertion you showed is from the 'C' library function, not the Windows API.
When the assertion occurs, what's the call stack?
Maybe some code in your hook function is the offender?
Xiaopo Yang - MSFT 11,501 Reputation points Microsoft Vendor

2022-01-26T07:59:57.917+00:00

Even _setmode also said, Unicode mode is for wide print functions (for example, wprintf) and is not supported for narrow print functions. Use of a narrow print function on a Unicode mode stream triggers an assert. Perhaps It‘s preferred to show a minimal, reproducible sample.
Ioann 61 Reputation points

2022-01-26T09:21:10.427+00:00
I emptied all code from the hook function just to see if some code in your hook function is the offender but still the same thing happens, and when it comes to main that's all the code i am using

int main()
{
_setmode(_fileno(stdout), _O_U16TEXT); // this line is to enable console to print unicode characters but causes an assertion failure if enabled

cout<<GetKeyboardLayout(0); SetWindowsHookExW(WH_KEYBOARD_LL, hookproc_keyboard, NULL, 0); MSG msg; while (!GetMessage(&msg, NULL, NULL, NULL)){ TranslateMessage(&msg); DispatchMessage(&msg); }

}
Ioann 61 Reputation points

2022-01-26T09:24:07.357+00:00

alright. Issue found. The problem was the cout<<GetKeyboardLayout(0); inside then main function . I commented it out and the program worked. I dont understand why though but it worked.
It's strange cause it the point where the exception is thrown is the
SetWindowsHookExW(WH_KEYBOARD_LL, hookproc_keyboard, NULL, 0);
and the cout<<GetKeyboardLayout(0);
is before it.
Xiaopo Yang - MSFT 11,501 Reputation points Microsoft Vendor

2022-01-26T09:37:27.647+00:00

Do you know std::cout?
Ioann 61 Reputation points

2022-01-26T09:41:34.327+00:00

sure .i now tried wcout and it worked as well