c++ - Redirecting cout to a console in windows

Question

I have an application which is a relatively old. Through some minor changes, it builds nearly perfectly with Visual C++ 2008. One thing that I've noticed is that my "debug console" isn't quite working right. Basically in the past, I've use AllocConsole() to create a console for my debug output to go to. Then I would use freopen to redirect stdout to it. This worked perfectly with both C and C++ style IO.

Now, it seems that it will only work with C style IO. What is the proper way to redirect things like cout to a console allocated with AllocConsole()?

Here's the code which used to work:

if(AllocConsole()) {
    freopen("CONOUT$", "wt", stdout);
    SetConsoleTitle("Debug Console");
    SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_RED);
}

EDIT: one thing which occurred to me is that I could make a custom streambuf whose overflow method writes using C style IO and replace std::cout's default stream buffer with it. But that seems like a cop-out. Is there a proper way to do this in 2008? Or is this perhaps something that MS overlooked?

EDIT2: OK, so I've made an implementaiton of the idea I spelled out above. Basically it looks like this:

class outbuf : public std::streambuf {
public:
    outbuf() {
        setp(0, 0);
    }

    virtual int_type overflow(int_type c = traits_type::eof()) {
        return fputc(c, stdout) == EOF ? traits_type::eof() : c;
    }
};

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPTSTR lpCmdLine, int nCmdShow) {
    // create the console
    if(AllocConsole()) {
        freopen("CONOUT$", "w", stdout);
        SetConsoleTitle("Debug Console");
        SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_RED);  
    }

    // set std::cout to use my custom streambuf
    outbuf ob;
    std::streambuf *sb = std::cout.rdbuf(&ob);

    // do some work here

    // make sure to restore the original so we don't get a crash on close!
    std::cout.rdbuf(sb);
    return 0;
}

Anyone have a better/cleaner solution than just forcing std::cout to be a glorified fputc?

Answer 1

Updated Feb 2018:

Here is the latest version of a function which fixes this problem:

void BindCrtHandlesToStdHandles(bool bindStdIn, bool bindStdOut, bool bindStdErr)
{
    // Re-initialize the C runtime "FILE" handles with clean handles bound to "nul". We do this because it has been
    // observed that the file number of our standard handle file objects can be assigned internally to a value of -2
    // when not bound to a valid target, which represents some kind of unknown internal invalid state. In this state our
    // call to "_dup2" fails, as it specifically tests to ensure that the target file number isn't equal to this value
    // before allowing the operation to continue. We can resolve this issue by first "re-opening" the target files to
    // use the "nul" device, which will place them into a valid state, after which we can redirect them to our target
    // using the "_dup2" function.
    if (bindStdIn)
    {
        FILE* dummyFile;
        freopen_s(&dummyFile, "nul", "r", stdin);
    }
    if (bindStdOut)
    {
        FILE* dummyFile;
        freopen_s(&dummyFile, "nul", "w", stdout);
    }
    if (bindStdErr)
    {
        FILE* dummyFile;
        freopen_s(&dummyFile, "nul", "w", stderr);
    }

    // Redirect unbuffered stdin from the current standard input handle
    if (bindStdIn)
    {
        HANDLE stdHandle = GetStdHandle(STD_INPUT_HANDLE);
        if(stdHandle != INVALID_HANDLE_VALUE)
        {
            int fileDescriptor = _open_osfhandle((intptr_t)stdHandle, _O_TEXT);
            if(fileDescriptor != -1)
            {
                FILE* file = _fdopen(fileDescriptor, "r");
                if(file != NULL)
                {
                    int dup2Result = _dup2(_fileno(file), _fileno(stdin));
                    if (dup2Result == 0)
                    {
                        setvbuf(stdin, NULL, _IONBF, 0);
                    }
                }
            }
        }
    }

    // Redirect unbuffered stdout to the current standard output handle
    if (bindStdOut)
    {
        HANDLE stdHandle = GetStdHandle(STD_OUTPUT_HANDLE);
        if(stdHandle != INVALID_HANDLE_VALUE)
        {
            int fileDescriptor = _open_osfhandle((intptr_t)stdHandle, _O_TEXT);
            if(fileDescriptor != -1)
            {
                FILE* file = _fdopen(fileDescriptor, "w");
                if(file != NULL)
                {
                    int dup2Result = _dup2(_fileno(file), _fileno(stdout));
                    if (dup2Result == 0)
                    {
                        setvbuf(stdout, NULL, _IONBF, 0);
                    }
                }
            }
        }
    }

    // Redirect unbuffered stderr to the current standard error handle
    if (bindStdErr)
    {
        HANDLE stdHandle = GetStdHandle(STD_ERROR_HANDLE);
        if(stdHandle != INVALID_HANDLE_VALUE)
        {
            int fileDescriptor = _open_osfhandle((intptr_t)stdHandle, _O_TEXT);
            if(fileDescriptor != -1)
            {
                FILE* file = _fdopen(fileDescriptor, "w");
                if(file != NULL)
                {
                    int dup2Result = _dup2(_fileno(file), _fileno(stderr));
                    if (dup2Result == 0)
                    {
                        setvbuf(stderr, NULL, _IONBF, 0);
                    }
                }
            }
        }
    }

    // Clear the error state for each of the C++ standard stream objects. We need to do this, as attempts to access the
    // standard streams before they refer to a valid target will cause the iostream objects to enter an error state. In
    // versions of Visual Studio after 2005, this seems to always occur during startup regardless of whether anything
    // has been read from or written to the targets or not.
    if (bindStdIn)
    {
        std::wcin.clear();
        std::cin.clear();
    }
    if (bindStdOut)
    {
        std::wcout.clear();
        std::cout.clear();
    }
    if (bindStdErr)
    {
        std::wcerr.clear();
        std::cerr.clear();
    }
}

In order to define this function, you'll need the following set of includes:

#include <windows.h>
#include <io.h>
#include <fcntl.h>
#include <iostream>

In a nutshell, this function synchronizes the C/C++ runtime standard input/output/error handles with the current standard handles associated with the Win32 process. As mentioned in the documentation, AllocConsole changes these process handles for us, so all that's required is to call this function after AllocConsole to update the runtime handles, otherwise we'll be left with the handles that were latched when the runtime was initialized. Basic usage is as follows:

// Allocate a console window for this process
AllocConsole();

// Update the C/C++ runtime standard input, output, and error targets to use the console window
BindCrtHandlesToStdHandles(true, true, true);

This function has gone through several revisions, so check the edits to this answer if you're interested in historical information or alternatives. The current answer is the best solution to this problem however, giving the most flexibility and working on any Visual Studio version.