How to stop command line from spawning separately from the pseudo console?

grease 0 Reputation points
2024-05-14T17:23:25.8266667+00:00

As the title suggests, I am trying to create a pseudo console with a command line process inside of it. I've been following the microsoft guide on creating a pseudo console session step by step, except I ported the code to Rust. Unfortunately, I have been dealing with a problem. For some reason, CreateProcessW is spawning the command line separately from the pseudo console, a whole new visible window pops up, and closing the Handles (including that of the pseudo console) does absolutely nothing to it. Similarly, the output of the console is empty.


use std::{ffi::OsStr, os::windows::ffi::OsStrExt};
use windows::{
    core::{Error, PWSTR},
    Win32::{
        Foundation::{CloseHandle, HANDLE},
        System::{
            Console::{ClosePseudoConsole, CreatePseudoConsole, COORD, HPCON},
            Memory::{GetProcessHeap, HeapAlloc},
            Pipes::CreatePipe,
            Threading::{
                CreateProcessW, InitializeProcThreadAttributeList, UpdateProcThreadAttribute,
                EXTENDED_STARTUPINFO_PRESENT, LPPROC_THREAD_ATTRIBUTE_LIST, PROCESS_INFORMATION,
                PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, STARTUPINFOEXW,
            },
        },
    },
};
pub struct PTY {
    input_read_side: HANDLE,
    input_write_side: HANDLE,
    output_read_side: HANDLE,
    output_write_side: HANDLE,
    pi: PROCESS_INFORMATION,
    hpcon: HPCON,
}
impl PTY {
    pub unsafe fn new(size: COORD, cmd_line: &OsStr) -> Result<Self, Error> {
        let (mut input_read_side, mut input_write_side) = (HANDLE::default(), HANDLE::default());
        CreatePipe(&mut input_read_side, &mut input_write_side, None, 0)?;
        let (mut output_read_side, mut output_write_side) = (HANDLE::default(), HANDLE::default());
        CreatePipe(&mut output_read_side, &mut output_write_side, None, 0)?;
        let hpcon = CreatePseudoConsole(size, input_read_side, output_write_side, 0)?;
        let mut startup_info_ex = std::mem::zeroed::<STARTUPINFOEXW>();
        startup_info_ex.StartupInfo.cb = std::mem::size_of::<STARTUPINFOEXW>() as u32;
        let mut bytes_required: usize = 0;
        // This is expected to fail
        let _ = InitializeProcThreadAttributeList(
            LPPROC_THREAD_ATTRIBUTE_LIST::default(),
            1,
            0,
            &mut bytes_required,
        );
        let hheap = GetProcessHeap()?;
        startup_info_ex.lpAttributeList = LPPROC_THREAD_ATTRIBUTE_LIST(HeapAlloc(
            hheap,
            windows::Win32::System::Memory::HEAP_FLAGS(0),
            bytes_required,
        ));
        InitializeProcThreadAttributeList(
            startup_info_ex.lpAttributeList,
            1,
            0,
            &mut bytes_required,
        )?;
        UpdateProcThreadAttribute(
            startup_info_ex.lpAttributeList,
            0,
            PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE as usize,
            Some((&hpcon as *const _) as *const _),
            std::mem::size_of::<HPCON>(),
            None,
            None,
        )?;
        let mut cmd_line = cmd_line.encode_wide().chain(Some(0)).collect::<Vec<_>>();
        let mut pi = std::mem::zeroed::<PROCESS_INFORMATION>();
        CreateProcessW(
            None,
            PWSTR(cmd_line.as_mut_ptr()),
            None,
            None,
            false,
            EXTENDED_STARTUPINFO_PRESENT,
            None,
            None,
            &startup_info_ex.StartupInfo,
            &mut pi,
        )?;
        Ok(Self {
            input_read_side,
            input_write_side,
            output_read_side,
            output_write_side,
            pi,
            hpcon,
        })
    }
}

Windows API - Win32
Windows API - Win32
A core set of Windows application programming interfaces (APIs) for desktop and server applications. Previously known as Win32 API.
2,453 questions
{count} votes

1 answer

Sort by: Most helpful
  1. grease 0 Reputation points
    2024-05-15T15:49:50.7666667+00:00

    Okay. I fixed it. The Rust HPCON struct has a 0 field which is an isize, casting that isize to the *const c_void works.

    0 comments No comments