583223 - [PATCH] PR_CreateProcess sholdn't open a command window

Status: RESOLVED FIXED

(NSPR :: NSPR, defect)

Description

[Phil Dibowitz]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.7) Gecko/20100716 Ubuntu/10.04 (lucid) Firefox/3.6.7
Build Identifier: 

Windows executables built as Console Applications will always get a console window opened for them unless the CREATE_NO_WINDOW flag is passed to the CreateProcess() system call. See:

http://msdn.microsoft.com/en-us/library/ms684863(VS.85).aspx

A common NSPR use case is to call PR_CreateProcess() on an executable that isn't expected to present any user interface of its own (for example because they write to stdout, which has been redirected to a pipe shared with the parent process). These executables often built as Console Applications and will currently pop up an empty console window (which then immediately disappears again as soon as the process terminates). It is, obviously, undesirable to have empty windows flashing on screen.

[ side note: I'm creating this bug on behalf of Fredrik Roubert who's having trouble accessing bugzilla at the moment but will add himself as a CC later. ]

Reproducible: Always

Steps to Reproduce:
1. Use PR_CreateProcess()
Actual Results:  
A command prompt window pops up each time.

Expected Results:  
A command prompt window does not pop up.

Comment 1

[Fredrik Roubert]

Attachment: Set the CREATE_NO_WINDOW flag automatically if stdout is redirected.

This is one possible way to solve this problem: If stdout has been redirected we can assume that the process will not write anything useful to the console window, and therefore automatically set the CREATE_NO_WINDOW flag.

Comment 2

[Fredrik Roubert]

Attachment: Add a field to PRProcessAttr to control the CREATE_NO_WINDOW flag.

This is another way to solve the problem: Add a field to the PRProcessAttr structure to let the user of the library control whether the CREATE_NO_WINDOW flag should be set or not.

This would allow more fine-grained control, and enable for example the following scenarios:

1. The executable doesn't write anything at all to stdout, so there is no point in redirecting stdout, but it does have some other valuable side-effect and no-one wants to see an empty window flashing on screen.

2. The executable does write to stdout, but this shouldn't be shown to the user and just setting this flag to not open any console window is easier than redirecting stdout to NUL:.

Comment 3

[Fredrik Roubert]

OK, I've attached two different patches to this bug report. Each one of them solves the problem from my point-of-view, but they have different pros and cons so if you like one of them better than the other then please just apply that one.

Comment 4

[Wan-Teh Chang]

Thanks for the patches.  I tend to like NSPR doing the
right thing automatically, so I'd prefer the first approach
unless people find a problem with it.  Using the redirection
of stdout seems like a safe way to determine if the app needs
the console window.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Is this problem peculiar to Windows 7? or a particular version of MSVC?
I do NOT see this problem on WinXP with VC 2005.

Comment 6

[Wan-Teh Chang]

I don't think it's perculiar to a particular version of
Windows or MSVC.

By the way, I accidentially changed the "OS" field of this
bug from Windows 7 to Windows XP.  I've changed it back.

Comment 7

[Phil Dibowitz]

I'm nearly positive Fredrik is developing on XP. When I set the bug to Windows 7 it was a mistake.

Comment 8

[Fredrik Roubert]

The automatic creation of console windows for Console Application executables is a basic feature of the Win32 API, so it is present in all versions of MS Windows (and all other implementations of the Win32 API as well, eg. WINE). See the MSDN article mentioned for more details.

Comment 9

[Wan-Teh Chang]

Attachment: Set the CREATE_NO_WINDOW flag automatically if stdout is redirected, v2, by Fredrik Roubert (checked in)

I made two edits (declaring the creationFlags variable for the
WINCE case as well, and adding a comment to explain why we set
the CREATE_NO_WINDOWS flag automatically if stdout is redirected).
I want the WINCE case to be as close to the non-WINCE case as
possible.

Checked in on the NSPR trunk (NSPR 4.8.7).

Checking in ntmisc.c;
/cvsroot/mozilla/nsprpub/pr/src/md/windows/ntmisc.c,v  <--  ntmisc.c
new revision: 3.30; previous revision: 3.29
done