Output stdout to the console on Windows

NEW
Unassigned

Status

Firefox Build System
Mach Core
2 years ago
3 months ago

People

(Reporter: padenot, Unassigned)

Tracking

(Depends on: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Comment hidden (empty)
(Reporter)

Comment 1

2 years ago
Created attachment 8731270 [details] [diff] [review]
patch

This looks like the right thing to do, but does not work, maybe something else is needed ?
Assignee: nobody → padenot
Attachment #8731270 - Flags: review?(ted)

Comment 3

2 years ago
What is the problem here?  Is this the known issue about e10s sandboxing blocking the console output on child process?
(Reporter)

Comment 4

2 years ago
Oh yes that might be it. That said, I mentioned on IRC, doing `| tee` after `./mach run` makes it work.
I assume that makes it work because the output is a pipe, not a console, then? bkelly: is there a bug filed on that issue?
Comment on attachment 8731270 [details] [diff] [review]
patch

Review of attachment 8731270 [details] [diff] [review]:
-----------------------------------------------------------------

Since this doesn't fix the issue right now it's clearly not worth landing. If we find a bug on the e10s console bit, we could make it block this. Alternately, we could change the invocation part to not pass `pass_thru=True` when we aren't trying to run under a debugger:
https://dxr.mozilla.org/mozilla-central/rev/494289c72ba3997183e7b5beaca3e0447ecaf96d/python/mozbuild/mozbuild/mach_commands.py#1243
Attachment #8731270 - Flags: review?(ted)

Comment 7

2 years ago
So this is officially Really Annoying, to the point that people file stack overflow issues ( http://stackoverflow.com/questions/36720019/printf-to-console-on-firefox-build/36766260 ) and bemusedly ask around on IRC. Pinging some engineering productivity folks - can we get someone to look at this and fix whatever is breaking it?
Flags: needinfo?(ted)
Flags: needinfo?(gps)
Flags: needinfo?(armenzg)

Updated

2 years ago
Depends on: 1189223

Comment 8

2 years ago
I have waited to see someone would have some idea of what is going on. I can't tell from the bug what could be.
What are the steps to reproduce?
Would calling Firefox in a command line have the same problem as running it through mach run?
Flags: needinfo?(armenzg)

Comment 9

2 years ago
Armen, this is a known issue with sandboxing on windows.  Please see bug 1189223 which I marked as a dependency above.

Comment 10

2 years ago
Thanks Ben.

Gijs, can we remove the NIs?

Comment 11

2 years ago
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #10)
> Thanks Ben.
> 
> Gijs, can we remove the NIs?

From what I saw last week I don't think Ben is correct. In particular, I think the behaviour with and without ./mach run is currently different. I'm mid-build; I'll update the bug with more details in the next 10 minutes.

Comment 12

2 years ago
(In reply to :Gijs Kruitbosch from comment #11)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #10)
> > Thanks Ben.
> > 
> > Gijs, can we remove the NIs?
> 
> From what I saw last week I don't think Ben is correct. In particular, I
> think the behaviour with and without ./mach run is currently different. I'm
> mid-build; I'll update the bug with more details in the next 10 minutes.

So I was wrong - --console and -console don't work regardless. I presume that's the sandboxing issue in bug 1189223.

However, -attach-console works, but --attach-console doesn't, and as the patch on this bug suggests, it should really be the default for ./mach run, rather than something we have to pass just to get working logs.

Note that all this output shows up when running mochitests, but doesn't when running with ./mach run, which is confusing.

Comment 13

2 years ago
Maybe it has to do if the Python logger is properly initialized at the right time.
I don't know if this is what is happening but it could be if running mochitest does it. However, I don't know why this would not be an issue for other platforms.

Where's the entry point for |./mach run|?

If we're calling run_process [1], then it would be a matter to know if mochitests also use that way of executing Firefox. 
Maybe mozprocess [2] is used instead.

This piece of code shows what happens if the Python logger is not set up the right time.

> import logging
> import sys
> 
> logger = logging.getLogger()
> logger.info("This won't print") # This won't fail
> 
> logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
> 
> logger.info("This will print")

[1] https://dxr.mozilla.org/mozilla-central/source/python/mach/mach/mixin/process.py#43
[2] https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess
(In reply to :Gijs Kruitbosch from comment #12)
> However, -attach-console works, but --attach-console doesn't, and as the
> patch on this bug suggests, it should really be the default for ./mach run,
> rather than something we have to pass just to get working logs.

Does the patch on this bug work? If so, we should just land it. When it was written padenot said it did not fix the issue.

> Note that all this output shows up when running mochitests, but doesn't when
> running with ./mach run, which is confusing.

When running tests, we redirect the browser's output to a pipe so we can parse test status messages from stdout. Redirecting stdout fixes this issue because it's a pipe, not a console. You can test this locally by running `./mach run | tee`. `mach run` doesn't redirect output by default (but it could, if that would be the best fix for this issue).
Flags: needinfo?(ted)
What Ted said.
Flags: needinfo?(gps)

Comment 16

2 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] (Vacation May 5th-11th) from comment #14)
> > Note that all this output shows up when running mochitests, but doesn't when
> > running with ./mach run, which is confusing.
> 
> When running tests, we redirect the browser's output to a pipe so we can
> parse test status messages from stdout. Redirecting stdout fixes this issue
> because it's a pipe, not a console. You can test this locally by running
> `./mach run | tee`. `mach run` doesn't redirect output by default (but it
> could, if that would be the best fix for this issue).

I don't know if that's the "best fix" - I'm not a domain expert on pipes and what is "expected" stdout/stderr behaviour. It just strikes me as odd that things "just work" on Linux and OS X and don't on Windows. As far as I'm concerned, we should do whatever it takes to make things "Just Work" on Windows, too. From what you're saying mach run doesn't redirect output, so how come it "just works" on Linux/OSX, then? Am I just misunderstanding what that means?

(ni to :gps because ted's on vacation.)
Flags: needinfo?(gps)
`mach run` passes the original {stdin,stderr,stdout} file descriptors to firefox.exe. So firefox.exe is bound to a terminal/console and can do magic there. Other mach commands (like the ones that launch tests) typically start firefox via mozprocess, which uses pipes for the standard file descriptors (and probably doesn't provide a stdin).

There are various reasons `mach run` doesn't use mozprocess. One is buffering. Another is debugging. We've had issues where interactive debuggers work best if there is no intermediate mechanism between firefox and the terminal.

So, `mach run` needs to continue using the terminal's file descriptors (not pipes). The proper fix is to use something like the code linked in comment #6 to get the Firefox processes to properly attach and write to the Windows console in e-10s mode.

Alternatively, we /might/ be able to use mozprocess when we're not using a debugger. But IIRC there are still issues with pipes with things like running inside of emacs. There have been enough bugs filed over the years regarding `mach run` standard file descriptor behavior that I'd rather not change `mach run`.
Flags: needinfo?(gps)
(Reporter)

Updated

2 years ago
Assignee: padenot → nobody
(In reply to :Gijs Kruitbosch from comment #16)
> I don't know if that's the "best fix" - I'm not a domain expert on pipes and
> what is "expected" stdout/stderr behaviour. It just strikes me as odd that
> things "just work" on Linux and OS X and don't on Windows. As far as I'm
> concerned, we should do whatever it takes to make things "Just Work" on
> Windows, too. From what you're saying mach run doesn't redirect output, so
> how come it "just works" on Linux/OSX, then? Am I just misunderstanding what
> that means?

Windows and Linux/OSX differ in the way they handle stdio. On Windows, we build Firefox as a GUI application, which means it doesn't get a "console" by default, so you don't see a Windows console when you run it (which is the right thing). Unfortunately, when you run a GUI application *from* a console on Windows, you *still* don't get a console by default, so your stdio goes nowhere. We have supported a `-console` argument for a long time that does force us to create one, but this doesn't do what you want when you're running from an existing console (it pops up a brand-new one). We attempted to make Firefox do the right thing when you ran it from an existing console some time ago (bug 673383), but this broke some users' use cases (running Firefox from batch files), so we instead added `-attach-console` (bug 787313) which is sort of a crummy workaround.

`-attach-console` *should* produce the desired behavior (attach stdio to the current console), and is what padenot's patch implements in `mach run`, but padenot indicated that it did not work, which might just be bug 1189223.

Comment 19

a year ago
I'm using Thunderbird and this is now working for me starting from version 53.0a1 (Build ID 20170115030207) while it wasn't working until Thunderbird 52.

Updated

4 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.