[meta] Headless browsing mode

REOPENED
Assigned to

Status

()

Firefox
Headless
REOPENED
4 months ago
2 days ago

People

(Reporter: bdahl, Assigned: bdahl)

Tracking

(Depends on: 2 bugs, {dev-doc-needed, leave-open})

unspecified
Firefox 55
dev-doc-needed, leave-open
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(1 attachment, 9 obsolete attachments)

(Assignee)

Description

4 months ago
This will be the tracking bug for headless browsing. The first overall goal of headless browsing is to make automated testing of Firefox less painful for developers. One of the first steps towards this will be to support a headless mode in Firefox that can run slimer.js on linux without an X server. 

The initial plan is to have either a command line flag or environment variable that starts Firefox in a mode that doesn’t call into gtk and creates enough stubs for pages to load and be rendered by a software renderer. 

In a previous effort to make Firefox headless (bug 446591) a completely new headless widget and shell were created. The new approach will be slightly different in that headless won’t be it’s own shell/widget, but instead a mode that is supported by all platforms. The benefits of this approach are: any Firefox becomes testable, headless Firefox will closely match release versions, binaries are already being built and distributed, and this will hopefully align with work Chrome is doing (making cross browser testing easier).
(Assignee)

Comment 1

3 months ago
Created attachment 8838813 [details] [diff] [review]
Prototype headless browser mode.
(Assignee)

Comment 2

3 months ago
Created attachment 8840239 [details] [diff] [review]
Prototype headless browser mode.
(Assignee)

Updated

3 months ago
Attachment #8838813 - Attachment is obsolete: true
(Assignee)

Comment 3

3 months ago
Comment on attachment 8840239 [details] [diff] [review]
Prototype headless browser mode.

Hey Jeff,
I've got some basics tests working and I'll have another patch that introduces a headless mode for xpcshell tests. Overall, I've found myself creating a few stubs and having to scatter around some checks to avoid calling into gdk/X. I'm not crazy about sprinkling these 'ifs', but I'm not seeing a better way since a lot of the calls are in only one small spot in a class and stubbing the whole class to avoid it would cause a lot of duplication. I'm looking for some general feedback if this approach is looking acceptable.

Some other notes/todos:
 - I haven't got full Firefox running in headless mode yet, I've been focusing on slimerjs (which works).
 - Need to get window resizing working
 - Add support for a headless flavor of xpcshell tests (and maybe mochitest)
 - Add --headless flag for FF
 - Sort out popup and new windows
Attachment #8840239 - Flags: feedback?(jmuizelaar)
(In reply to Brendan Dahl [:bdahl] from comment #3)
> Comment on attachment 8840239 [details] [diff] [review]
> Prototype headless browser mode.

I've only taken a very quick look. But it seems reasonable. You should replace getenv("MOZ_HEADLESS") with gfxPlatform::IsHeadless() or something like that.
(Assignee)

Comment 5

3 months ago
Created attachment 8842596 [details] [diff] [review]
Add headless flavor of xpcshell tests.
(Assignee)

Comment 6

3 months ago
Comment on attachment 8842596 [details] [diff] [review]
Add headless flavor of xpcshell tests.

Hi Ted,
It looks like you're a peer/owner of all the overlapping pieces for this patch. Let me know if there's someone better or if you want me to break out some of the changes.

Try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d9769cc03563fe680cbe9974af48d53026f9e35
Attachment #8842596 - Flags: review?(ted)
Comment on attachment 8840239 [details] [diff] [review]
Prototype headless browser mode.

Review of attachment 8840239 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, this doesn't seem so ugly and has some potentially for cleaning up.
Attachment #8840239 - Flags: feedback?(jmuizelaar) → feedback+
I dunno if you've seen info on the equivalent work that's happening in Chromium, but it could be informative:
https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md
https://bugs.chromium.org/p/chromium/issues/detail?id=546953

I'll review this patch soon, exciting to see someone working on this again! I love the plan to make it work with our stock Firefox builds--that makes it a lot more useful. I think having Marionette built-in helps shortcut a lot of the issues around embedding that previous efforts hit, since you ought to be able to just run a headless instance and drive it via Marionette.
(Assignee)

Comment 9

3 months ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #8)
> I dunno if you've seen info on the equivalent work that's happening in
> Chromium, but it could be informative:

Yeah, I've been following some of their progress. It looks like they also plan support webdriver, so in the future it may be much easier to write cross browser headless tests.
Comment on attachment 8842596 [details] [diff] [review]
Add headless flavor of xpcshell tests.

Review of attachment 8842596 [details] [diff] [review]:
-----------------------------------------------------------------

I wonder whether we actually need a separate test manifest type + mach command and all that. If the harness just needs to execute xpcshell with `MOZ_HEADLESS=1`, what if we just allowed xpcshell tests to specify `headless = true` in the xpcshell.ini manifest, like:
```
[test_foo.js]
headless = true
```

That seems like it'd work just as well and not require as much other fiddling. You could implement support for that by doing something like  `headless = test_object.get('headless', false)` here:
https://dxr.mozilla.org/mozilla-central/rev/8d026c60151005ad942e3d4389318fe28a0c8c54/testing/xpcshell/runxpcshelltests.py#1322

Then pass the `headless` var down to the test execution and set the env vars appropriately. If you want to run the headless tests only on Linux for now you could special-case that there as well.

If you really feel that these tests ought to run as a separate test job, you could use a subsuite for this in the test manifest. We use that for some other test types, but it'd let us keep the tests in regular xpcshell test manifests, just run them as separate jobs.
Attachment #8842596 - Flags: review?(ted)
(Assignee)

Comment 11

3 months ago
Created attachment 8844245 [details] [diff] [review]
Add headless mode xpcshell tests.
(Assignee)

Comment 12

3 months ago
Comment on attachment 8844245 [details] [diff] [review]
Add headless mode xpcshell tests.

That approach is much easier! I had started down a similar path before, but I seem to have misunderstood and thought that multiple xpcshell tests ran per process and thought I'd have to do grouping anyway.

Having a separate headless xcpshell group does have a few advantages such as breaking up tests and eventually being able to run them on a truly headless environment. However, just having a headless attribute is much simper for now.
Attachment #8844245 - Flags: review?(ted)
(Assignee)

Updated

3 months ago
Attachment #8842596 - Attachment is obsolete: true
Comment on attachment 8844245 [details] [diff] [review]
Add headless mode xpcshell tests.

Review of attachment 8844245 [details] [diff] [review]:
-----------------------------------------------------------------

That's much simpler! :)
Attachment #8844245 - Flags: review?(ted) → review+
(Assignee)

Comment 14

3 months ago
Created attachment 8846112 [details] [diff] [review]
Add headless browser mode.

Supports creating a windowless browser on Linux without an X server. Most of the
changes are just adding branches to avoid calls in to GTK which calls
into X. Some of the bigger additions were adding a separate headless look and
feel and a headless screen manager. These were added since there
were many calls into GTK and in the future we can re-use these on other
platforms.
(Assignee)

Updated

3 months ago
Attachment #8840239 - Attachment is obsolete: true
(Assignee)

Comment 15

3 months ago
Created attachment 8846115 [details] [diff] [review]
Support re-sizing a windowless browser.
(Assignee)

Updated

3 months ago
Attachment #8846112 - Flags: review?(jmuizelaar)
(Assignee)

Comment 16

3 months ago
Comment on attachment 8846115 [details] [diff] [review]
Support re-sizing a windowless browser.

I'm not quite sure the proper way to support resizing a windowless browser. This approach seemed the cleanest, but there are other ways too. 

Originally, I didn't have the changes to the widget code and only called window->SetSize in WebBrowserChrome2Stub::SetDimensions, but this didn't cause the window to actually be resized. To fix it, I added the IsPuppetWidget to basically cause mAttachedToParent to be set to true which in turn causes us to follow the code path in http://searchfox.org/mozilla-central/rev/b035220d0f939559f4f8cf7b9079bc12f6adfc35/layout/base/nsDocumentViewer.cpp#1997 which calls nsViewManager::SetWindowDimensions with the proper dimensions. Alternatively, in WebBrowserChrome2Stub::SetDimensions I could  get the presshell, prescontext, and viewmanager and call nsViewManager::SetWindowDimensions directly. Or maybe there's a even better way?
Attachment #8846115 - Flags: review?(jmuizelaar)
Attachment #8846112 - Flags: review?(jmuizelaar) → review+
Comment on attachment 8846115 [details] [diff] [review]
Support re-sizing a windowless browser.

Review of attachment 8846115 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/base/nsDocumentViewer.cpp
@@ +4291,5 @@
>    if (!containerItem)
>      return false;
>  
>    // We always attach when using puppet widgets
> +  if (nsIWidget::UsePuppetWidgets() || mParentWidget->IsPuppetWidget())

When is nsIWidget::UsePuppetWidgets() false but mParentWidget->IsPuppetWidget() true?
(Assignee)

Comment 18

2 months ago
(In reply to Jeff Muizelaar [:jrmuizel] from comment #17)
> When is nsIWidget::UsePuppetWidgets() false but
> mParentWidget->IsPuppetWidget() true?

This can happen when we create a windowless browser in the parent process (http://searchfox.org/mozilla-central/rev/ca7015fa45b30b29176fbaa70ba0a36fe9263c38/xpfe/appshell/nsAppShellService.cpp#523). Alternatively, I could change UsePuppetWidgets() to do XRE_IsContentProcess() || gfxPlatform::IsHeadless()
(Assignee)

Comment 19

2 months ago
The comment above could still be a fix, but as I've started fixing tests in the slimer js test suite it's looking like creating a separate HeadlessWidget will be cleaner than abusing the PuppetWidget. Most of the code paths in PuppetWidget are safe and don't call to the tab child if it doesn't exist, but there are still a few that can crash (also I don't think the PuppetWidget was really designed to be used this way). So far in the HeadlessWidget I've had to implement just a few methods to get most things working. Any thoughts on this approach instead?
Flags: needinfo?(jmuizelaar)
Yes, please. Let's use a HeadlessWidget.
Flags: needinfo?(jmuizelaar)
(Assignee)

Comment 21

2 months ago
Created attachment 8852307 [details] [diff] [review]
Support re-sizing a windowless browser.
(Assignee)

Updated

2 months ago
Attachment #8846115 - Attachment is obsolete: true
Attachment #8846115 - Flags: review?(jmuizelaar)
(Assignee)

Comment 22

2 months ago
Created attachment 8852313 [details] [diff] [review]
Support re-sizing a windowless browser.
(Assignee)

Updated

2 months ago
Attachment #8852307 - Attachment is obsolete: true
(Assignee)

Comment 23

2 months ago
Comment on attachment 8852313 [details] [diff] [review]
Support re-sizing a windowless browser.

One thing I wasn't sure on is if I need to do anything during the HeadlessWidget::Invalidate.
Attachment #8852313 - Flags: review?(jmuizelaar)
(Assignee)

Comment 24

2 months ago
Created attachment 8853151 [details] [diff] [review]
Add headless browser mode.

Supports creating a windowless browser on Linux without an X server. Most of the
changes are just adding branches to avoid calls in to GTK which calls
into X. Some of the bigger additions were adding a separate headless look and
feel and a headless screen manager. These were added since there
were many calls into GTK and in the future we can re-use these on other
platforms.
(Assignee)

Updated

2 months ago
Attachment #8846112 - Attachment is obsolete: true
(Assignee)

Comment 25

2 months ago
Comment on attachment 8853151 [details] [diff] [review]
Add headless browser mode.

As discussed on IRC with jmuizelaar, did some clean up removing 'ns' prefix and moved things into mozilla::widget namespace. Carrying r+ forward.
Attachment #8853151 - Flags: review+
Attachment #8852313 - Flags: review?(jmuizelaar) → review+
(Assignee)

Comment 26

2 months ago
Created attachment 8854502 [details] [diff] [review]
Add headless browser mode. (for checkin)

Supports creating a windowless browser on Linux without an X server. Most of the
changes are just adding branches to avoid calls in to GTK which calls
into X. Some of the bigger additions were adding a separate headless widget
which implements just enough to render a page. A headless look and
feel were also added since there are many calls into GTK in the platform
specific one.
(Assignee)

Comment 27

2 months ago
The new patch is the squash of the above reviewed patches and resolving merge conflicts from the removal of ScreenManager.

Try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e86ed1de993f3c704021d880d4572428de2ac3be
Keywords: checkin-needed
Attachment #8853151 - Attachment is obsolete: true
Attachment #8844245 - Attachment is obsolete: true
Attachment #8852313 - Attachment is obsolete: true

Comment 28

2 months ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/23674d708903
Add headless browser mode. r=jrmuizel, r=ted
Keywords: checkin-needed
This change breaks xpcshell tests that create windowless browsers when I run them locally:

#0  0x00002aaab04818f9 in ?? () from /usr/lib/libgtk-3.so.0
#1  0x00002aaab03292a8 in ?? () from /usr/lib/libgtk-3.so.0
#2  0x00002aaab033dcf4 in ?? () from /usr/lib/libgtk-3.so.0
#3  0x00002aaab032a5ec in ?? () from /usr/lib/libgtk-3.so.0
#4  0x00002aaab033dbdc in ?? () from /usr/lib/libgtk-3.so.0
#5  0x00002aaab033dc74 in ?? () from /usr/lib/libgtk-3.so.0
#6  0x00002aaab032af52 in ?? () from /usr/lib/libgtk-3.so.0
#7  0x00002aaab1f0133f in g_type_create_instance () from /usr/lib/libgobject-2.0.so.0
#8  0x00002aaab1ee320b in ?? () from /usr/lib/libgobject-2.0.so.0
#9  0x00002aaab1ee4c1d in g_object_newv () from /usr/lib/libgobject-2.0.so.0
#10 0x00002aaab1ee53d4 in g_object_new () from /usr/lib/libgobject-2.0.so.0
#11 0x00002aaab0345ffa in ?? () from /usr/lib/libgtk-3.so.0
#12 0x00002aaab05154e7 in ?? () from /usr/lib/libgtk-3.so.0
#13 0x00002aaab1f0133f in g_type_create_instance () from /usr/lib/libgobject-2.0.so.0
#14 0x00002aaab1ee320b in ?? () from /usr/lib/libgobject-2.0.so.0
#15 0x00002aaab1ee511e in g_object_new_valist () from /usr/lib/libgobject-2.0.so.0
#16 0x00002aaab1ee53c1 in g_object_new () from /usr/lib/libgobject-2.0.so.0
#17 0x00002aaaad256468 in CreateScrollbarWidget (aWidgetType=MOZ_GTK_BUTTON, aOrientation=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:57
#18 CreateWidget (aWidgetType=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:550
#19 0x00002aaaad25d2dd in GetWidget (aWidgetType=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:628
#20 GetWidgetRootStyle (aNodeType=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:769
#21 0x00002aaaad25d36b in CreateChildCSSNode (aName=0x2aaaaeaaa25e "contents", aParentNodeType=2990728240) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:782
#22 0x00002aaaad2574bf in GetCssNodeStyleInternal (aNodeType=MOZ_GTK_SCROLLBAR_CONTENTS_VERTICAL) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:818
#23 CreateChildCSSNode (aParentNodeType=MOZ_GTK_SCROLLBAR_CONTENTS_VERTICAL, aName=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:782
#24 GetCssNodeStyleInternal (aNodeType=MOZ_GTK_SCROLLBAR_TROUGH_VERTICAL) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:822
#25 0x00002aaaad256ed5 in ClaimStyleContext (aNodeType=MOZ_GTK_SCROLLBAR_TROUGH_VERTICAL, aDirection=GTK_TEXT_DIR_NONE, aStateFlags=GTK_STATE_FLAG_NORMAL, aFlags=<optimized out>) at /home/kris/code/mozilla-central/widget/gtk/WidgetStyleCache.cpp:1131
#26 0x00002aaaad26c726 in nsLookAndFeel::EnsureInit (this=0x2aaabcc18090) at /home/kris/code/mozilla-central/widget/gtk/nsLookAndFeel.cpp:1175
#27 0x00002aaaad26c3d4 in nsLookAndFeel::NativeGetColor (this=0x2aaabcc18090, aID=mozilla::LookAndFeel::ColorID::eColorID_WindowBackground, aColor=@0x2aaac35da9a0: 0) at /home/kris/code/mozilla-central/widget/gtk/nsLookAndFeel.cpp:226
#28 0x00002aaaad23fc55 in nsXPLookAndFeel::GetColorImpl (this=0x2aaabcc18090, aID=mozilla::LookAndFeel::ColorID::eColorID_WindowBackground, aUseStandinsForNativeColors=false, aResult=<optimized out>) at /home/kris/code/mozilla-central/widget/nsXPLookAndFeel.cpp:822
#29 0x00002aaaadedb3f1 in nsWebBrowser::Create (this=0x2aaac35da880) at /home/kris/code/mozilla-central/toolkit/components/browser/nsWebBrowser.cpp:1208
#30 0x00002aaaaddc4fe2 in nsAppShellService::CreateWindowlessBrowser (this=<optimized out>, aIsChrome=<optimized out>, aResult=0x7fffffff84a8) at /home/kris/code/mozilla-central/xpfe/appshell/nsAppShellService.cpp:539

I'm not sure why it appears to work on inbound.
Could be a difference in the version of Gtk. IIRC the test machines are on Ubuntu 12.04.
(Assignee)

Comment 31

2 months ago
I'm also able to reproduce the failures locally. I thought I had successfully run several of the windowless browser tests locally (maybe they passed for the same reason try is working?). 

I was curious if maybe some of the tests were disabled on linux, but looks like they're running: js/xpconnect/tests/unit/test_nuke_sandbox_event_listeners.js in https://public-artifacts.taskcluster.net/cq7Igmc6TjK0BSJDPhwW7w/0/public/logs/live_backing.log

I'll look into this more when tomorrow when I have physical access to my linux desktop.
(Assignee)

Comment 32

2 months ago
Btw: If you need a quick fix revert nsappshellservice	
from:
  nsCOMPtr<nsIWidget> widget = nsIWidget::CreateHeadlessWidget();
to
  nsCOMPtr<nsIWidget> widget = nsIWidget::CreatePuppetWidget(nullptr);
Yup, already did. That's how I confirmed that this was the cause.

Comment 34

2 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/23674d708903
Status: NEW → RESOLVED
Last Resolved: 2 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 55
(Assignee)

Updated

2 months ago
Depends on: 1353814

Updated

2 months ago
Depends on: 1354068

Updated

2 months ago
Depends on: 1354166
Just a question from an enthusiastic of this feature.
It is available in Nightly?
What is the parameter? --headless?
Maybe a documentation on MDN will be helpful :-D

Comment 36

2 months ago
(In reply to Daniele "Mte90" Scasciafratte from comment #35)
> Just a question from an enthusiastic of this feature.
> It is available in Nightly?
> What is the parameter? --headless?
> Maybe a documentation on MDN will be helpful :-D

MOZ_HEADLESS=1 /path/firefox

Updated

2 months ago
Depends on: 1355036
Depends on: 1355047
(Assignee)

Updated

2 months ago
Depends on: 1355147
(Assignee)

Updated

2 months ago
No longer depends on: 1355047
(Assignee)

Updated

2 months ago
Depends on: 1355150
(Assignee)

Updated

2 months ago
No longer depends on: 1355036
(Assignee)

Updated

2 months ago
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Summary: Headless browsing mode → [meta] Headless browsing mode
(Assignee)

Updated

2 months ago
Depends on: 1353935
(Assignee)

Updated

a month ago
Depends on: 1356681

Updated

a month ago
Keywords: dev-doc-needed
Depends on: 1359374
(Assignee)

Updated

a month ago
No longer depends on: 1359374
(Assignee)

Updated

a month ago
Depends on: 1359480
(Assignee)

Updated

2 days ago
Component: General → Headless
You need to log in before you can comment on or make changes to this bug.