Open Bug 1488837 Opened 6 years ago Updated 2 years ago

Test fails with headless, passes without


(Testing :: geckodriver, defect, P3)

59 Branch


(Not tracked)



(Reporter: jim.brannlund, Unassigned)



(1 file)

Attached file Reproducible testcase
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:61.0) Gecko/20100101 Firefox/61.0
Build ID: 20180807170231

Steps to reproduce:

Running the test (incl. file with on Ubuntu using Firefox in headless mode, will result in a failure. All things equal, without headless, test passes.

I've tested running it against:
- TestingBot (Ubuntu/FF59)
- selenium/standalone-firefox-debug:3.14.0-arsenic/curium docker (FF61.0.1)

- my local machine (macosx) 
- Bitbucket Pipelines

All with the same result, fail w/ headless - pass w/o. 

All instances have used Geckodriver 0.21.0.

Actual results:

Test fails on Linux using headless.

Expected results:

Test should pass. :)
Should mention that in all instances selenium python bindings was 3.14.0
In the TestingBot case Selenium Grid was 3.13.0
I can replicate this on an Ubuntu 14.04 machine. The traceback is:

python ~/Downloads/ 
Traceback (most recent call last):
  File "/home/dave/Downloads/", line 81, in <module>
  File "/home/dave/Downloads/", line 71, in run_it
  File "/home/dave/Downloads/", line 12, in get_element
    return WebDriverWait(driver, timeout).until(exp_cond((By.CSS_SELECTOR, locator)))
  File "/home/dave/.bug1488837/local/lib/python2.7/site-packages/selenium/webdriver/support/", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Looking at the script, this suggests that the click on the first dropdown hasn't worked? As reported, it passes when run without headless set, and it passes with/without headless for me on macOS.
Dave, did you create a minimized HTML testcase? Would be good to just have that listed here.
(In reply to Henrik Skupin (:whimboo) from comment #3)
> Dave, did you create a minimized HTML testcase? Would be good to just have
> that listed here.

No, I just used Jim's. It's only requires the selenium dependency.
Attachment #9006633 - Attachment mime type: text/x-python-script → text/plain
Dave, the problem you are seeing here is because in some cases the initial page load is not finishing. When running the script outside of headless you can see that the spinner is spinning forever. I can always replicate that with a developer build of Firefox but not with a Nightly build.

When I change the attached script to customize the Firefox binary the test works fine in both modes:

>    from selenium.webdriver.firefox.options import Options
>    opts = Options()
>    opts.binary = "/Applications/"
>    driver = webdriver.Firefox(options=opts)

Jim, please attach a trace level log to this bug so that I can see which step is failing for you. See in how to get one.
Flags: needinfo?(jim.brannlund)
Here's a link to the TestingBot job for

Here's a link to the TestingBot job for the real test:

As far as inifinite spinner goes, that's not what I'm seeing, in either test. Not according to the screenshots at least. For some reason, the screenshots isn't working on TB for headless. I will raise an issue there. They do work and are added to the html-report.
Ok, the hang for the spinner I can workaround by using a Nightly build of Firefox. But when I then run your script without headless, it hangs in the following line:

> get_element(first_dropdown).click()

Jim, do you have a chance to create a minimized HTML testcase for your issue? As long as I cannot even run the test outside of headless there is nothing I can do.

Do you have a testingbot video for a non-headless job? I would like to see what it's actually doing.
(In reply to jim.brannlund from comment #6)
> Here's the trace for
> Here's the trace for the ACTUAL test:

Both links are not accessible.
Priority: -- → P3
Summary: Test fails with headless, passes without. → Test fails with headless, passes without
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.