Open Bug 1488837 Opened 4 years ago Updated 2 months ago

Test fails with headless, passes without

Categories

(Testing :: geckodriver, defect, P3)

59 Branch
defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: jim.brannlund, Unassigned)

Details

Attachments

(1 file)

Attached file Reproducible testcase
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:61.0) Gecko/20100101 Firefox/61.0
Build ID: 20180807170231

Steps to reproduce:

Running the test (incl. file with bug.py) on Ubuntu using Firefox in headless mode, will result in a failure. All things equal, without headless, test passes.

I've tested running it against:
- TestingBot (Ubuntu/FF59)
- selenium/standalone-firefox-debug:3.14.0-arsenic/curium docker (FF61.0.1)

using:
- my local machine (macosx) 
- Bitbucket Pipelines

All with the same result, fail w/ headless - pass w/o. 

All instances have used Geckodriver 0.21.0.


Actual results:

Test fails on Linux using headless.


Expected results:

Test should pass. :)
Should mention that in all instances selenium python bindings was 3.14.0
In the TestingBot case Selenium Grid was 3.13.0
I can replicate this on an Ubuntu 14.04 machine. The traceback is:

python ~/Downloads/bug.py 
Traceback (most recent call last):
  File "/home/dave/Downloads/bug.py", line 81, in <module>
    exit(run_it())
  File "/home/dave/Downloads/bug.py", line 71, in run_it
    get_element(value).click()
  File "/home/dave/Downloads/bug.py", line 12, in get_element
    return WebDriverWait(driver, timeout).until(exp_cond((By.CSS_SELECTOR, locator)))
  File "/home/dave/.bug1488837/local/lib/python2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Looking at the script, this suggests that the click on the first dropdown hasn't worked? As reported, it passes when run without headless set, and it passes with/without headless for me on macOS.
Dave, did you create a minimized HTML testcase? Would be good to just have that listed here.
(In reply to Henrik Skupin (:whimboo) from comment #3)
> Dave, did you create a minimized HTML testcase? Would be good to just have
> that listed here.

No, I just used Jim's. It's only requires the selenium dependency.
Attachment #9006633 - Attachment mime type: text/x-python-script → text/plain
Dave, the problem you are seeing here is because in some cases the initial page load is not finishing. When running the script outside of headless you can see that the spinner is spinning forever. I can always replicate that with a developer build of Firefox but not with a Nightly build.

When I change the attached script to customize the Firefox binary the test works fine in both modes:

>    from selenium.webdriver.firefox.options import Options
>
>    opts = Options()
>    opts.binary = "/Applications/FirefoxNightly.app/Contents/MacOS/firefox"
>
>    driver = webdriver.Firefox(options=opts)

Jim, please attach a trace level log to this bug so that I can see which step is failing for you. See https://firefox-source-docs.mozilla.org/testing/geckodriver/geckodriver/TraceLogs.html in how to get one.
Flags: needinfo?(jim.brannlund)
Here's a link to the TestingBot job for bug.py: https://testingbot.com/tests/6fc237f4-b453-49b3-a3b0-1dbb6c2e5710

Here's a link to the TestingBot job for the real test: https://testingbot.com/tests/c29a8eff-05af-411e-8606-c6c66a2878e7

As far as inifinite spinner goes, that's not what I'm seeing, in either test. Not according to the screenshots at least. For some reason, the screenshots isn't working on TB for headless. I will raise an issue there. They do work and are added to the html-report.
Ok, the hang for the spinner I can workaround by using a Nightly build of Firefox. But when I then run your script without headless, it hangs in the following line:

> get_element(first_dropdown).click()

Jim, do you have a chance to create a minimized HTML testcase for your issue? As long as I cannot even run the test outside of headless there is nothing I can do.

Do you have a testingbot video for a non-headless job? I would like to see what it's actually doing.
(In reply to jim.brannlund from comment #6)
> Here's the trace for bug.py:
[..]
> Here's the trace for the ACTUAL test:
[..]

Both links are not accessible.
Priority: -- → P3
Summary: Test fails with headless, passes without. → Test fails with headless, passes without
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.