Closed Bug 1516015 Opened 10 months ago Closed 9 months ago

Permafailing tier2 raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox/chrome' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase

Categories

(Testing :: Raptor, defect, P5)

Version 3
defect

Tracking

(firefox67 fixed)

RESOLVED FIXED
mozilla67
Tracking Status
firefox67 --- fixed

People

(Reporter: ccoroiu, Assigned: davehunt)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disable-recommended])

Attachments

(1 file, 1 obsolete file)

Summary: Perma tier 2 raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase → Intermitent raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
Thanks for looking into this, overall I think this is the right set of changes.

almost all of those landed directly on mozilla-central, I suspect we would need to bisect on try ( https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=30d8fe076fd0 ).  If we see a higher frequency of failures, then I would like to bisect- we need to run 100 cycles of the test, so it would be a drain on resources and time- that is why I want to wait to see if this happens more.
Summary: Intermitent raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase → Permafailing tier2 raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox/chrome' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
Duplicate of this bug: 1518978

In the last 7 days there have been 47 occurrences, most on MacOSX Nightly, Windows 7 32 Nightly, Windows 10 64 Nightly build type Opt.

This appears to be failing most frequently against Chromium. I only see three failures in https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516015&startday=2019-01-07&endday=2019-01-13&tree=all for Firefox, and 65 for Chromium.

:bebe: Could you see if you can replicate this timeout locally? Do we need to increase the timeout here? Would it help to re-record the requests/responses?

Flags: needinfo?(fstrugariu)

when running the test locally in Chrome we get redirected to a different URL:

https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase

tested this with a local payback proxy and chrome and we get the correct page
tested this with the mach started playback proxy and a separate chrome instance and we get the correct page.
tested this with a local playback proxy and the mach test(not starting the mach proxy) and we get the wrong page.

from the looks of it when starting chrome with the web extension we get redirected to the wrong website

Also here is the mitmproxy log:

127.0.0.1:55026: clientconnect
server_playback: killed non-replay request https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
127.0.0.1:55026: GET https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase HTTP/2.0
              << 404 Not Found 0b
127.0.0.1:55060: clientconnect

From this we see that the browser is requesting the wrong url and the proxy just does it's job and return 404

:rwood can you test this on your side please?

Flags: needinfo?(fstrugariu) → needinfo?(rwood)

Tested on firefox and all works fine

I'm able to replicate this when running mach raptor-test -t raptor-tp6-wikia-chrome --app chrome --binary /path/to/chrome. The test times out and Chrome shows as trying to open URL: https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase instead of the requested URL: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase.

It looks like all wikia.com domains will be changing to fandom.com domains, however as we have recorded the responses this shouldn't affect our tests.

If I start mitmdump and Chrome exactly as the command lines mentioned in the logs from running mach raptor-test I am unable to replicate the issue. It would seem that for some reason when these are launched via mach raptor-test, the request is at some point redirected to the new domain.

We could simple create a new recording for the new domain, however I'm concerned that our tests are running without being independant from changes to the live sites. Bebe: could you investigate further into why Chrome is able to determine the domain has been redirected when we're using a recording?

Flags: needinfo?(rwood)

When investigating the timeouts for wikia.com on Google Chrome I discovered differences in the way proxy configuration is interpreted. It meant that insecure sites such as wikia were not using the proxy when launched via mach, and now that wikia.com redirects to a secure site we get redirected to a site that is not in the recordings. I have resolved this by fixing the command line argument for the proxy to include all protocols and bypass the Raptor control server.

When investigating the timeouts for wikia.com on Google Chrome I discovered differences in the way proxy configuration is interpreted. It meant that insecure sites such as wikia were not using the proxy when launched via mach, and now that wikia.com redirects to a secure site we get redirected to a site that is not in the recordings. I have resolved this by fixing the command line argument for the proxy to include all protocols. It was also necesary to allow communication with the control server by adding localhost to the pass-through hosts.

I also removed the trailing disable-sync command line argument that was introduced recently in error.

Attachment #9039642 - Attachment is obsolete: true
Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED
Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5e594ff11d9e
Fix proxy configuration for Raptor in Google Chrome; r=rwood
Status: ASSIGNED → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.