Closed Bug 1509009 Opened 6 years ago Closed 6 years ago

Perma tier 2 raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase

Categories

(Testing :: Raptor, defect, P5)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
defect

Tracking

(firefox65 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla65
Tracking Flags:
Tracking Status
firefox65 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: rwood)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(1 file, 1 obsolete file)

Bug 1509009 - Fix proxy policy for raptor http pageload; r?davehunt,bebe
47 bytes, text/x-phabricator-request
Details | Review 
Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=213114060&repo=mozilla-central

https://queue.taskcluster.net/v1/task/cIte6_pwRNyR-Gmyc8YMog/runs/0/artifacts/public/logs/live_backing.log

11:47:26     INFO -  raptor-mitmproxy Stopping mitmproxy playback, klling process 5116
11:47:36     INFO -  raptor-mitmproxy Successfully killed the mitmproxy playback process
11:47:36     INFO -  raptor-mitmproxy Turning off the browser proxy
11:47:36     INFO -  raptor-mitmproxy writing: C:\Users\task_1542798928\build\application\firefox\distribution\policies.json
11:47:36     INFO -  raptor-main removing webext C:\Users\task_1542798928\build\tests\raptor\webext\raptor
11:47:36     INFO -  results-handler summarizing raptor test results
11:47:36     INFO -  raptor-output ignoring the first dcf value due to initial pageload noise
11:47:36     INFO -  raptor-output ignoring the first fnbpaint value due to initial pageload noise
11:47:36     INFO -  raptor-output ignoring the first ttfi value due to initial pageload noise
11:47:36     INFO -  raptor-output ignoring the first dcf value due to initial pageload noise
11:47:36     INFO -  raptor-output ignoring the first fnbpaint value due to initial pageload noise
11:47:36     INFO -  raptor-output ignoring the first ttfi value due to initial pageload noise
11:47:36     INFO -  raptor-output PERFHERDER_DATA: {"framework": {"name": "raptor"}, "suites": [{"extraOptions": [], "name": "raptor-tp6-imdb-firefox", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 240.38, "subtests": [{"name": "raptor-tp6-imdb-firefox-dcf", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 244.5, "replicates": [447, 109, 105, 105, 267, 264, 233, 244, 246, 250, 247, 245, 248, 245, 246, 120, 290, 248, 103, 103, 105, 123, 270, 104, 121], "unit": "ms"}, {"name": "raptor-tp6-imdb-firefox-fnbpaint", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 131.0, "replicates": [438, 109, 105, 104, 108, 109, 226, 238, 239, 243, 240, 239, 242, 238, 240, 119, 139, 242, 102, 103, 105, 123, 263, 103, 120], "unit": "ms"}, {"name": "raptor-tp6-imdb-firefox-ttfi", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 433.0, "replicates": [868, 529, 519, 553, 442, 467, 411, 425, 424, 418, 428, 426, 425, 423, 428, 449, 469, 424, 433, 427, 433, 453, 456, 477, 498], "unit": "ms"}], "type": "pageload", "unit": "ms"}, {"extraOptions": [], "name": "raptor-tp6-imgur-firefox", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 325.63, "subtests": [{"name": "raptor-tp6-imgur-firefox-dcf", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 259.0, "replicates": [712, 339, 346, 357, 255, 262, 258, 259, 253, 251, 272, 254, 259, 299, 258, 266, 259, 250, 256, 262, 270, 248, 257, 253, 262], "unit": "ms"}, {"name": "raptor-tp6-imgur-firefox-fnbpaint", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 269.5, "replicates": [715, 348, 357, 373, 270, 272, 268, 269, 261, 265, 280, 263, 269, 104, 270, 277, 271, 268, 267, 281, 279, 267, 269, 265, 272], "unit": "ms"}, {"name": "raptor-tp6-imgur-firefox-ttfi", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 494.5, "replicates": [1029, 609, 606, 612, 494, 501, 494, 497, 482, 480, 505, 488, 493, 523, 488, 500, 487, 484, 499, 491, 514, 482, 495, 493, 496], "unit": "ms"}], "type": "pageload", "unit": "ms"}]}
11:47:36     INFO -  raptor-output results can also be found locally at: C:\Users\task_1542798928\build\raptor.json
11:47:36     INFO -  raptor-control-server shutting down control server
11:47:36     INFO -  raptor-main finished
11:47:36     INFO -  raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
11:47:37     INFO -  53] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:44:54] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:44:55] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:44:55] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:05] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:06] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:07] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:07] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:18] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:19] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:20] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:20] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:30] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:30] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:30] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:45:30] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:09] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:09] "GET /raptor-tp6-wikia-firefox.json HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:09] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:39] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:39] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:40] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:41] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:46:41] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:47:10] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:47:10] "POST / HTTP/1.1" 200 -
11:47:37     INFO -  127.0.0.1 - - [21/Nov/2018 11:47:10] "POST / HTTP/1.1" 200 -
WARNING | IO Completion Port failed to signal process shutdown
Parent process 7376 exited with children alive:
PIDS: 3564, 2348, 6640, 6548
Attempting to kill them, but no guarantee of success
11:50:42    ERROR - Return code: 1
Summary: Intermittent raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase → Perma tier 2 raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6-wikia-firefox' timed out loading test page: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
I believe this is caused by bug 1508692, Robert, can you take a look?

This only occurs on tier2.
Flags: needinfo?(rwood)
Whiteboard: [stockwell needswork]
(In reply to Andreea Pavel [:apavel] from comment #2)
> I believe this is caused by bug 1508692, Robert, can you take a look?
> 
> This only occurs on tier2.

Thanks Andrea. It's not Bug 1508692 - the test wouldn't have gotten that far if it was (the cert and proxy are being set and other pages are loading fine before this one). It's a page timeout:

INFO -  raptor-control-server received webext_raptor-page-timeout: [u'raptor-tp6-wikia-firefox', u'http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase']
23:45:12     INFO -  raptor-control-server received webext_status: __raptor_shutdownBrowser

Looks like we aren't giving enough time for this page to load on windows. I'll update the timeout for this test and push to try.
Flags: needinfo?(rwood)
Thanks Robert!
(In reply to Andreea Pavel [:apavel] from comment #2)
> I believe this is caused by bug 1508692, Robert, can you take a look?
> 
> This only occurs on tier2.

Ahhh - yes, my apologies Andreea - you are totally correct, it was bug 1508692 just as you suspected. Turns out the other pages in the same suite (tp6-3) are https - but this one is http - and that is why the others were working fine. There is definitely an issue with the http proxy on Windows. Might be a bug in policies not working the same on Windows or something, I'll look into it further... thanks for bringing this to my attention.
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Ok, figured out the problem. For some reason in the prefs in testing/profiles/perf/user.js [1] (which are applied when running Raptor) they include these:

user_pref("network.proxy.http", "localhost");
user_pref("network.proxy.http_port", 80);
user_pref("network.proxy.type", 1);

[1] https://searchfox.org/mozilla-central/rev/876022232b15425bb9efde189caf747823b39567/testing/profiles/perf/user.js#80

When I comment out those, http pageload (i.e. wikia) works. Looks like the policies.json is applied - which sets the proxy for http and https - but then the above prefs are overriding that, and incorrectly setting the http proxy to 'localhost' port 80 - which isn't correct, and causes the page not to be able to be loaded by Raptor.
Attachment #9026978 - Attachment is obsolete: true
Solution: Turns out in the policy you can lock the settings so that they can't be changed outside of that; this seems to fix it. Patch incoming.
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d997f8cd27fc
Fix proxy policy for raptor http pageload; r=davehunt
https://hg.mozilla.org/mozilla-central/rev/d997f8cd27fc
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox65: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Thanks for fixing this Robert!
It looks fine on the latest beta pushes; 2 failures in 2485 pushes in the last 7 days is ok. Thanks.
Flags: needinfo?(rwood)
Attachment #9026978 - Attachment is obsolete: false
Attachment #9026978 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: