Intermittent test_software_update.py TestSoftwareUpdate.test_build_info | TypeError: argument of type 'NoneType' is not iterable

RESOLVED FIXED in Firefox 54

Status

Testing
Marionette
RESOLVED FIXED
a year ago
4 days ago

People

(Reporter: Treeherder Bug Filer, Assigned: whimboo)

Tracking

({ateam-marionette-firefox-puppeteer, intermittent-failure})

Version 3
mozilla55
ateam-marionette-firefox-puppeteer, intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox54 fixed, firefox55 fixed)

Details

(Whiteboard: [stockwell infra])

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

a year ago
treeherder
Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=33004885&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/XUEt3K5CSbOA1QaBmdGJgQ/runs/0/artifacts/public%2Flogs%2Flive_backing.log
(Assignee)

Comment 1

a year ago
 10:25:40     INFO - TEST-UNEXPECTED-ERROR | test_software_update.py TestSoftwareUpdate.test_build_info | TypeError: argument of type 'NoneType' is not iterable

 10:25:40     INFO - Traceback (most recent call last):

 10:25:40     INFO -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test.py", line 344, in run

 10:25:40     INFO -     testMethod()

 10:25:40     INFO -   File "/home/worker/workspace/build/tests/firefox-ui/tests/puppeteer/test_software_update.py", line 42, in test_build_info

 10:25:40     INFO -     self.assertIn('xml', build_info['update_snippet'])

 10:25:40     INFO -   File "/usr/lib/python2.7/unittest/case.py", line 798, in assertIn

10:25:40 INFO - if member not in container: 

Looks like we got an invalid snippet returned which was not an XML document. We better should tweak the test to use the local web server. In that case we would not depend on network.

Comment 2

9 months ago
12 failures in 812 pushes (0.015 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 4
* mozilla-beta: 3
* try: 2
* graphics: 2
* mozilla-aurora: 1

Platform breakdown:
* linux64: 9
* linux32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-02-20&endday=2017-02-26&tree=all
this has failed 38 times today :(

https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-04-10&endday=2017-04-13&tree=all

:whimboo, could you look into this or help find someone to look at this?
Flags: needinfo?(hskupin)
Whiteboard: [stockwell needswork]
(Assignee)

Comment 4

7 months ago
I had a look at the test and that's what happened:

[task 2017-04-13T03:53:07.990215Z] 03:53:07     INFO -  1492055587985	Marionette	TRACE	22 -> [0,52,"executeScript",{"scriptTimeout":null,"newSandbox":true,"args":["https://aus5.mozilla.org/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%SYSTEM_CAPABILITIES%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml"],"filename":"software_update.py","script":"\n          Components.utils.import(\"resource://gre/modules/UpdateUtils.jsm\")\n          return UpdateUtils.formatUpdateURL(arguments[0]);\n        ","sandbox":"default","line":383}]
[task 2017-04-13T03:53:08.005849Z] 03:53:08     INFO -  1492055588003	Marionette	TRACE	22 <- [1,52,null,{"value":"https://aus5.mozilla.org/update/6/Firefox/55.0a1/20170413031514/Linux_x86-gcc3/en-US/default/Linux%203.13.0-112-generic%20(GTK%203.18.9%2Clibpulse%208.0.0)/NA/default/default/update.xml"}]

First we retrieve the update URL from Firefox and then use a HTTP connection to retrieve the update snippet details. So as the above URL shows we access a remote site which is aus5.mozilla.org. I believe that there might have been an outage today, which caused this problem.

To solve this problem I suggest that we override the update URL with a local one as served via wptserve.

I'm out the next days and will be back on Tuesday. So I won't have time before, and hope that aus5.mozilla.org will deliver data fine again.
Flags: needinfo?(hskupin)
(Assignee)

Comment 5

7 months ago
Ben, can you see some anomalies of aus5.mozilla.org in delivering data today?
Flags: needinfo?(bhearsum)
(In reply to Henrik Skupin (:whimboo) from comment #5)
> Ben, can you see some anomalies of aus5.mozilla.org in delivering data today?

We had a brief outage that caused a bunch of 500s from roughly  8:48 to about 9:05 PM PDT on April 12th - does that line up with the recent failures?
Flags: needinfo?(bhearsum)

Comment 7

7 months ago
38 failures in 205 pushes (0.185 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 15
* autoland: 15
* mozilla-aurora: 8

Platform breakdown:
* linux64: 31
* linux32: 6
* android-6-0-armv8-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-04-13&endday=2017-04-13&tree=all

Comment 8

7 months ago
38 failures in 894 pushes (0.043 failures/push) were associated with this bug in the last 7 days. 

This is the #44 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 15
* autoland: 15
* mozilla-aurora: 8

Platform breakdown:
* linux64: 31
* linux32: 6
* android-6-0-armv8-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-04-10&endday=2017-04-16&tree=all
this looks to be related exclusively to the outage.
Whiteboard: [stockwell needswork] → [stockwell infra]
(Assignee)

Comment 10

7 months ago
Thanks Ben and Joel, for checking this. 

We should improve the assertion for the unit test, by simply forcing a local AUS URL. That would kill any failures due to a remote outage, and also is something we need for making this unit test a tier-1.

I will see what I can do this week.
(Assignee)

Updated

7 months ago
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
(Assignee)

Updated

7 months ago
Component: Firefox UI Tests → Marionette
Keywords: ateam-marionette-firefox-puppeteer
QA Contact: hskupin
Comment hidden (mozreview-request)

Comment 12

7 months ago
mozreview-review
Comment on attachment 8860952 [details]
Bug 1290850 - Force local update URL for software update unit tests.

https://reviewboard.mozilla.org/r/132954/#review135920

::: testing/firefox-ui/tests/puppeteer/test_software_update.py:50
(Diff revision 1)
> -        self.assertTrue(build_info['locale'])
> +            self.assertTrue(build_info['locale'])
> -        self.assertIn('force=1', build_info['update_url'])
> +            self.assertIn('force=1', build_info['update_url'])
> -        self.assertIn('xml', build_info['update_snippet'])
> +            self.assertIn('xml', build_info['update_snippet'])
> -        self.assertEqual(build_info['channel'], self.software_update.update_channel)
> +            self.assertEqual(build_info['channel'], self.software_update.update_channel)
> +        finally:
> +            self.restart(clean=True)

I'm not sure why a restart is being added here in the context of this bug. Could you describe this in the commit message for future archaeologists?
Attachment #8860952 - Flags: review?(mjzffr) → review+
(Assignee)

Comment 13

7 months ago
mozreview-review-reply
Comment on attachment 8860952 [details]
Bug 1290850 - Force local update URL for software update unit tests.

https://reviewboard.mozilla.org/r/132954/#review135920

> I'm not sure why a restart is being added here in the context of this bug. Could you describe this in the commit message for future archaeologists?

I don't think a comment in the commit message is useful here. If we need one it has to be in the code right before this line. I will do that before pushing.
(Assignee)

Comment 14

7 months ago
mozreview-review-reply
Comment on attachment 8860952 [details]
Bug 1290850 - Force local update URL for software update unit tests.

https://reviewboard.mozilla.org/r/132954/#review135920

> I don't think a comment in the commit message is useful here. If we need one it has to be in the code right before this line. I will do that before pushing.

So in short, we have to reset the custom update url which we set via the default preferences. But I see now that I should move the line which sets the URL into this test, because it's not needed somewhere else.
Comment hidden (mozreview-request)

Comment 16

7 months ago
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d467cc23f675
Force local update URL for software update unit tests. r=maja_zf

Comment 17

7 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d467cc23f675
Status: ASSIGNED → RESOLVED
Last Resolved: 7 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
(Assignee)

Comment 18

7 months ago
A test-only change which stabilizes the firefox puppeteer unit tests. Please uplift to mozilla-beta. Thanks.
status-firefox54: --- → affected
Whiteboard: [stockwell infra] → [stockwell infra][checkin-needed-beta]

Comment 19

7 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/b310658e4537
status-firefox54: affected → fixed
Whiteboard: [stockwell infra][checkin-needed-beta] → [stockwell infra]
(Assignee)

Updated

7 months ago
Depends on: 1361849

Comment 20

3 months ago
1 failures in 901 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-08-07&endday=2017-08-13&tree=all
1 failures in 762 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1

Platform breakdown:
* gecko-decision: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1290850&startday=2017-11-13&endday=2017-11-19&tree=all
You need to log in before you can comment on or make changes to this bug.