1304781 - Intermittent test_safe_browsing_initial_download.py TestSafeBrowsingInitialDownload.test_safe_browsing_initial_download | TimeoutException: Timed out after 30.1 seconds with message: Safe Browsing File: ^goog-badbinurl-shavar.pset$ not found!

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3886039&repo=autoland

https://queue.taskcluster.net/v1/task/aV-VhMvxRaGsyMXJxXXrjw/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Downloaded safebrowsing files: ['test-unwanted-simple.pset', 'test-block-simple.pset', 'test-malware-simple.pset', 'test-trackwhite-simple.pset', 'test-track-simple.sbstore', 'test-phish-simple.pset', 'test-phish-simple.sbstore', 'test-trackwhite-simple.sbstore', 'test-malware-simple.sbstore', 'test-track-simple.pset', 'test-unwanted-simple.sbstore', 'test-block-simple.sbstore']

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Francois, fyi we are failing a lot in this test on Aurora those days mainly on OS X and Windows, but not Linux. It happens across locales.

Maybe someone should investigate?

Comment 3

[François Marier [:francois]]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Francois, fyi we are failing a lot in this test on Aurora those days mainly
> on OS X and Windows, but not Linux. It happens across locales.
> 
> Maybe someone should investigate?

Is it just slow? Depending on the kind of VM and network that these tests run on, it's possible we need to give the tests a minute or two to download the lists.

Comment 4

[François Marier [:francois]]

Matt, would you be able to confirm that Safe Browsing is working in Aurora on OSX and Windows, just in case?

I'm thinking:

1. creating a new profile
2. checking that the list files have been downloaded to the cache directory (~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/ and C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\) after a few minutes
3. checking the Webpage Warnings links on https://testsafebrowsing.appspot.com/ after a few hours

Comment 5

[Intermittent Failures Robot]

21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 21

Platform breakdown:
* windows7-64: 4
* osx-10-9: 4
* windows8-64: 3
* osx-10-11: 3
* windowsxp: 2
* windows8-32: 2
* windows7-32: 1
* osx-10-10: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304781&startday=2016-10-12&endday=2016-10-12&tree=all

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to François Marier [:francois] from comment #3)
> (In reply to Henrik Skupin (:whimboo) from comment #2)
> Is it just slow? Depending on the kind of VM and network that these tests
> run on, it's possible we need to give the tests a minute or two to download
> the lists.

It's hard to tell without the changes from bug 1304983. Should we maybe backport this addition to aurora?

Comment 7

[François Marier [:francois]]

(In reply to Henrik Skupin (:whimboo) from comment #6)
> It's hard to tell without the changes from bug 1304983. Should we maybe
> backport this addition to aurora?

Right, I had forgotten that the test on Aurora was different from the one in Nightly. I think it makes sense to uplift given that this test has caused problems in the past.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, I requested the uplift of the changes. It will also mean we won't hit a failure signature like this one anymore once landed. Once we see a similar bug I will duplicate it accordingly. Maybe its related to bug 1309764.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Francois, on mozilla-aurora the first failures clearly started between those two nightly builds:

https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=41593448b1ae&tochange=fb564edbd98d

Not sure what this "Automated HSTS preload list update" is, but maybe it is related? I will try to retrigger some tests from before those changes to see if those tests are still green. If not we might see a remote change by Google?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Most of the newly triggered tests are failing:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=6eefe948a169&filter-searchStr=os%20x%20firefox%20ui&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3856149

So I would expect a change on the remote site around this time.

Comment 11

[François Marier [:francois]]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> Not sure what this "Automated HSTS preload list update" is, but maybe it is
> related? I will try to retrigger some tests from before those changes to see
> if those tests are still green.

Interesting. The HSTS preload list is not delivered through Safe Browsing. I would guess it's delivered by Kinto.

Maybe it affects the timing of other downloads that happen at startup time? Would disabling network.stricttransportsecurity.preloadlist actually change anything?

> If not we might see a remote change by Google?

If that were the case then we'd see manual tests also fail. Hopefully Matt can confirm that this is not happening.

Comment 12

[François Marier [:francois]]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> https://hg.mozilla.org/releases/mozilla-aurora/
> pushloghtml?fromchange=41593448b1ae&tochange=fb564edbd98d
> 
> Not sure what this "Automated HSTS preload list update" is, but maybe it is
> related?

Oh, I was wrong. This isn't about letting kinto ship HPKP/HSTS lists to Firefox, but rather about doing a sync of the Google list.

Nothing in that pushlog seems related to the timeouts we're seeing.

Comment 13

[Intermittent Failures Robot]

27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 27

Platform breakdown:
* windows8-64: 6
* windows7-64: 6
* windowsxp: 3
* linux32: 3
* windows8-32: 2
* windows7-32: 2
* osx-10-11: 2
* linux64: 2
* osx-10-9: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304781&startday=2016-10-14&endday=2016-10-14&tree=all

Comment 14

[Intermittent Failures Robot]

63 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 63

Platform breakdown:
* windows7-64: 13
* windows8-64: 12
* osx-10-9: 7
* osx-10-11: 7
* windowsxp: 6
* linux32: 5
* windows8-32: 4
* windows7-32: 3
* osx-10-10: 3
* linux64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304781&startday=2016-10-10&endday=2016-10-16&tree=all

Comment 15

[Matt Wobensmith [:mwobensmith][:matt:]]

Using steps from comment 4, I see no issues with today's Aurora 51.0a2, on Windows 10 and Mac OSX. As always, the updates take several hours, but eventually the sites are blocked.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Fixed by bug 1304983 for Firefox 51.0.