1009695 - Gaia-Try is seeing 404s while downloading test zip files.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Example: https://tbpl.mozilla.org/?tree=Gaia-Try&rev=c34c827db3d9

Every test failed because none of them could download the tests.  All of the tests started at 4:49 - 5:11 in whatever timezone TBPL uses.  22 minutes seems a little long for the problem to be related to a race uploading the test files.  I don't see anything that explains this and the tests ran fine when Julien retriggered them.

Do we have any idea what's causing this problem?

Comment 1

[Aki Sasaki (not active)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=39558005&tree=Gaia-Try started at 4:02 and got

04:06:58  WARNING - Server returned status 404 HTTP Error 404: Not Found for https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tar.bz2

https://tbpl.mozilla.org/php/getParsedLog.php?id=39557434&tree=Gaia-Try started at 4:02 and got

04:06:04     INFO - Copy/paste: /builds/slave/test/build/venv/bin/mozinstall --app b2g /builds/slave/test/build/b2g-32.0a1.en-US.linux-x86_64.tar.bz2 --destination /builds/slave/test/build/application
04:06:34     INFO - Reading from file tmpfile_stdout
04:06:34     INFO - Output received:
04:06:34     INFO -  /builds/slave/test/build/application/b2g/b2g

https://tbpl.mozilla.org/php/getParsedLog.php?id=39558052&tree=Gaia-Try&full=1#error0
04:08:13  WARNING - Server returned status 404 HTTP Error 404: Not Found for https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tests.zip
04:08:13     INFO - retry: Failed, sleeping 60 seconds before retrying

From http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/1399977777/mozilla-central-linux64_gecko-bm82-build1-build0.txt.gz it looks like we upload en-US before multi, which looks correct.

Going to try debugging a bit more before giving up and moving us back to using nightlies.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Caching at the load balancer is a likely culprit:

$ curl -I https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tar.bz2
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: ftp5.dmz.scl3.mozilla.com
Cache-Control: max-age=3600

Comment 3

[Aki Sasaki (not active)]

Hah.

Set the url to

    https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tar.bz2?rand=%(rand)d

and replace %(rand)d with random.randint(0,10000) at fetch time?

?

Comment 4

[Aki Sasaki (not active)]

or even time.strftime("%Y%m%d%H%M%S"), which will result in not repeating.

Comment 5

[Aki Sasaki (not active)]

Attachment: gaia_try_kill_cache

John: can you apply this to your mozharness user repo and let me know if it works?  If so, I'll ask :jgriffin for review.

Comment 6

[Aki Sasaki (not active)]

Attachment: gaia_try_kill_cache

With a self.info() so we're not doing this blind.

Comment 7

[Aki Sasaki (not active)]

Any luck? I don't see it landed in your user repo.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

http://hg.mozilla.org/users/jford_mozilla.com/mozharness/rev/12c72cfcb94c

And merged to my production branch.

Comment 9

[Aki Sasaki (not active)]

Are we good here?
Should I get review and point gaia-try back to production mozharness?

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Yes.

Please feel free to switch back to production mozharness branch for gaia-try.

Comment 11

[Aki Sasaki (not active)]

Backed out 13c606457d3d (pointing gaia-try back at production mozharness), will take effect next reconfig.
https://hg.mozilla.org/build/buildbot-configs/rev/3cafb81abf97

Comment 12

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8422149 [details] [diff] [review]
gaia_try_kill_cache

Review of attachment 8422149 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8422149 [details] [diff] [review]
gaia_try_kill_cache

https://hg.mozilla.org/build/mozharness/rev/f2ce40efae6c

Comment 14

[Justin Wood (:Callek)]

Something here went live today