652765 - Run final-verification.sh against release channel to test AUS throttling

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P4)

Description

[Rail Aliiev [:rail]]

We should run update verify tests against beta channel after pushing snippets to the beta channel to test AUS throttling and to prevent bugs like bug 651923.

Comment 1

[Rail Aliiev [:rail]]

Attachment: verify.sh changes

Example output:

bash verify.sh -t -f beta moz192-firefox-linux.cfg --old-updater                
Using config file moz192-firefox-linux.cfg
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.16/20110319135224/Linux_x86-gcc3/af/beta/update.xml?force=1
Testing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.17-candidates/build2/update/linux-i686/af/firefox-3.6.16-3.6.17.partial.mar
HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
X-Backend-Server: dm-ftp01
Content-Type: application/octet-stream
Date: Tue, 26 Apr 2011 09:59:22 GMT
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
ETag: "9c9b0d-cd271-3d776640"
Connection: Keep-Alive
Last-Modified: Thu, 14 Apr 2011 19:50:41 GMT
X-Cache-Info: caching
Content-Length: 840305

Comment 2

[Rail Aliiev [:rail]]

Attachment: buildbotcustom

Not tested yet.

It supposed to be run manually after pushing snippets to the beta channel and unthrottling AUS.

Comment 3

[Rail Aliiev [:rail]]

Attachment: verify.sh changes

* force=1 shouldn't be added to the AUS requests for such tests

Comment 4

[Rail Aliiev [:rail]]

Attachment: verify.sh changes

Err, wrong file.

Comment 5

[Rail Aliiev [:rail]]

Attachment: buildbotcustom

* pass "-n" to prevent "force=1" style URLs

Comment 6

[Rail Aliiev [:rail]]

I tested the patches in staging.

No changes in update verify requests:
================================================
bash verify.sh -c moz20-firefox-mac64.cfg
......
Using config file moz20-firefox-mac64.cfg
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1
05:31:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0rc2-4.0.1.partial.mar [2385810/2385810] -> "update/partial.mar" [1]
05:31:53 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1]
05:31:56 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1]
installing downloads/Firefox 4.0 RC 2.dmg
...........
FINISH ADD force_plist_reload
backup_discard: backup file doesn't exist: force_plist_reload.moz-backup
succeeded
calling QuitProgressUI
Only in source/Firefox.app: force_plist_reload
WARN: non-binary files found in diff
WARN: check_updates returned warning for Darwin_x86_64-gcc3 downloads/Firefox 4.0 RC 2.dmg vs. downloads/Firefox 4.0.1.dmg: 2
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1
05:32:40 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0.1.complete.mar [27029123/27029123] -> "update/complete.mar" [1]
05:32:46 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1]
05:32:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1]
installing downloads/Firefox 4.0 RC 2.dmg
..................
================================================


Beta update verify output looks like this:

Throttled:
================================================
bash verify.sh -t -f beta -n moz20-firefox-linux.cfg
..........
Using config file moz20-firefox-linux.cfg
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml
FAIL: no partial update found for http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml
FAIL: download_mars returned non-zero exit code: 1
================================================

Unthrottled:
================================================
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/3.7a1/20100208064801/Linux_x86-gcc3/en-US/beta/update.xml
Testing http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/linux-i686/en-US/firefox-4.0.1.complete.mar
HTTP/1.1 200 OK
Date: Tue, 26 Apr 2011 12:25:38 GMT
Server: Apache/2.0.52 (CentOS)
Last-Modified: Thu, 14 Apr 2011 06:20:50 GMT
ETag: "5a0003-d41997-ed379480"
Accept-Ranges: bytes
Content-Length: 13900183
Connection: close
Content-Type: text/plain; charset=UTF-8
================================================

It normally takes 3 minutes to verify beta snippets on darwin10 machines.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 528301 [details] [diff] [review]
buildbotcustom

This is a good start, but I think we can do even better. For old branches (3.5/3.6) we need to check that the throttling is disabled for the release channel too. Not clear what we'll need yet for 4.0 going upwards, so we should make the creation of these builders configurable.

(In reply to comment #0)
> We should run update verify tests against beta channel after pushing snippets
> to the beta channel to test AUS throttling and to prevent bugs like bug 651923.

Slight correction - ... after pushing snippets to the beta channel and deploying the AUS config change ...

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 528299 [details] [diff] [review]
verify.sh changes

http://hg.mozilla.org/build/tools/rev/fc636a1a8680

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 528299 [details] [diff] [review]
verify.sh changes

Ooops, pushed both patches while the second one was r-. Backed out both.

Comment 10

[Rail Aliiev [:rail]]

Not working on this bug actively. Back to the pool.

Comment 11

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Updating summary, as we don't throttle betas these days but do on release. Also, I think it makes sense to run this as part of final_verification since that's now parallelized. 

QA is currently covering this testing using a script that makes repeated requests on a few urls.

A few months ago we added 1 min caching on the Zeus load balancer, so some sort of random query parameter would be necessary if throttling is not 0 or 100%.

Comment 13

[Aki Sasaki (not active)]

Guessing this is still needed?

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #13)
> Guessing this is still needed?

I haven't heard a peep about wanting this since Balrog came along. I think we have much higher confidence in it doing the right thing (vs needing to deploy changes to PHP files in AUS2). I can imagine that we might want something that sanity checks update rate on a continuous basis, but I highly doubt we're ever doing something like this as part of the release process.