Run final-verification.sh against release channel to test AUS throttling

RESOLVED WONTFIX

Status

defect
P4
normal
RESOLVED WONTFIX
8 years ago
2 years ago

People

(Reporter: rail, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releases][automation])

Attachments

(2 attachments, 3 obsolete attachments)

We should run update verify tests against beta channel after pushing snippets to the beta channel to test AUS throttling and to prevent bugs like bug 651923.
Posted patch verify.sh changes (obsolete) — Splinter Review
Example output:

bash verify.sh -t -f beta moz192-firefox-linux.cfg --old-updater                
Using config file moz192-firefox-linux.cfg
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.16/20110319135224/Linux_x86-gcc3/af/beta/update.xml?force=1
Testing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.17-candidates/build2/update/linux-i686/af/firefox-3.6.16-3.6.17.partial.mar
HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
X-Backend-Server: dm-ftp01
Content-Type: application/octet-stream
Date: Tue, 26 Apr 2011 09:59:22 GMT
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
ETag: "9c9b0d-cd271-3d776640"
Connection: Keep-Alive
Last-Modified: Thu, 14 Apr 2011 19:50:41 GMT
X-Cache-Info: caching
Content-Length: 840305
Attachment #528286 - Flags: feedback?(nrthomas)
Posted patch buildbotcustom (obsolete) — Splinter Review
Not tested yet.

It supposed to be run manually after pushing snippets to the beta channel and unthrottling AUS.
Attachment #528287 - Flags: feedback?(nrthomas)
Posted patch verify.sh changes (obsolete) — Splinter Review
* force=1 shouldn't be added to the AUS requests for such tests
Attachment #528286 - Attachment is obsolete: true
Attachment #528286 - Flags: feedback?(nrthomas)
Attachment #528298 - Flags: review?(nrthomas)
Err, wrong file.
Attachment #528298 - Attachment is obsolete: true
Attachment #528298 - Flags: review?(nrthomas)
Attachment #528299 - Flags: review?(nrthomas)
* pass "-n" to prevent "force=1" style URLs
Attachment #528287 - Attachment is obsolete: true
Attachment #528287 - Flags: feedback?(nrthomas)
Attachment #528301 - Flags: review?(nrthomas)
I tested the patches in staging.

No changes in update verify requests:
================================================
bash verify.sh -c moz20-firefox-mac64.cfg
......
Using config file moz20-firefox-mac64.cfg
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1
05:31:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0rc2-4.0.1.partial.mar [2385810/2385810] -> "update/partial.mar" [1]
05:31:53 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1]
05:31:56 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1]
installing downloads/Firefox 4.0 RC 2.dmg
...........
FINISH ADD force_plist_reload
backup_discard: backup file doesn't exist: force_plist_reload.moz-backup
succeeded
calling QuitProgressUI
Only in source/Firefox.app: force_plist_reload
WARN: non-binary files found in diff
WARN: check_updates returned warning for Darwin_x86_64-gcc3 downloads/Firefox 4.0 RC 2.dmg vs. downloads/Firefox 4.0.1.dmg: 2
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Darwin_x86_64-gcc3/af/betatest/update.xml?force=1
05:32:40 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/mac/af/firefox-4.0.1.complete.mar [27029123/27029123] -> "update/complete.mar" [1]
05:32:46 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/releases/4.0rc2/mac/af/Firefox%204.0%20RC%202.dmg [27893468/27893468] -> "Firefox 4.0 RC 2.dmg" [1]
05:32:49 URL:http://staging-stage.build.mozilla.org/pub/mozilla.org//firefox/nightly/4.0.1-candidates/build1/mac/af/Firefox%204.0.1.dmg [27904105/27904105] -> "Firefox 4.0.1.dmg" [1]
installing downloads/Firefox 4.0 RC 2.dmg
..................
================================================


Beta update verify output looks like this:

Throttled:
================================================
bash verify.sh -t -f beta -n moz20-firefox-linux.cfg
..........
Using config file moz20-firefox-linux.cfg
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml
FAIL: no partial update found for http://staging-stage.build.mozilla.org/update/1/Firefox/4.0/20110318052756/Linux_x86-gcc3/af/beta/update.xml
FAIL: download_mars returned non-zero exit code: 1
================================================

Unthrottled:
================================================
Using  http://staging-stage.build.mozilla.org/update/1/Firefox/3.7a1/20100208064801/Linux_x86-gcc3/en-US/beta/update.xml
Testing http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/4.0.1-candidates/build1/update/linux-i686/en-US/firefox-4.0.1.complete.mar
HTTP/1.1 200 OK
Date: Tue, 26 Apr 2011 12:25:38 GMT
Server: Apache/2.0.52 (CentOS)
Last-Modified: Thu, 14 Apr 2011 06:20:50 GMT
ETag: "5a0003-d41997-ed379480"
Accept-Ranges: bytes
Content-Length: 13900183
Connection: close
Content-Type: text/plain; charset=UTF-8
================================================

It normally takes 3 minutes to verify beta snippets on darwin10 machines.
Attachment #528299 - Flags: review?(nrthomas) → review+
Comment on attachment 528301 [details] [diff] [review]
buildbotcustom

This is a good start, but I think we can do even better. For old branches (3.5/3.6) we need to check that the throttling is disabled for the release channel too. Not clear what we'll need yet for 4.0 going upwards, so we should make the creation of these builders configurable.

(In reply to comment #0)
> We should run update verify tests against beta channel after pushing snippets
> to the beta channel to test AUS throttling and to prevent bugs like bug 651923.

Slight correction - ... after pushing snippets to the beta channel and deploying the AUS config change ...
Attachment #528301 - Flags: review?(nrthomas) → review-
Comment on attachment 528299 [details] [diff] [review]
verify.sh changes

Ooops, pushed both patches while the second one was r-. Backed out both.
Attachment #528299 - Flags: checked-in+ → checked-in-
Priority: P2 → P4
Not working on this bug actively. Back to the pool.
Assignee: rail → nobody
No longer blocks: hg-automation
Mass move of bugs to Release Automation component.
Component: Release Engineering → Release Engineering: Automation (Release Automation)
Flags: checked-in-
No longer blocks: hg-automation
Product: mozilla.org → Release Engineering
Updating summary, as we don't throttle betas these days but do on release. Also, I think it makes sense to run this as part of final_verification since that's now parallelized. 

QA is currently covering this testing using a script that makes repeated requests on a few urls.

A few months ago we added 1 min caching on the Zeus load balancer, so some sort of random query parameter would be necessary if throttling is not 0 or 100%.
Summary: Run update verify against beta channel to test AUS throttling → Run finale_verification against release channel to test AUS throttling
Summary: Run finale_verification against release channel to test AUS throttling → Run final-verification.sh against release channel to test AUS throttling
QA Contact: rail
Guessing this is still needed?
(In reply to Aki Sasaki [:aki] from comment #13)
> Guessing this is still needed?

I haven't heard a peep about wanting this since Balrog came along. I think we have much higher confidence in it doing the right thing (vs needing to deploy changes to PHP files in AUS2). I can imagine that we might want something that sanity checks update rate on a continuous basis, but I highly doubt we're ever doing something like this as part of the release process.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.