Closed Bug 621075 Opened 14 years ago Closed 13 years ago

Test AUS deployment at PHX

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cshields, Assigned: fox2mike)

References

Details

(Whiteboard: [before ffx4])

Attachments

(8 files, 1 obsolete file)

Releases to test
23.15 KB, application/zip
Details
Nightlies to test
11.54 KB, application/zip
Details
Update verification script
4.44 KB, text/plain
Details
Throttling test
4.58 KB, application/zip
Details
Fallback test
4.83 KB, application/zip
Details
SSL Certificates test
14.36 KB, application/zip
Details
PPC test
2.24 KB, application/zip
Details
AUS2 tests (combined)
37.21 KB, application/zip
Details 
IT will work with webdev to get the appropriate test suite and test the new AUS setup in PHX1. This is a continuation of bug 618088 and needs to happen before b9.

rhelmer, do you have the test bits?
(In reply to comment #0)
> IT will work with webdev to get the appropriate test suite and test the new AUS
> setup in PHX1. This is a continuation of bug 618088 and needs to happen before
> b9.
> 
> rhelmer, do you have the test bits?

We have fitnesse tests that are run on checkin by the Hudson job:
https://hudson.mozilla.org/job/AUSv2/

This is a functional test which validates the install against synthetic test data (which means we need to load the test data into the AUS install).

It would probably be a good idea to run a test against real data too (f.e. make sure that both servers are serving the same result for all valid update paths). We used a tool to do this a few years ago when AUS was moved from OSL, I'm having trouble digging up that bug right now but might be useful (should be pretty easy to write if not). I'll link the bug when I find it.
So the snippets are already being pushed to 10.8.84.240:/vol/pio_aus2 right?
(In reply to comment #3)
> IT will work with webdev to get the appropriate test suite and test the new AUS
> setup in PHX1. This is a continuation of bug 618088 and needs to happen before
> b9.


fyi: beta9 date moved forward, now scheduled for "go to build" Monday 10th, with release approx Wed/Thu. If this testing cannot safely be done in time, we should defer to beta10 a week later.
(In reply to comment #1)
> It would probably be a good idea to run a test against real data too (f.e. make
> sure that both servers are serving the same result for all valid update paths).
> We used a tool to do this a few years ago when AUS was moved from OSL, I'm
> having trouble digging up that bug right now but might be useful (should be
> pretty easy to write if not). I'll link the bug when I find it.

I found the migration bug 378579.

Is the /mofo repo still around? The tests we used last time are probably in there.
Attached file old aus verification script? (obsolete) — 
Dug this out of the old mofo repo (thanks to aravind).  Can we reuse this script or use this as a starting point at least?
(In reply to comment #5)
> Created attachment 502052 [details]
> old aus verification script?
> 
> Dug this out of the old mofo repo (thanks to aravind).  Can we reuse this
> script or use this as a starting point at least?

Yep this is the one I was thinking of; thanks!
(In reply to comment #6)
> (In reply to comment #5)
> > Created attachment 502052 [details]
> > old aus verification script?
> > 
> > Dug this out of the old mofo repo (thanks to aravind).  Can we reuse this
> > script or use this as a starting point at least?
> 
> Yep this is the one I was thinking of; thanks!

(from a quick look I believe it's still applicable, getting some releng eyes on it would be nice. We would need to generate the input file this wants)
John, can your guys look at this?
Whiteboard: [before ffx4]
(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > Created attachment 502052 [details]
> > > old aus verification script?
> > > 
> > > Dug this out of the old mofo repo (thanks to aravind).  Can we reuse this
> > > script or use this as a starting point at least?
> > 
> > Yep this is the one I was thinking of; thanks!
> 
> (from a quick look I believe it's still applicable, getting some releng eyes on
> it would be nice. We would need to generate the input file this wants)

(In reply to comment #8)
> John, can your guys look at this?

/me looks to catlee & nthomas. Any ideas if this is the best/right script to use for testing AUS-in-PHX?
Looks plausible to me. Just watch out for the comment at the top of the file neglecting to mention the data format has a 'MajorVersion' at the end. 

How big a test do you want to do, given we have a nightly job ensuring data integrity between both locations ? Based on the aus_data.csv in /mofo we did a comprehensive check last time but I'm not convinced that's required here.
rhelmer,

Do you have the time and background to set this test up on the seamicro nodes you used for socorro?
(In reply to comment #11)
> rhelmer,
> 
> Do you have the time and background to set this test up on the seamicro nodes
> you used for socorro?

Sure I can help with this, is there a desire to load-test too or just a functional/correctness test? Doing a functional test that compares the two environments probably only needs one node (basically just gets a bunch of URLs, and compares the response from SJC to the response from PHX).

(In reply to comment #10)
> Looks plausible to me. Just watch out for the comment at the top of the file
> neglecting to mention the data format has a 'MajorVersion' at the end. 
> 
> How big a test do you want to do, given we have a nightly job ensuring data
> integrity between both locations ? Based on the aus_data.csv in /mofo we did a
> comprehensive check last time but I'm not convinced that's required here.

I'm not sure it's really any harder to build a list of all existing update paths, versus just a subset. I don't have a strong opinion on this, as long as the subset is representative of the most important update types (major update, partner update, nightly update, ???). Making those decisions seems harder to me than just taking everything, but I don't know :)

Has anyone from QA been looped in on this yet? Might be nice to have keep this or a test like it maintained, run when AUS is updated, etc. instead of one-offing it every once in a while like this. They might have some advice on more modern tools and techniques at least.
(In reply to comment #12)
> (In reply to comment #11)
> > rhelmer,
> > 
> > Do you have the time and background to set this test up on the seamicro nodes
> > you used for socorro?
> 
> Sure I can help with this, is there a desire to load-test too or just a
> functional/correctness test? Doing a functional test that compares the two
> environments probably only needs one node (basically just gets a bunch of URLs,
> and compares the response from SJC to the response from PHX).

I would have assumed load testing was superfluous on newer, better hardware before Socorro, but it would be good to rule out any network/firewall issues.
I think it's just a correctness test - the AUS cluster in Phoenix is a dup of San Jose (but on newer hardware).
Summary of conf call w/rhelmer, laura, cshields, nthomas, catlee, joduinn:


0) nthomas will setup "server comparison" test. This same test was used last time a big AUS change was done. Also, cshields will look through logs in MPT in case of any weirdness, and we'll use some of those logs to enhance tests further. nthomas will give rhelmer csv file of tests to run.
1) rhelmer has tests that already run on checkin-to-AUS. We need to make sure that same version of AUS is being run in MPT and PHX.
2) RelEng will run "final verification" test (same as used in release automation)
3) RelEng will setup updates on betatest, releasetest channels based in PHX, so that QA can do update verification on those test channels.


I was taking notes, while in this meeting, so if there's anything I missed, please add it.
OS: Mac OS X → All
Attached file Releases to test — 
This contains
* Firefox 3.0.19 -> 3.6.13 major update
* Firefox 3.5.16 (current release for 3.5.x, has betatest/releasetest/release going to 3.5.17)
* Firefox 3.6.12 (one-off current 3.6.x)
* Firefox 3.6.13 (current 3.6.x, has betatest/releasetest/release going to 3.6.14)
* Firefox 4.0b9 (pointing at b10/b11 depending on channel)
* Thunderbird 2.0.0.24 major update

A total of 6744 individual update queries.
Attached file Nightlies to test — 
This contains nightly updates for
* Firefox 4 from mozilla-central
* Firefox 4 from tracemonkey
* Fennec for Android
The changes here are
* update hostname for MPT, still needs one for PHX (replace SOME_HOST with this)
* adds keep-alive and x-backend-server to headers allowed to differ, so that we  
 * don't get hung up on the max value changing in response headers like
     Keep-Alive: timeout=5, max=191
 * don't care which machine we get in a cluster (*)
* send something close to the update/2/ and update/3/ schemes by padding out the URL. Setting OS_VERSION to AUS_Checker gives us a way to exclude counts in metrics if they become a problem

* Ideally we'd make sure that all the nodes at in Phoenix really are the same in a controlled way, rather than just hitting the lot of them and having to debug rando-failures for any that need help.
Attachment #502052 - Attachment is obsolete: true
In addition to running the verification script against the lists in the release and nightly zip files, we could also
* verify that requests for an update for Fx4.0b4 running on PowerPC get no updates. AUS is looking for 'PPC' in the useragent
 http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88
* verify that fallback works for partner builds, eg take the fx3.6.13 list and append '-cck-euballot' to the channel to get release-cck-euballot
* throttling - hack the script to append '?force=1' to the query and use the firefox 3.5.16 snippets on the release channel (this'll work until 3.5.17 ships on Feb 8th). Should get nothing without the force, a major update with it
* make sure https://aus2.mozilla.org/bug602275/please_reinstall.xml is served the same (from bug 602275)

Right, that's plenty.

Rando-results (failures you can't reproduce manually)
* might see some of these for nightlies, due to new builds being published at MPT and rsync'd over to PHX every 10 minutes. This would be a 'Content check' or 'Content length' failure.
* shouldn't get any for release, unless you're testing when they're shipping Firefox 4.0b11/3.5.17/3.6.14.
Forgot something important. aus2.m.o and aus3.m.o use different SSL certificates - we need to make sure the PHX boxes have both. There's also a backup cert for aus3, but I'm not sure what the setup is for that.

In case 'Right, that's plenty' wasn't clear, this is over to IT and rhelmer now.
(In reply to comment #20)
> Forgot something important. aus2.m.o and aus3.m.o use different SSL
> certificates - we need to make sure the PHX boxes have both. There's also a
> backup cert for aus3, but I'm not sure what the setup is for that.
> 
> In case 'Right, that's plenty' wasn't clear, this is over to IT and rhelmer
> now.

Just a reminder that I am out next week, and I have quite a bit to finish up today. If you need something done that can't wait until I get back, please find and tell me directly today :)
cshields: what next? 

(fyi, for scheduling between betas, know that beta11 is likely to ship today/tomorrow, and we're likely to get beta12 sometime next week if I read tea leaves correctly)
(In reply to comment #22)
> cshields: what next? 

our SUMO move has been a priority and everything got preempted by the snippets deployment.  This is up in line along with those.  Will try to get this testing going in the next 24 hours.
fyi, expect "go to build" beta12 friday, or monday.
I'll do some testing and get back to you folks
Assignee: cshields → shyam
(In reply to comment #24)
> fyi, expect "go to build" beta12 friday, or monday.

(In reply to comment #25)
> I'll do some testing and get back to you folks

ping?
So, here's an update. AUS in phx wasn't setup to use the aus3.mozilla.org SSL cert, which I fixed (created a new traffic group and virtual server hitting the same pool of backends).

Ran the tests - releases looks fine :

[shyam@boris aus-phx-test]$ for i in `ls releases/`; do echo -e Testing ${i}; ./aus-verify.pl releases/${i} | grep -v PASS; done;

Testing fx3.0.19-mu
Testing fx3.5.16
Testing fx3.6.12
Testing fx3.6.13
Testing fx4.0b9
Testing tb2.0.0.24-mu

Basically, everything passed. 

Nightlies weren't so nice :

FAIL Header check failed for content-length NO MATCH, main content-length: 42 test content-length: 518
FAIL Content length check failed for Firefox|4.0b10pre|Linux_x86-gcc3|20110130030342|mai|nightly|3 (main=42 test=518

FAIL Content check failed for Firefox|4.0b10pre|Linux_x86-gcc3|20110131064712|ml|nightly|3 (main:<?xml version="1.0"?>
<updates>
    <update type="minor" version="4.0b12pre" extensionVersion="4.0b12pre" buildID="20110214030347">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/02/2011-02-14-03-mozilla-central-l10n/firefox-4.0b12pre.ml.linux-i686.complete.mar" hashFunction="sha512" hashValue="43f906066f2b0ed4d5af38f0734933d93c1d5baba84830822568852dbc36b3d2c7737e6107a7fcd2f264bb839a2649518e00ced5055562d6b25c35673d82146a" size="13371117"/>
    </update>
</updates> test:<?xml version="1.0"?>
<updates>
    <update type="minor" version="4.0b12pre" extensionVersion="4.0b12pre" buildID="20110214030347">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/02/2011-02-14-03-mozilla-central-l10n/firefox-4.0b12pre.ml.linux-i686.complete.mar" hashFunction="sha512" hashValue="89a6133206ce20f512ce5d29eecbd2d84966720c69b4499d6f1e7ad8837f7885efbbffbfbf759f46f6f8811ba40a109cd7681747689f3280ec033e7cc548862e" size="13372079"/>
    </update>
</updates>

The hash and size differs? 

FAIL Content check failed for Firefox|4.0b10pre|WINNT_x86-msvc|20110128030350|rm|nightly|3 (main:<?xml version="1.0"?>
<updates>
    <update type="minor" version="4.0b12pre" extensionVersion="4.0b12pre" buildID="20110214030347">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/02/2011-02-14-03-mozilla-central-l10n/firefox-4.0b12pre.rm.win32.complete.mar" hashFunction="sha512" hashValue="c6c5d94761d65e04c2bc916710cc61a0943bf2907568ef65111c6c4d3a4bf2f7c1c589e5e196540500add2588f78e493f0df4e3a6f83ed70a670176c6fdc88f6" size="15645629"/>
    </update>
</updates> test:<?xml version="1.0"?>
<updates>
    <update type="minor" version="4.0b12pre" extensionVersion="4.0b12pre" buildID="20110214030347">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/02/2011-02-14-03-mozilla-central-l10n/firefox-4.0b12pre.rm.win32.complete.mar" hashFunction="sha512" hashValue="f56238e9d9b057b3072ecc6040596be783e0d95d1c3eadbd847ce555c3ff2ae51a10460f23d18c54577322d13176c181f8778ff406344325c3a4fc2b9ae2668a" size="15631210"/>
    </update>
</updates>

Another hash and size difference

FAIL Header check failed for content-length NO MATCH, main content-length: 42 test content-length: 513
FAIL Content length check failed for Firefox|4.0b10pre|WINNT_x86-msvc|20110128030350|mai|nightly|3 (main=42 test=513

FAIL Header check failed for content-length NO MATCH, main content-length: 42 test content-length: 511
FAIL Content length check failed for Firefox|4.0b10pre|Darwin_x86_64-gcc3-u-i386-x86_64|20110201030339|mai|nightly|3 (main=42 test=511

And so on. Seems like the content-length match issues were all the main AUS not returning anything and the test servers were.

Nick, ideas?
fox2mike and I spent some time trying to debug what was going on here and came to the conclusion that pm-app-dist03 in MPT is misbehaving. So we need to fix that before we can make any conclusions about PHX.

Eg first example in comment #27
$ while [[ 1 ]]; do echo; rm update.xml; wget -S -O update.xml --no-check-certificate https://aus3.mozilla.org/update/3/Firefox/4.0b10pre/20110130030342/Linux_x86-gcc3/mai/nightly/AUS_Checker/default/default/update.xml 2>&1 | grep -E 'Backend|Content-Length'; openssl dgst -sha1 update.xml; done

  X-Backend-Server: pm-app-dist04
  Content-Length: 518
SHA1(update.xml)= 5e83d7fc6fbeda8a3776a713b0145bf469078bc8

  X-Backend-Server: pm-app-dist05
  Content-Length: 518
SHA1(update.xml)= 5e83d7fc6fbeda8a3776a713b0145bf469078bc8

  X-Backend-Server: pm-app-dist03
  Content-Length: 42
SHA1(update.xml)= cb6049f830a54d14a19d4104fc0bb5ab5fdedbe6

EG 2nd example in comment #27
$ while [[ 1 ]]; do echo; rm update.xml; wget -S -O update.xml --no-check-certificate https://aus3.mozilla.org/update/3/Firefox/4.0b10pre/20110131064712/Linux_x86-gcc3/ml/nightly/AUS_Checker/default/default/update.xml 2>&1 | grep -E 'Backend|Content-Length'; openssl dgst -sha1 update.xml; done

  X-Backend-Server: pm-app-dist05
  Content-Length: 517
SHA1(update.xml)= 22f580ad1b380666e76651004fe347de15281100

  X-Backend-Server: pm-app-dist03
  Content-Length: 517
SHA1(update.xml)= 411c56bfd023ed8d24727d395e9ebc2f2182e6da

How it would have the wrong file size and hash but the right URL & buildID I can't explain. 

We've made sure pm-app-dist03 has the right version of xml/inc/patch.class.php (rev 1.25 from bug 517947), and git thinks the code is fine in general. The files look to be mounted properly, but something is caching stale data or response. Bouncing Apache didn't help, maybe remount the nfs partition ?
So the culprit, pm-app-dist03 had an uptime of over 431 days, so I upgraded everything on it and rebooted the box and had Nick test again, to no avail.

This box returns a different response from the rest of the AUS nodes. I'm trying to see what's going on.
So basically, 03 seems to be doing no logic at all, running nothing to generate the right update.xml :

trinity:~ shyam$ cat update.xml-dist07 
<?xml version="1.0"?>
<updates>
    <update type="minor" version="4.0b12pre" extensionVersion="4.0b12pre" buildID="20110210030400">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/02/2011-02-10-03-mozilla-central-l10n/firefox-4.0b12pre.mai.linux-i686.complete.mar" hashFunction="sha512" hashValue="c6dc16af09f4ddd181219187436547bfdc6b8c63cc7f1158c71f62dd74a90e3165eb55a4c6131fbdb6280a00af7f95aead955887d5cc81956020f40d99e57369" size="13339218"/>
    </update>

</updates>trinity:~ shyam$ cat update.xml-dist03 
<?xml version="1.0"?>
<updates>
</updates>
We are pulling 03 out of the production pool until it can be figured out...
(In reply to comment #31)
> We are pulling 03 out of the production pool until it can be figured out...

This is done now. 

I spent some more time and got nowhere, I'll ping morgamic for ideas when he's around.
So once pm-app-dist03 was pulled from the pool, I re-ran the tests :

[shyam@boris aus-phx-test]$ for i in `ls nightlies/`; do echo -e Testing ${i}; ./aus-verify.pl nightlies/${i} | grep -v PASS; done;
Testing fennec-nightly
Testing fx4nightly
Testing fx4tracemonkey-nightly

No failz.

And...

[shyam@boris aus-phx-test]$ for i in `ls releases/`; do echo -e Testing ${i}; ./aus-verify.pl releases/${i} | grep -v PASS; done;
Testing fx3.0.19-mu
Testing fx3.5.16
Testing fx3.6.12
Testing fx3.6.13
Testing fx4.0b9
Testing tb2.0.0.24-mu

No failz.

I'd call this good to go, without that one bad node (which we may get around to debugging). Thoughts?
(In reply to comment #33)
> I'd call this good to go, without that one bad node (which we may get around to
> debugging). Thoughts?

nthomas, cshields: anything else you'd like done before we use this live in production?
Comment #33 is great progress. What changes did you make to the script to get around SSL issues ? Can we verify the SSL is working before enabling this for real ? There are certificates for both aus2.m.o and aus3.m.o which need to be installed. That's mostly comment #20

There are some additional checks in comment #19 that I think we should do. I'll get an URL to test for the PPC case.
(In reply to comment #35)
> Comment #33 is great progress. What changes did you make to the script to get
> around SSL issues ? Can we verify the SSL is working before enabling this for
> real ? There are certificates for both aus2.m.o and aus3.m.o which need to be
> installed. That's mostly comment #20

The early part of comment #27 explains that. Basically we didn't have aus3 setup in phx, and now we do. Two different public IPs though, same backends, different SSL certs.

> There are some additional checks in comment #19 that I think we should do. I'll
> get an URL to test for the PPC case.

Sure, get me those and I'll be happy to run them. Once that's done, we'll come up with a plan to turn on phx for AUS.
Attached file Throttling test — 
(In reply to comment #19)
> * throttling - hack the script to append '?force=1' to the query and use the
> firefox 3.5.16 snippets on the release channel (this'll work until 3.5.17 ships
> on Feb 8th). Should get nothing without the force, a major update with it

Call: perl verification-force.pl fx3.5.16
Attached file Fallback test — 
(In reply to comment #19)
> * verify that fallback works for partner builds, eg take the fx3.6.13 list and
> append '-cck-euballot' to the channel to get release-cck-euballot

Call: perl verification-fallback.pl fx3.5.16-fallback
(In reply to comment #37)
> Created attachment 513354 [details]
> Throttling test
> 
> (In reply to comment #19)
> > * throttling - hack the script to append '?force=1' to the query and use the
> > firefox 3.5.16 snippets on the release channel (this'll work until 3.5.17 ships
> > on Feb 8th). Should get nothing without the force, a major update with it
> 
> Call: perl verification-force.pl fx3.5.16

All clear :

[shyam@boris force]$ ./verification-force.pl fx3.5.16 | grep -v PASS
[shyam@boris force]$

(In reply to comment #38)
> Created attachment 513362 [details]
> Fallback test
> 
> (In reply to comment #19)
> > * verify that fallback works for partner builds, eg take the fx3.6.13 list and
> > append '-cck-euballot' to the channel to get release-cck-euballot
> 
> Call: perl verification-fallback.pl fx3.5.16-fallback

[shyam@boris cck-fallback]$ ./verification-fallback.pl fx3.5.16-fallback | grep -v PASS
[shyam@boris cck-fallback]$

Both of these are clear. I think Nick has a few more tests.
Attached file SSL Certificates test — 
(In reply to comment #20)
> Forgot something important. aus2.m.o and aus3.m.o use different SSL
> certificates - we need to make sure the PHX boxes have both. There's also a
> backup cert for aus3, but I'm not sure what the setup is for that.

Calls: perl verification-certs-aus3.pl fx4.0b9 
       perl verification-certs-aus2.pl fx3.5.16 

This is verifying the following headers:
main: client-ssl-cipher RC4-SHA
test: client-ssl-cipher RC4-SHA
main: client-ssl-cert-issuer /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
test: client-ssl-cert-issuer /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
main: client-ssl-warning Peer certificate not verified
test: client-ssl-warning Peer certificate not verified
main: client-ssl-cert-subject /serialNumber=vhiCn2kbHVxmzSKeR6dKZqOK/187nnD0/C=US/ST=California/L=Mountain View/O=Mozilla Corporation/OU=Secure Web Server/CN=*.mozilla.org
test: client-ssl-cert-subject /serialNumber=vhiCn2kbHVxmzSKeR6dKZqOK/187nnD0/C=US/ST=California/L=Mountain View/O=Mozilla Corporation/OU=Secure Web Server/CN=*.mozilla.org
but not actually showing that output.

Need to watch this still works when we make the changes to put PHX into production.
(In reply to comment #40)
> Created attachment 513399 [details]
> SSL Certificates test
> 
> (In reply to comment #20)
> > Forgot something important. aus2.m.o and aus3.m.o use different SSL
> > certificates - we need to make sure the PHX boxes have both. There's also a
> > backup cert for aus3, but I'm not sure what the setup is for that.

Not sure about that either, but I'm sure mrz knows and I'll make sure I check on this before we go live.

> Calls: perl verification-certs-aus3.pl fx4.0b9 
>        perl verification-certs-aus2.pl fx3.5.16 
> 
> This is verifying the following headers:
> main: client-ssl-cipher RC4-SHA
> test: client-ssl-cipher RC4-SHA
> main: client-ssl-cert-issuer /C=US/O=Equifax/OU=Equifax Secure Certificate
> Authority
> test: client-ssl-cert-issuer /C=US/O=Equifax/OU=Equifax Secure Certificate
> Authority
> main: client-ssl-warning Peer certificate not verified
> test: client-ssl-warning Peer certificate not verified
> main: client-ssl-cert-subject
> /serialNumber=vhiCn2kbHVxmzSKeR6dKZqOK/187nnD0/C=US/ST=California/L=Mountain
> View/O=Mozilla Corporation/OU=Secure Web Server/CN=*.mozilla.org
> test: client-ssl-cert-subject
> /serialNumber=vhiCn2kbHVxmzSKeR6dKZqOK/187nnD0/C=US/ST=California/L=Mountain
> View/O=Mozilla Corporation/OU=Secure Web Server/CN=*.mozilla.org
> but not actually showing that output.
> 
> Need to watch this still works when we make the changes to put PHX into
> production.

Passed.

[shyam@boris certs]$ ./verification-certs-aus2.pl fx3.5.16 | grep -v PASS
[shyam@boris certs]$

[shyam@boris certs]$ ./verification-certs-aus3.pl fx4.0b9 | grep -v PASS
[shyam@boris certs]$
For completeness :

01:07:39 < fox2mike> nthomas|away: we should put all these scripts into some source control
01:07:45 < fox2mike> nthomas|away: for the future :p
Attached file PPC test — 
(In reply to comment #19)
> * verify that requests for an update for Fx4.0b4 running on PowerPC get no
> updates. AUS is looking for 'PPC' in the useragent
>  http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88

Call: perl verification-ppc.pl fx4.0b3
(In reply to comment #43)
> Created attachment 513407 [details]
> PPC test
> 
> (In reply to comment #19)
> > * verify that requests for an update for Fx4.0b4 running on PowerPC get no
> > updates. AUS is looking for 'PPC' in the useragent
> >  http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88
> 
> Call: perl verification-ppc.pl fx4.0b3

Pass.

[shyam@boris ppc]$ ./verification-ppc.pl fx4.0b3  | grep -v PASS
[shyam@boris ppc]$
Blocks: 635650
No longer blocks: 631731, 621199
Attached file AUS2 tests (combined) — 
Nearly ready to deploy this for aus2.m.o and we discover most of the earlier testing compared aus3 in both locations. Need to do some rechecks to make sure aus2 is OK too.

Calls:
 perl _main/verification.pl _main/all
 perl cck-fallback/verification-fallback.pl cck-fallback/fx3.5.16-fallback
 perl certs/verification-certs-aus2.pl certs/fx3.5.16
 perl force/verification-force.pl force/fx3.5.16
 perl perl ppc/verification-ppc.pl ppc/fx4.0b3
(In reply to comment #45)
> Nearly ready to deploy this for aus2.m.o and we discover most of the earlier
> testing compared aus3 in both locations. Need to do some rechecks to make sure
> aus2 is OK too.

fox2mike will run these tonight and barring any failures we will deploy tomorrow morning.  Thanks Nick
(In reply to comment #45)
> Created attachment 515828 [details]
> AUS2 tests (combined)

Ran all there, tl;dr = All OK.

> Calls:
>  perl _main/verification.pl _main/all

[shyam@boris aus-phx-test-final]$ for i in `ls _main/`; do echo -e Testing ${i}; perl _main/verification.pl _main/${i} | grep -v PASS; done;
Testing fx3.0.19-mu
Testing fx3.5.16
Testing fx3.6.12
Testing fx3.6.13
Testing tb2.0.0.23

>  perl cck-fallback/verification-fallback.pl cck-fallback/fx3.5.16-fallback

[shyam@boris aus-phx-test-final]$ perl cck-fallback/verification-fallback.pl cck-fallback/fx3.5.16-fallback | grep -v PASS
[shyam@boris aus-phx-test-final]$

>  perl certs/verification-certs-aus2.pl certs/fx3.5.16

[shyam@boris aus-phx-test-final]$ perl certs/verification-certs-aus2.pl certs/fx3.5.16 | grep -v PASS
[shyam@boris aus-phx-test-final]$

>  perl force/verification-force.pl force/fx3.5.16

[shyam@boris aus-phx-test-final]$ perl force/verification-force.pl force/fx3.5.16 | grep -v PASS
[shyam@boris aus-phx-test-final]$

>  perl perl ppc/verification-ppc.pl ppc/fx4.0b3

[shyam@boris aus-phx-test-final]$ perl ppc/verification-ppc.pl ppc/fx4.0b3 | grep -v PASS
[shyam@boris aus-phx-test-final]$
Don't think there's anything left here to do. Re-open if that's the case.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: