378526 - Requirements for a comprehensive download link checker to use before release

Status: REOPENED

(Testing :: General, defect, P3)

Description

[Marcia Knous [:marcia]]

This was discussed at the Thunderbird Post Mortem held on 4-23-07. Our current method for checking links and checking things in general pre-release could be improved. This bug is about developing a set of requirements for a download link checker and any other scripts that could be developed to help catch possible issues before release, instead of issues surfacing on the day we are shipping.

*What are the actual requirements?
*Ensure that we test the website
*Ensure that we Test the bouncer URLS out of band of the website
*Confirm what previous versions of bc's script did and figure out if there are other useful things it could test.

I added some folks to the cc list - I will send out mail to the group that attended the meeting in case others want to add insight or track the development.

Comment 1

[Bob Clary [:bc] (inactive)]

(In reply to comment #0)

> *Confirm what previous versions of bc's script did and figure out if there are
> other useful things it could test.

1. Access the specified "all" download page, e.g. <http://www.mozilla.com/en-US/firefox/all.html>.

2. Output to a file a list of links constructed from the "download" links on the page for the specific OS where the script was run. For example, if run on Mac OS X, the link

<http://www.mozilla.com/products/download.html?product=firefox-2.0.0.3&os=win&lang=ar>

would be output as
 
href: http://download.mozilla.org/?product=firefox-2.0.0.3&os=win&lang=ar

3. For each url in the file created in step 2, 

3a. download the file using curl, 
3b. using the "utility" scripts described in the "Firefox and Thunderbird testing" posts on bclary.com, install the downloaded file, create and initialize a new profile, install the Spider extension, then start the build with the Spider extension pointing to a web page|userhook script which uses XHR to post build specific data to <http://bclary.com/log/2006/02/24/builds.html>.

This isn't what I would describe as a test with a PASS|FAIL type result. It does not currently check that specific builds are available (version, locale, platform). The output "builds.html" must be reviewed by hand and any missing builds must be investigated by looking in the logs for the reason. The typical failure mode is due to a bad mirror which is ignorable.

This process actually misses one of the major things which I expect most people think it does test. It really just tries to download from download.mozilla.org and install each of the locales specified on the "all" page. It should instead load the original link, make sure the new page loads correctly, check that the download is initiated, download the file, then install it, etc. I don't know if there are enough scriptable hooks into the browser to control this from a web page. It may be necessary to perform the test using something like eggplant. Ping Tracy.

The problem I had with Thunderbird 2 came up in step 3b where it was necessary to invoke Thunderbird itself and then Thunderbird+Spider pointing to a web page.

Finally, it can take well over 30 minutes to run the download check and sometimes considerably longer depending on the mirrors the downloads hit. If you get a bunch of mirrors with "dialup speeds" you can expect the test to run well over an hour. This makes running test on the staged version of the page prior to the release a requirement. The current situation where the tool was run after the release went live made it useless in my opinion and nothing more than a checkbox on a list that provided no benefit.

Comment 2

[Rob Campbell [:rc] (:robcee)]

Hey Bob. Have you got any code you could submit for this? This came up again during triage and we think it would still be great to have and automate.

Comment 3

[Bob Clary [:bc] (inactive)]

I have the pieces I described in comment 1. It uses Spider+Sisyphus. The code is actually checked into testing/sisyphus/tests/mozilla.org/download-page/

Comment 4

[Rob Campbell [:rc] (:robcee)]

Not sure this belongs to Release Engineering. Bumping to Core::Testing for reclassification.

Comment 5

[(not currently active) Ted Mielczarek]

No fair, Core:Testing is where bugs go do die. RelEng can retriage, but not back to here.

Comment 6

[cmtalbert]

(In reply to comment #5)
> No fair, Core:Testing is where bugs go do die. RelEng can retriage, but not
> back to here.

Honestly, Ted, to me this sounds like a "New Frameworks" bug (Product: Testing, Component: New Frameworks).  I read Marcia's description as a tool that QA needs to run manually and simplify their release process.  I don't think it's something that RelEng needs to be involved in as it doesn't have a build automation component (unless we convince them to add it to their release automation script's verification section).

My two cents...let's see what RelEng and build have to say.

Also, Marcia, did the stuff bc put together work to solve the problem?  Is this still an issue?

Comment 7

[[e-mail] (:bhearsum)]

We already have a test which checks Bouncer URLs for all past releases. It sounds like Bob was doing a bit more, but I'm not entirely clear what it's about.

Is this still desired?
Is this something that falls under RelEng? Other than the Bouncer URL test it feels much more like a QA thing.

Comment 8

[[e-mail] (:bhearsum)]

Moving this to New Frameworks based on Clint's comments.

Comment 9

[[e-mail] (:bhearsum)]

Oops, didn't mean to resolve this.