1040665 - Write an end-to-end test for funsize

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Anhad Jai Singh (:ffledgling)]

Attachment: Sample update.xml for use with generated MARs.

Funsize currently has an integration test that generates a Partial MAR from Firefox 28.0 to Firefox 29.0 (Release Channel) and uses the a pre-existing "correct" hash to verify the correctness of the generated partial.

We need a test that will do the full run through including generation and application of the partial to a recent version of firefox.

As I understand it, the test will have to do atleast the following:

1. Download a recent (but not latest) version of Firefox. (mozdownload)
2. Install it in a separate/isolated temporary directory. (mozinstall)
3. Setup a clean temporary testing profile for it. (mozprofile)
4. Update the profile to use the following parameters: (mozprofile or python/bash script)
   a. app.update.auto = false (Or can be kept at true depending on how test is written)
   b. app.update.url.override = http://localhost:13337/update.xml
   c. app.update.log = true
5. Serve the pre-defined update.xml on localhost (python's SimpleHTTPServer or mozhttpd)
6. Get firefox to acutally install the update and confirm that the update was sucessful. (See mozmill's update tests https://github.com/mozilla/qa-mozmill-tests/tree/master/firefox/tests/update, can also alternatively write a simpler test)
7. Perform Cleanup, remove:
   a. temporary profile
   b. download folder
   c. tarball/exe/dmg
   d. other test artifacts.

----------------------------------------

To get an idea of how this is supposed to work, it might be a good idea to first do a manual simulation of the test.

Here are the instructions curtsey Ben Hearsum (:bhearsum) :-

Here's how to get Firefox pointing at a custom MAR.

First, you need to create an update XML file. I'd suggest starting using this as a starting point: https://aus3.mozilla.org/update/1/Firefox/29.0.1/20140506152807/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml

Download it somewhere, and then tweak the <patch type="partial"> lines to point at your own MAR. Don't forget to update the hashValue (sha512) and filesize (bytes). Upload your modified update.xml somewhere.

Now, start up the version of Firefox that the partial applies to, create a new profile, and change the following prefs:
* app.update.log to true
* app.update.url.override to point directly at the update.xml you uploaded. Could be something like http://people.mozilla.org/~asingh/update.xml. Doesn't have to be on people.mozilla.org, though.
[you can host locally, use python -m SimpleHTTPServer and then use http://localhost:8080/update.xml]
[You can also use the update.xml attached with this bug instead]

Now you're ready to test! First open up the Browser Console (meta+J) and clear it. Then check for updates. You should see a bunch of lines that show Firefox talking to the server that hosts the update.xml, and then downloading, verifying, and applying the partial MAR.

---

Caveats and Advice:
If you download an old version of firefox and start it up, the browser will auto update silently and you will be unable to perform the test.
The recommended way to do this is, use your normal install of Firefox; Use the Profile manager to create a test profile and switch to that profile; Go to about:config and modify the aforementioned options; also set `app.update.auto` to `false`; close your browser.
Now launch the version under test with the profile you just setup;
Launch your Browser console;
Go to `About Firefox` and hit update.
Wait for the update to apply and observe logs.

Comment 1

[Harsimran Singh]

(In reply to Anhad Jai Singh (:ffledgling) from comment #0)
> Created attachment 8458547 [details]

> First, you need to create an update XML file. I'd suggest starting using
> this as a starting point:
> https://aus3.mozilla.org/update/1/Firefox/29.0.1/20140506152807/
> Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml/Darwin_x86_64-gcc3-
> u-i386-x86_64/en-US/release/update.xml
> 
> Download it somewhere, and then tweak the <patch type="partial"> lines to
> point at your own MAR. Don't forget to update the hashValue (sha512) and
> filesize (bytes). Upload your modified update.xml somewhere.

Need a little help here. Since i am running Ubuntu 14.04 (64-bit) and we need to update from version 29.0.1 to 31 whereas the current integration test does it for 28 to 29. So i went ahead and downloaded the partial MAR for my OS from 

http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/latest/update/linux-x86_64/en-US/

to update from 29.0.1 to 31. 

Now as far as modifying the update.xml is concerned i think the size here refers to size of the partial MAR downloaded. 

The url needs to be changed to the value where partial is, isn`t it ?
and os=osx&amp changes to os=Linux x86_64 ?

From where can i get the correct hashvalue form ?

> 
> Now, start up the version of Firefox that the partial applies to, create a
> new profile, and change the following prefs:
> * app.update.log to true
> * app.update.url.override to point directly at the update.xml you uploaded.

app.update.url.override was not present by default so i created it.

Hope i am not misusing the needinfo ?

PS. i have setup up the new profile, installed Firefox 20.0.1.

Comment 2

[Anhad Jai Singh (:ffledgling)]

(In reply to Harsimran Singh from comment #1)
> (In reply to Anhad Jai Singh (:ffledgling) from comment #0)
> > Created attachment 8458547 [details]
> 
> > First, you need to create an update XML file. I'd suggest starting using
> > this as a starting point:
> > https://aus3.mozilla.org/update/1/Firefox/29.0.1/20140506152807/
> > Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml/Darwin_x86_64-gcc3-
> > u-i386-x86_64/en-US/release/update.xml
> > 
> > Download it somewhere, and then tweak the <patch type="partial"> lines to
> > point at your own MAR. Don't forget to update the hashValue (sha512) and
> > filesize (bytes). Upload your modified update.xml somewhere.
> 
> Need a little help here. Since i am running Ubuntu 14.04 (64-bit) and we
> need to update from version 29.0.1 to 31 whereas the current integration
> test does it for 28 to 29. So i went ahead and downloaded the partial MAR
> for my OS from 
>

Not sure how the OS that you're using should matter here. If you're upgrading the system installed version of Firefox, please don't, instead download a separate instance from here: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

You can always download an old Firefox 28 and update it to 29, because there can exist multiple versions can co-exist simultaneously.

> http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/latest/
> update/linux-x86_64/en-US/
> 
> to update from 29.0.1 to 31. 
> 
> Now as far as modifying the update.xml is concerned i think the size here
> refers to size of the partial MAR downloaded. 

Correct, but be careful of the units, it's bytes rather than Kilobytes or Megabytes.
> 
> The url needs to be changed to the value where partial is, isn`t it ?
> and os=osx&amp changes to os=Linux x86_64 ?

You need to point the URL to a location on your own machine, so the parameters you send won't matter.
Those params are used by download.m.o to actually generate the partial and/or fetch them on the fly.
You can safely get rid of them.

I'll attach the version I used for personal testing.

> From where can i get the correct hashvalue form ?

Compute it :) (`sha512sum`)

> 
> > 
> > Now, start up the version of Firefox that the partial applies to, create a
> > new profile, and change the following prefs:
> > * app.update.log to true
> > * app.update.url.override to point directly at the update.xml you uploaded.
> 
> app.update.url.override was not present by default so i created it.

That's fine.

> 
> Hope i am not misusing the needinfo ?
> 
No, this is fine.

> PS. i have setup up the new profile, installed Firefox 20.0.1.

Are you sure you meant Firefox 20?

Comment 3

[Anhad Jai Singh (:ffledgling)]

Attachment: update.xml

Attaching a simpler update.xml to hopefully reduce confusion.

Comment 4

[Harsimran Singh]

(In reply to Anhad Jai Singh (:ffledgling) from comment #2)

> 
> Not sure how the OS that you're using should matter here. If you're
> upgrading the system installed version of Firefox, please don't, instead
> download a separate instance from here:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/
> 
> You can always download an old Firefox 28 and update it to 29, because there
> can exist multiple versions can co-exist simultaneously.

No i am not using the system installed version. Rather i downloaded Firefox 29.0.1 using mozdownload and then i also downloaded the corresponding partial file to update it to Firefox 31.0.

> I'll attach the version I used for personal testing.
>
Thanks, i modified it accordingly.
 
> > PS. i have setup up the new profile, installed Firefox 20.0.1.
> 
> Are you sure you meant Firefox 20?

No that was a typo. i intended to write 29.0.1 rather.

OBSERVATIONS:

1. Firefox on check for updates quickly locates the update.xml and then goes on and download it just at blink of an eye (think that is because it's hosted locally).

2. Restart and Firefox is updated.

3. Observed only two lines on the console before Firefox restarted.

GET http://localhost/xampp/update.xml [HTTP/1.1 200 OK 0ms]
GET http://localhost/xampp/firefox-29.0.1-31.0.partial.mar [HTTP/1.1 200 OK 180ms]

So i think i have managed to update it manually by pointing it at the partial MAR, isn`t it ?

Comment 5

[Anhad Jai Singh (:ffledgling)]

(In reply to Harsimran Singh from comment #4)

> OBSERVATIONS:
> 
> 1. Firefox on check for updates quickly locates the update.xml and then goes
> on and download it just at blink of an eye (think that is because it's
> hosted locally).
> 
> 2. Restart and Firefox is updated.
> 
> 3. Observed only two lines on the console before Firefox restarted.
> 
> GET http://localhost/xampp/update.xml [HTTP/1.1 200 OK 0ms]
> GET http://localhost/xampp/firefox-29.0.1-31.0.partial.mar [HTTP/1.1 200 OK
> 180ms]
> 
> So i think i have managed to update it manually by pointing it at the
> partial MAR, isn`t it ?

It seems like you've successfully managed to update the browser, I was expecting more output in the console though. Might be because of a variable I set, but I can't seem to recall what it was.

You might also want to check up with bhearsum on what existing infrastructure exists to test generated partials. I'm sure there is some.

NeedInfo'ing Ben here.

Ben: Any existing test infrastructure for applying partials and making sure they updated we could leverage for this end-to-end test?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Anhad Jai Singh (:ffledgling) from comment #5)
> (In reply to Harsimran Singh from comment #4)
> 
> > OBSERVATIONS:
> > 
> > 1. Firefox on check for updates quickly locates the update.xml and then goes
> > on and download it just at blink of an eye (think that is because it's
> > hosted locally).
> > 
> > 2. Restart and Firefox is updated.
> > 
> > 3. Observed only two lines on the console before Firefox restarted.
> > 
> > GET http://localhost/xampp/update.xml [HTTP/1.1 200 OK 0ms]
> > GET http://localhost/xampp/firefox-29.0.1-31.0.partial.mar [HTTP/1.1 200 OK
> > 180ms]
> > 
> > So i think i have managed to update it manually by pointing it at the
> > partial MAR, isn`t it ?
> 
> It seems like you've successfully managed to update the browser, I was
> expecting more output in the console though. Might be because of a variable
> I set, but I can't seem to recall what it was.
> 
> You might also want to check up with bhearsum on what existing
> infrastructure exists to test generated partials. I'm sure there is some.
> 
> NeedInfo'ing Ben here.
> 
> Ben: Any existing test infrastructure for applying partials and making sure
> they updated we could leverage for this end-to-end test?

We use our "update verify" tests to apply updates without launching the browser. It'd require a fair bit of hacking up to use here, though. All the helper functions it uses are over here: https://github.com/mozilla/build-tools/tree/master/release/common

I'm a bit confused as to why applying MARs with an application is part of this, though...that's well beyond the scope of Funsize.

Comment 7

[Harsimran Singh]

> I'm a bit confused as to why applying MARs with an application is part of
> this, though...that's well beyond the scope of Funsize.

Anhad what should be done next about this bug ?

Comment 8

[Anhad Jai Singh (:ffledgling)]

(In reply to Ben Hearsum [:bhearsum] from comment #6)

> We use our "update verify" tests to apply updates without launching the
> browser. It'd require a fair bit of hacking up to use here, though. All the
> helper functions it uses are over here:
> https://github.com/mozilla/build-tools/tree/master/release/common
> 
> I'm a bit confused as to why applying MARs with an application is part of
> this, though...that's well beyond the scope of Funsize.

Given the number of changes that break the way we're testing our generated MARs in Funsize (Verifying hashes against a "known good" version) I thought it would be best if we resorted to testing the MARs directly by applying them.
Right now any change that changes the hash of the MAR causes tests to fail, and more often than not it's because the older "known good" hash is no longer valid.

Do you have suggestions for a better way to go about this?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Anhad Jai Singh (:ffledgling) from comment #8)
> (In reply to Ben Hearsum [:bhearsum] from comment #6)
> 
> > We use our "update verify" tests to apply updates without launching the
> > browser. It'd require a fair bit of hacking up to use here, though. All the
> > helper functions it uses are over here:
> > https://github.com/mozilla/build-tools/tree/master/release/common
> > 
> > I'm a bit confused as to why applying MARs with an application is part of
> > this, though...that's well beyond the scope of Funsize.
> 
> Given the number of changes that break the way we're testing our generated
> MARs in Funsize (Verifying hashes against a "known good" version) I thought
> it would be best if we resorted to testing the MARs directly by applying
> them.
> Right now any change that changes the hash of the MAR causes tests to fail,
> and more often than not it's because the older "known good" hash is no
> longer valid.
> 
> Do you have suggestions for a better way to go about this?

(Sorry if we've covered this before - it feels like ages ago since we talked.)

A possible alternative is to shrink down your test cases. You don't necessarily need to generate a MAR for an entire Firefox build. You could build two small MARs and generate a partial from them instead. That might get you to a place where you have predictable results. That may make it easier to have tests for different update manifest formats in the future, too.

If you still want to go with your original plan I'm not sure what other advice to offer -- the update verify code that I pointed at is the only thing we have that may help.