497653 - Test partial mar generation for l10n nightlies on AUS

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

Nick: my apologies if this has already been done, but I lost track of this issue over the last few days.

Armen has generated some one-off snippet/complete mars that we are hoping are enough, AUS-willing, to generate a partial mar for the purposes of testing l10n nightly updates.

I *think* Nick has all the pieces he needs to run this test now, but I'm sure he'll let Armen know if that's not the case.

Comment 1

[Armen [:armenzg]]

I just checked all the pieces that I had left in place and the scripts, the snippets and the MAR files are living under /home/ffxbld/armen

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #1)
> I just checked all the pieces that I had left in place and the scripts, the
> snippets and the MAR files are living under /home/ffxbld/armen

This is now superseded by running staging over the weekend with the snippet production in place. However staging-stage had a storage problem and went read-only on the disk where builds would be uploaded to, so we missed lots bits.

Given we also expire builds more than a few days old, it would be really great if Armen could generate two "nightly" builds in staging, that
* have some small set of locales (say more than 3, less than 10)
* have all three platforms for both nightlies
* they can be just a few hours apart, small or no difference doesn't matter at this point
You will probably need to disable some HgPollers and scheduled builds so that the slaves you've got aren't swamped with other jobs.

Then I can try to get the partial update generator running on staging-stage and you can proceed independently of me.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to comment #2)
> This is now superseded by running staging over the weekend with the snippet
> production in place. However staging-stage had a storage problem and went
> read-only on the disk where builds would be uploaded to, so we missed lots
> bits.

Ah, nuts. Never do get a break, do we?

> Given we also expire builds more than a few days old, it would be really great
> if Armen could generate two "nightly" builds in staging, that
> * have some small set of locales (say more than 3, less than 10)
> * have all three platforms for both nightlies
> * they can be just a few hours apart, small or no difference doesn't matter at
> this point
> You will probably need to disable some HgPollers and scheduled builds so that
> the slaves you've got aren't swamped with other jobs.
> 
> Then I can try to get the partial update generator running on staging-stage and
> you can proceed independently of me.

Armen: any chance you can generate these new sets of builds today?

Comment 4

[Armen [:armenzg]]

(In reply to comment #3)
> (In reply to comment #2)
> > Given we also expire builds more than a few days old, it would be really great
> > if Armen could generate two "nightly" builds in staging, that
> > * have some small set of locales (say more than 3, less than 10)
> > * have all three platforms for both nightlies
> > * they can be just a few hours apart, small or no difference doesn't matter at
> > this point
> > You will probably need to disable some HgPollers and scheduled builds so that
> > the slaves you've got aren't swamped with other jobs.
> > 
> > Then I can try to get the partial update generator running on staging-stage and
> > you can proceed independently of me.
> 
> Armen: any chance you can generate these new sets of builds today?
Yes, will do.

Comment 5

[Armen [:armenzg]]

I have reduced the numbers of locales to: cs, fr, gl, it, lt and pl.
I have put the MAR files under /home/ffxbld/armen/test_partial_l10n_updates on staging-stage:
2009-06-15-03-mozilla-1.9.1-l10n/ (linux & mac)
2009-06-15-04-mozilla-1.9.1-l10n/ (win32)
2009-06-12-04-mozilla-1.9.1-l10n/ (win32)
2009-06-12-05-mozilla-1.9.1-l10n/ (mac)
2009-06-12-12-mozilla-1.9.1-l10n/ (linux)
and the relatives snippets on cltbld's account rather than ffxbld in /home/cltbld/armen/mozilla-1.9.1 (I am only considering one branch - Nick tell me if you need mozilla-central as well)


NOTE: The snippets are pointing to staging-stage rather than ftp.mozilla.org (the following is a line of complete.txt)
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly//2009/06/2009-06-12-05-mozilla-1.9.1-l10n/firefox-3.5pre.cs.mac.complete.mar

Nick, if you have a chance could you please look at this other comment https://bugzilla.mozilla.org/show_bug.cgi?id=496196#c30

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I really wanted to sets of builds as produced by the staging master instance, and to not introduce any problems from moving stuff around, but I'll see what I can do.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I had to use dos2unix on the windows snippets, hack up the generator a bit, and update AUS to the latest code, but otherwise generating partials worked OK. Give it a whirl by taking one of the 20090612 builds and using the trick in bug 496196 comment 40. Here's an update url too,
http://staging-stage.build.mozilla.org/update/3/Firefox/3.5pre/20090612121500/Linux_x86-gcc3/cs/nightly/Linux%202.6.28-11-generic%20%28GTK%202.16.1%29/default/default/update.xml

It took about 50 seconds to generate each linux and windows partial, and about 100 seconds for mac (yay, univeral builds), so that's about four hours per branch of 70 locales. Yuck. This code has never been optimised for lots of work because it only generates 14 partials each day, but I think it should die anyway. Can we live with that in the meantime ?

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to comment #7)
> I had to use dos2unix on the windows snippets, hack up the generator a bit, and
> update AUS to the latest code, but otherwise generating partials worked OK.

Where did you end up generating the partials? Was this on staging-stage? I'm just wondering how big a delta this was from the existing en-US process. 

I'd love to get a proper internal partial update/AUS system setup for testing purposes sometime next quarter.

> Give it a whirl by taking one of the 20090612 builds and using the trick in bug
> 496196 comment 40. Here's an update url too,
> http://staging-stage.build.mozilla.org/update/3/Firefox/3.5pre/20090612121500/Linux_x86-gcc3/cs/nightly/Linux%202.6.28-11-generic%20%28GTK%202.16.1%29/default/default/update.xml

Armen, that's your cue.

> It took about 50 seconds to generate each linux and windows partial, and about
> 100 seconds for mac (yay, univeral builds), so that's about four hours per
> branch of 70 locales. Yuck. This code has never been optimised for lots of work
> because it only generates 14 partials each day, but I think it should die
> anyway. Can we live with that in the meantime ?

4 hours? Wow, and also, ugh. We're not using the slave that does partial generation for anything else, are we? I'm not too worried about 4 hours in that case.
 
Definitely something we can improve on next quarter. Whether that means allowing slaves to generate their own partials or something else.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #8)
> Where did you end up generating the partials? Was this on staging-stage? I'm
> just wondering how big a delta this was from the existing en-US process. 
> 
> I'd love to get a proper internal partial update/AUS system setup for testing
> purposes sometime next quarter.

I set it up on staging-stage, with a copy of what we use on prometheus-vm. There was some slight config tweaking (usernames, locations), and disabling some Thunderbird specific stuff, but to all intents and purposes it's the same. We could probably set it up to run from cron and see how often it breaks.
 
> Armen, that's your cue.

Please be fixing the DOS line endings (on the windows snippets) on the buildbot side too.
 
> 4 hours? Wow, and also, ugh. We're not using the slave that does partial
> generation for anything else, are we? I'm not too worried about 4 hours in that
> case.

That's all prometheus-vm does, and the l10n nightlies start long enough after the en-US ones that we're probably not going to block those. Just need to watch out for the long running m-1.9.1/m-c win32 nightlies.

Comment 10

[Armen [:armenzg]]

(In reply to comment #9)
> (In reply to comment #8)
> > Armen, that's your cue.
> 
> Please be fixing the DOS line endings (on the windows snippets) on the buildbot
> side too.
> 
I have tested the python script on windows and I have to write the file with the parameter 'wb' instead of just 'w'. The extra 'b' has no effect on either mac nor linux.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #9)
> We could probably set it up to run from cron and see how often it breaks.

Can't do this at the moment because the en-US snippets say the files are files at ftp://ftp.mozilla.org/ instead of http://staging-stage.build.mozilla.org. Looks like this comes from DOWNLOAD_BASE_URL in mozilla2-staging/config.py. I'd suggest changing it, but that var is also used for LATEST_MAR_URL. That's old or incoming changes ?

Comment 12

[Armen [:armenzg]]

Attachment: mozilla2-staging/config.py - Change DOWNLOAD_BASE_URL to the correct value

(In reply to comment #11)
> (In reply to comment #9)
> > We could probably set it up to run from cron and see how often it breaks.
> 
> Can't do this at the moment because the en-US snippets say the files are files
> at ftp://ftp.mozilla.org/ instead of http://staging-stage.build.mozilla.org.
> Looks like this comes from DOWNLOAD_BASE_URL in mozilla2-staging/config.py. I'd
> suggest changing it, but that var is also used for LATEST_MAR_URL. That's old
> or incoming changes ?
>
In my patches DOWNLOAD_BASE_URL has the correct value. I am attaching the patch in case we want to have it running before my changes go in.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 383911 [details] [diff] [review]
mozilla2-staging/config.py - Change DOWNLOAD_BASE_URL to the correct value

Looks fine to me, but I'd be interested to know what LATEST_MAR_URL is for, or used to be for. Coop said it was unused, so perhaps we should remove it from the staging config too (not necessarily here). Minimizing the differences between production and staging is a worthy goal.

Comment 14

[Armen [:armenzg]]

I have been able to update the French locale from the 12th to the 15th but I don't think that I update through a partial update.
It probably has to do that on latest-mozilla-1.9.1-l10n the installer is from the regular l10n nightly repackages that are running on staging-master. I will test this again sometime next week.

This is the update URL:
http://staging-stage.build.mozilla.org/update/3/Firefox/3.5pre/20090612053046/Darwin_Universal-gcc3/fr/nightly/Darwin%209.7.0/default/default/update.xml?force=1&force=1

I had to hack the 2 following files to point to http://localhost:8083
incoming/2/Firefox/mozilla-1.9.1/Darwin_Universal-gcc3/20090612053046/fr/complete.txt 
incoming/2/Firefox/mozilla-1.9.1/Darwin_Universal-gcc3/20090612053046/fr/partial.txt
to point to localhost:8083 instead of staging-master (I am wainting on IT to trouble shoot me with my VPN)


Here is the link to the partial MAR (http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-12-05-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.partial.20090612053046-20090615031034.mar). We can see in the following log that it was attempted to be downloaded but I don't know why it failed:
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC Downloader:downloadUpdate - downloading from http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-12-05-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.partial.20090612053046-20090615031034.mar to /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.mar
*** AUS:SVC Downloader:onStartRequest - spec: http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-12-05-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.partial.20090612053046-20090615031034.mar
*** AUS:SVC Downloader.onProgress:onProgress - progress: 26191/26191
*** AUS:SVC Downloader:onStopRequest - spec: http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-12-05-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.partial.20090612053046-20090615031034.mar, status: 0
*** AUS:SVC Downloader:onStopRequest - setting state to: pending
*** AUS:SVC UpdateService:canUpdate - testing /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/update.test
*** AUS:SVC UpdateService:canUpdate - testing /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.test
*** AUS:SVC UpdateService:canUpdate - able to update
*** AUS:SVC General:readStatusFile - status: failed: 4, path: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.status
*** AUS:SVC General:cleanUpUpdatesDir - successfully removed update directory: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0
*** AUS:SVC UpdateService:_postUpdateProcessing - install of partial patch failed, downloading complete patch
*** AUS:SVC General:getUpdatesDir - update directory /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0 doesn't exist, creating...
*** AUS:SVC General:readStringFromFile - file doesn't exist: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.status
*** AUS:SVC General:readStatusFile - status: null, path: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.status
*** AUS:SVC Downloader:_selectPatch - found existing patch with state: null
*** AUS:SVC Downloader:downloadUpdate - downloading from http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-15-03-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.complete.mar to /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.mar
*** AUS:SVC UpdateService:downloadUpdate - no support for downloading more than one update at a time
*** AUS:SVC General:readStatusFile - status: downloading, path: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.status
*** AUS:SVC Downloader:onStartRequest - spec: http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-15-03-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.complete.mar
*** AUS:SVC Downloader.onProgress:onProgress - progress: 300000/17570877
*** AUS:SVC General:readStatusFile - status: downloading, path: /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.status
*** AUS:SVC Downloader:_selectPatch - found existing patch with state: downloading
*** AUS:SVC Downloader:_selectPatch - resuming download
*** AUS:SVC Downloader:downloadUpdate - downloading from http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-15-03-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.complete.mar to /Users/armenzg/Desktop/15th - 1.9.1/Shiretoko.app/Contents/MacOS/updates/0/update.mar
*** AUS:SVC Downloader:onStopRequest - spec: http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-15-03-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.complete.mar, status: 2152398850
*** AUS:SVC Downloader:onStopRequest - setting state to: downloading
*** AUS:SVC Downloader:onStartRequest - spec: http://localhost:8083/pub/mozilla.org/firefox/nightly//2009/06/2009-06-15-03-mozilla-1.9.1-l10n/firefox-3.5pre.fr.mac.complete.mar
*** AUS:SVC Downloader.onProgress:onProgress - progress: 600000/17570877
*** AUS:SVC Downloader.onProgress:onProgress - progress: 900000/17570877

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

It works fine for me, using just a tunnel and not the full VPN. I verified that Contents/MacOS/updates/0/update.mar was 26191 bytes, so definitely the partial, and it applies fine. No idea why the updater would be failing to read/write update.status for you.

Comment 16

[Armen [:armenzg]]

These are good news. Thanks for trying this.

Comment 17

[Chris Cooper [:coop] (he/him)]

(In reply to comment #15)
> It works fine for me, using just a tunnel and not the full VPN. I verified that
> Contents/MacOS/updates/0/update.mar was 26191 bytes, so definitely the partial,
> and it applies fine. No idea why the updater would be failing to read/write
> update.status for you.

Nice.

Comment 18

[Armen [:armenzg]]

Attachment: remove LATEST MAR URL value from staging's config.py file

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 384471 [details] [diff] [review]
remove LATEST MAR URL value from staging's config.py file

Looks good, r+

Comment 20

[Armen [:armenzg]]

Comment on attachment 383911 [details] [diff] [review]
mozilla2-staging/config.py - Change DOWNLOAD_BASE_URL to the correct value

no need to land this since it is already in one of my other patches which will soon be landing

Comment 21

[Chris Cooper [:coop] (he/him)]

Comment on attachment 384471 [details] [diff] [review]
remove LATEST MAR URL value from staging's config.py file

changeset:   1219:4761562a2c3a

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Bug 502612 set this up in an automated way, but the manual tests that this bug was for are done.