510080 - Mac l10n nightly repacks do not update

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

We turned on the l10n nightly repacks this Tuesday and on Wednesday we realized that the Mac installers do not upload because the URL they check shows as <update></update>

For instance, the URL
https://aus2.mozilla.org/update/3/Firefox/3.6a2pre/20090810031114/Darwin_Universal-gcc3/af/nightly/Darwin%209.8.0/default/default/update.xml?force=1
I believe matches to
/opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_Universal-gcc3/20090810031114/af
in aus2-staging which contains the snippets (complete.txt and partial.txt) but the update is not provided. If we switch "af" for "en-US" we can see that the URL should be correct.

morgamic is right now looking into this doing some AUS debugging; he might be able to drop some light into it.

Comment 1

[Armen [:armenzg]]

Transferring notes from email thread.

Morgamic wrote:
###############
22:57 < morgamic> nthomas: joduinn: that anomoly that armenz was seeing is caused by the fact that there is not an ar locale in mozilla-central
22:58 < morgamic> the logic for nightlies is to always offer the latest build
22:58 < morgamic> the latest build according to our system is the second to last build directory
22:59 < morgamic> in our case,:
22:59 < morgamic> [morgamic@dm-ausstage01 Darwin_Universal-gcc3]$ cd 20090811103133/
22:59 < morgamic> [morgamic@dm-ausstage01 20090811103133]$ ls -l
22:59 < morgamic> total 4
22:59 < morgamic> drwxr-xr-x 2 ffxbld users 4096 Aug 11 12:21 en-US
22:59 < morgamic> that means, if you request with /update/3/Firefox/3.6a2pre/20090810031114/Darwin_Universal-gcc3/en-US/nightly/Darwin 
                  9.8.0/default/default/update.xml?force=1
23:00 < morgamic> it'll show up null because it's look at 20090811103133 for the latest updates and that has no /ar/ directory

So, question is why does 20090811103133 not have anything but an en-US build?  Do we need to change the logic in AUS?

Right now AUS is set to do this:
check the explicit dir (the ones we use for directories) -- if something exists, use that data
check partner fallback -- if something exists, use that data
check the _second to last_ build in the platform dir to offer the latest nightly -- if that exists, use it
if nothing found, offer null updates
If that default behavior should change, let's discuss what we should change it to.  I can write the accompanying tests to verify, etc.  Logic is described here:
AUS tests - http://mxr.mozilla.org/mozilla/source/webtools/aus/tests/
Screenshot of said tests - http://www.grabup.com/uploads/7392af50f2ebcf2339de1466288f5338.png
Actual test runner - http://khan.mozilla.org:8080/VerifyAus?test
Mike

Comment 2

[Armen [:armenzg]]

Nick Thomas wrote:
Ok, so I bet what happened to get that en-US only 20090811 was the Tuesday morning downtime pushing the en-US was 7 hours later than normal, so the l10n nightlies for that day used the 20090810 build (ie they ran before the new en-US completed). This should just come right tomorrow when we get a proper set of nightlies for both en-US and l10n. 

We could still hit this problem if there is bustage in the future, but I'm not sure how much time and effort is worth expending to fix it.

Comment 3

[Armen [:armenzg]]

I wrote:
This is not specific to one locale but to all locales for Mac (I rename the subject of this thread).

What Nick says it is right, the en-US nightly for Windows and Mac happened after the locales were finished due to the downtime and they were repackage off the 10th's en-US installers.
We will have to wait a couple of nights since today no l10n repackage succeeded due to version bumping.

I was wondering why on Wednesday (11th) the l10n updates for Windows did work but not for Mac even though both of them their nightlies happened later than l10n repackages.
What I think is the reason is that we never ended up having a nightly on the 11th for Windows even though we reschedule and forced it (builds #142 and #143) meaning that we did not end up with a directory with only "en-US" in it:
-bash-3.2$ ls Darwin_Universal-gcc3/ | grep 200908
...
20090810031114
20090811103133
20090812031634
20090813031939
-bash-3.2$ ls WINNT_x86-msvc/ | grep 200908
...
20090810050215
20090812045356
20090813042749

This problem of en-US happening after the l10n repackages happen should be fixed the day that we will have a TrigerableL10n scheduler rather than triggering both independently

Comment 4

[Armen [:armenzg]]

Nick Thomas wrote:
> Ok, so I bet what happened to get that en-US only 20090811 was the
> Tuesday morning downtime pushing the en-US was 7 hours later than
> normal, so the l10n nightlies for that day used the 20090810 build
> (ie they ran before the new en-US completed). This should just come
> right tomorrow when we get a proper set of nightlies for both en-US
> and l10n.

Unfortunately this didn't happen due to the version bump on trunk. Windows and Linux have (a reduced subset of) locales dirs on dm-ausstage01 for today, but still no Mac.

We *should* get more partials tomorrow, but that still doesn't explain why we are not even getting complete update offers for Mac l10n nightlies.

> We could still hit this problem if there is bustage in the future,
> but I'm not sure how much time and effort is worth expending to fix
> it.

Agreed, but only if we can make sure complete updates are always working as the fallback.

> If that default behavior should change, let's discuss what we should
> change it to. I can write the accompanying tests to verify, etc.
> Logic is described here:
> 
> * AUS tests -
> http://mxr.mozilla.org/mozilla/source/webtools/aus/tests/
> * Screenshot of said tests -
> http://www.grabup.com/uploads/7392af50f2ebcf2339de1466288f5338.png
> Actual test runner - http://khan.mozilla.org:8080/VerifyAus?test

Mike: do we need to add some l10n tests to that matrix, or would that not be testing anything new? I would be glad help with writing those tests if they would be useful.

Comment 5

[Armen [:armenzg]]

(In reply to comment #4)
Comment 4 was written by Chris Cooper. My only comments are comment 0 and comment 3.

Comment 6

[Axel Hecht [:Pike]]

Marking FIXED, just updated a German 3.7a1pre on the mac with an incremental update from the 15th to the 16th.

Comment 7

[Armen [:armenzg]]

Confirmed with es-MX.
Yay!