Closed Bug 741756 Opened 12 years ago Closed 12 years ago

l10n dep builds update to 'default' instead of l10n revision

Categories

(Release Engineering :: General, defect, P3)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Assigned: catlee)

References

Details

(Whiteboard: [l10n][fixed by bug 791209]) 

As I have noticed recently by our L10n PoC for Mozmill CI there seems to be a broken behavior in building/publishing l10n tinderbox builds to the FTP server across platforms. While those locales get daily builds the tinderbox ones are missing.

For example take the fi locale:
http://hg.mozilla.org/releases/l10n/mozilla-aurora/fi/log

It has been changed on April 2nd and it should have been triggered a build process for a tinderbox build. Sadly that didn't seem to happen. I have never seen a notification via Pulse and also there is no fi locale present in the tinderbox folder:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-aurora-l10n/

Beside that I'm also missing 'de' builds. The latest change was on March 14th, which should have triggered a build. The question I have here is, do we remove builds after a given amount of time with no change happen?
Whiteboard: [buildduty]
Yes, I believe builds are deleted after a few days.
Priority: -- → P3
Whiteboard: [buildduty] → [buildduty][l10n]
I didn't find any signs of the fi build for firefox on tinderbox either, though there might be one for thunderbird. don't understand the tb buildernames, so I'm not really sure.
are the builds you're looking for in here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/04/2012-04-03-04-20-11-mozilla-aurora-l10n/

the logs are there as well, so I'd be really surprised if no pulse events were generated for them.
catlee, that's nightlies. This bug is about tinderbox-builds, triggered on l10n landings.
Severity: normal → major
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Those have been build today but were not present before. So please check:
> 
> http://hg.mozilla.org/releases/l10n/mozilla-aurora/hi-IN/ (missing)

last checkin here was march 28, so it probably got cleaned up

> http://hg.mozilla.org/releases/l10n/mozilla-aurora/pt-PT/ (should be March
> 22)

ditto

> http://hg.mozilla.org/releases/l10n/mozilla-aurora/uk/ (missing)

ditto

> http://hg.mozilla.org/releases/l10n/mozilla-aurora/zu/ (missing or too old?)

ditto.

all of these are older than a week, so pretty sure they've been deleted by now. we don't keep the per-change l10n builds very long since we build the full set every night.
(In reply to Chris AtLee [:catlee] from comment #7)
> > http://hg.mozilla.org/releases/l10n/mozilla-aurora/hi-IN/ (missing)
> 
> last checkin here was march 28, so it probably got cleaned up
> 
> all of these are older than a week, so pretty sure they've been deleted by
> now. we don't keep the per-change l10n builds very long since we build the
> full set every night.

There are builds in this FTP folder from March 14th. So why those have not been cleaned-up? Looks like it's a bit inconsistent. Can anyone give a clear statement on how long we retain those builds?
I hope this helps:
#0 4 * * * nice -n 19 find /pub/mozilla.org/mobile/tinderbox-builds/*-l10n -type f -mtime +1 -exec rm {} \;

# l10n cleanup
## cleanup dep build logs
@daily nice -n 19 find /pub/mozilla.org/firefox/tinderbox-builds/*-l10n -type f -mtime +29 -name *txt.gz -exec rm -f {} \;
## clean up logs from latest builds
30 1 * * * for i in central aurora 1.9.2; do nice -n 19 find /home/ftp/pub/firefox/nightly/latest-mozilla-${i}-l10n/ -type f -name \*.txt.gz -mtime +2 -exec rm -f {} \; ; done
## clean up localized nightly builds older than a few days, to keep disk usage under control but still provide nightly updates
0  23 * * * find /pub/mozilla.org/{firefox,mobile}/nightly/201? -mindepth 2 -maxdepth 2 -type d -name '*-mozilla-*-l10n' -mtime +5 -exec rm -rf {} \; && symlinks -d /pub/mozilla.org/{firefox,mobile}/nightly > /dev/null
40 1 * * * for i in central aurora 1.9.2; do nice -n 19 find /home/ftp/pub/firefox/nightly/latest-mozilla-${i}-l10n/ -type f ! -name \*.txt.gz ! -name README\* -mtime +2 -exec rm -f {} \; ; done
## clean up dep builds older than latest version
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/mozilla-central/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-l10n -type f -name "firefox*" ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-aurora-l10n -type f -name "firefox*" ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
@hourly VERSION=`wget --quiet -O- http://hg.mozilla.org/releases/mozilla-1.9.2/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-1.9.2-l10n -type f -name "firefox*" ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
So I can confirm that no builds were triggered for http://hg.mozilla.org/releases/l10n/mozilla-aurora/hi-IN/rev/4fbc1fe5d5f8, and AFAICT there should have been. Unfortunately the master logs from this time are now gone.

Are there more examples of more recent repacks that are missing?
https://l10n-dev-sj.mozilla.org/source/pushes/?repo=l10n/mozilla-aurora shows recent pushes to l10n/mozilla-aurora, that'd be the incoming data, I guess.
There was a push today for rm:
https://l10n-dev-sj.mozilla.org/source/pushes/releases/l10n/mozilla-aurora/rm

The builds for this locale are still from yesterday:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/

Also check the lij locale:

firefox-13.0a2.lij.linux-x86_64.tar.bz2 10-Apr-2012
firefox-13.0a2.lij.mac.dmg              08-Apr-2012 09:16 39M
firefox-13.0a2.lij.win32.zip            09-Apr-2012 08:40 21M 

Why are builds not getting build on all platforms at the same time? Mac is kinda outdated. Or is there something wrong with the dates on FTP?
Assignee: nobody → catlee
(In reply to Henrik Skupin (:whimboo) from comment #12)
> There was a push today for rm:
> https://l10n-dev-sj.mozilla.org/source/pushes/releases/l10n/mozilla-aurora/rm
> 
> The builds for this locale are still from yesterday:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-
> l10n/
> 
> Also check the lij locale:
> 
> firefox-13.0a2.lij.linux-x86_64.tar.bz2 10-Apr-2012
> firefox-13.0a2.lij.mac.dmg              08-Apr-2012 09:16 39M
> firefox-13.0a2.lij.win32.zip            09-Apr-2012 08:40 21M 
> 
> Why are builds not getting build on all platforms at the same time? Mac is
> kinda outdated. Or is there something wrong with the dates on FTP?

According to http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/mozilla-aurora-macosx64-l10n-nightly-lij-bm26-build1-build2504.txt.gz the macosx64 builds repack for the 10th finished fine. However the build on the 9th failed:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/mozilla-aurora-macosx64-l10n-nightly-lij-bm26-build1-build2466.txt.gz

Repacks are triggered after the en-US builds finish. The en-US builds start at around 3am pacific, but when they finish is variable depending on if there are available slaves, or how long the en-US build takes. It's not unusual for l10n repacks to still be running for many hours after the en-US builds are done.
This bug is about dep l10n builds, not about nightlies. whimboo labeled them as tinderbox builds, as their ftp dir is named, resummarizing.
Summary: Tinderbox builds missing for some locales → l10n dep builds missing for some locales
I was responding to whimboo's latest comment which was referring to nightly builds. Are there problems with both?
The only missing dep build I can find is for http://hg.mozilla.org/releases/l10n/mozilla-aurora/hi-IN/rev/4fbc1fe5d5f8. 

For dep repacks, we only do builds if files in the following directories change:
browser
extensions/reporter
other-licenses/branding/firefox
netwerk
dom
toolkit
security/manager
(In reply to Chris AtLee [:catlee] from comment #15)
> I was responding to whimboo's latest comment which was referring to nightly
> builds. Are there problems with both?

That was an accident. Sorry for the confusion. Axel, what do you think? Are we ok?
I've created a sibling dashboard to the nightlies now, http://l10n.mozilla-community.org/~axel/l10n-deps/. As unstable and basically unsupported, but I figured better with some data than without.

I only processed the data for this month, but there are several gaps in which platforms show up for a given l10n revision.

PS: Filed bug 744864 to get the watched directories updated to be in sync with l10n.ini again.
(In reply to Axel Hecht [:Pike] from comment #18)
> I've created a sibling dashboard to the nightlies now,
> http://l10n.mozilla-community.org/~axel/l10n-deps/. As unstable and
> basically unsupported, but I figured better with some data than without.

Sweet, we should sit together and see how to even integrate the results from the Mozmill tests. Everything can be accessed via a RestAPI so it shouldn't be that hard.
I don't think that we should extend that hack to display richer results.
(In reply to Axel Hecht [:Pike] from comment #18)
> I've created a sibling dashboard to the nightlies now,
> http://l10n.mozilla-community.org/~axel/l10n-deps/. As unstable and
> basically unsupported, but I figured better with some data than without.
> 
> I only processed the data for this month, but there are several gaps in
> which platforms show up for a given l10n revision.

It would be super helpful to have links to specific revisions, platforms, etc. when you notice things missing. Saves me from having to guess at what you're looking for.

For example for the the 16th, I do see linux64 and win32 builds in the database completing successfully for http://hg.mozilla.org/l10n-central/gl/pushloghtml?changeset=ef78b65bd4d9.
So the issue here is caused by two pushes close in time to each other. The buildbot code doesn't update to a specific 110n revision, it just updates to the latest revision. In this case you end up with 2 jobs for the latest revision, instead of one job per revision.
Summary: l10n dep builds missing for some locales → l10n dep builds update to 'default' instead of l10n revision
Whiteboard: [buildduty][l10n] → [l10n]
So what's the status here? Axel, are we ok with the current behavior?
I guess we're not optimal here. If I look at http://l10n.mozilla-community.org/~axel/l10n-deps/?date=2012-05-27, notably http://hg.mozilla.org/l10n-central/sv-SE/pushloghtml?changeset=5445af89e21c, I see a depend build for each changeset in the single push (with 7 changesets). I guess that'd trigger a test run each, if that happened on aurora. Which I expect to happen at least around migration for a few locales.
Axel, given that we do not run l10n tests on mozilla-central builds right now, it would be great to have the l10n-deps table for mozilla-aurora. That would make it easier to compare check-in changes, created builds, and our results.
The l10n-deps table is for all versions, I don't have cycles to restrict that to particular branches right now, sorry.
The root of this problem is here:
http://hg.mozilla.org/build/buildbotcustom/file/default/misc.py#l796

The l10n dep scheduler is setting 'l10n_revision' to 'default'. FWIW, 'revision' is set to the correct value.

Either we need to set 'l10n_revision' to the correct value in the poller or scheduler, or have the l10n dep builds use 'revision'
I think this is now fixed.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Chris, can you point us to a patch which fixed it? So long it would be WFM.
Resolution: FIXED → WORKSFORME
It should have been fixed by https://bug791209.bugzilla.mozilla.org/attachment.cgi?id=664978

specifically, the parts that reference 'l10n_revision'
Thank you Chris. Would be nice if someone from the l10n team could verify that it is fixed.
Depends on: 791209
Resolution: WORKSFORME → FIXED
Whiteboard: [l10n] → [l10n][fixed by bug 791209]
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.