698910 - Generate "l10n repack" status reports

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

When we tried to disable tinderbox mail everywhere we broke the only public reporting tool for localization [1]. This is for both L10n nightly builds and L10n dep builds.

I would like to work with you guys to create a web site that can allow us to know:
* for a branch
* for every platform we support
* for every locale

A proof of concept for that is http://l10n.mozilla.org/~axel/nightlies but it is based on tinderbox.

Currently the same data can be retrieved from our json dumps [2]

I want to answer these questions:
- Did an L10n nightly get generated?
- Did the job succeed/fail?
- Where is the log?
- Where is the nightly build?

Particularly I need:
1) where to host this dashboard
2) how to parse the logs (I really don't know how parsers work)
   - tbpl uses [3]
   - tinderbox uses [4]
3) Answer the questions listed above
   - on tbox we have a "L" link for the log
   - "D" log to point to the builds
   - green/red colours to indicate status

To see the nightly builds I imagine something like this:
* www.L10n-builds?type=nightly&date=2011-10-31
To see the dep builds I can imagine something like this:
* www.L10n-builds?type=dep&locale=ab-CD
** we would have to deal with the lack of commits

If we look at [5] we can see that I can load up a .js file and filter L10n nightly and dep jobs. Then I can get information from the "properties" key and generate all of the information I need.

I can write an script that can extract all the info we need but I don't know anything about what is the right way to do 2) and/or proper practices to create a web page out of that info. If you help me with 1) and 2) I can help with 3) (if you give me some guidelines on how to web-ize it).


[1] http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n
[2] http://build.mozilla.org/builds/builds-2011-10-31.js.gz
[3] https://tbpl.mozilla.org/php/getParsedLog.php?id=7147494&tree=Firefox&full=1
[4] http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n/1320176137.1320176826.19693.gz&fulltext=1
[5]
j = json.load(open('builds-2011-10-30.js'));
for item in j["builds"]:
    if item["properties"].get("buildername", "").endswith("l10n nightly")
        # we have an L10n nightly job
    if item["properties"].get("buildername", "").endswith("l10n nightly")
        # we have an L10n dep job
[6] L10n nightly log and download:
* http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/mozilla-central-macosx64-l10n-nightly-ka-build6978.txt.gz
* http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/firefox-10.0a1.pl.linux-x86_64.tar.bz2
[7] L10n *dep* log and download:
* http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-l10n/mozilla-central-linux-l10n-dep-ar-build202.txt.gz
* http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-l10n/firefox-10.0a1.zh-TW.linux-i686.tar.bz2

Comment 1

[Axel Hecht [:Pike]]

Should this block a bug to switch off tinderbox mails for l10n builds?

Comment 2

[Armen [:armenzg]]

(In reply to Axel Hecht [:Pike] from comment #1)
> Should this block a bug to switch off tinderbox mails for l10n builds?

I filed bug 700290 to keep track of it.

Comment 3

[Armen [:armenzg]]

Laura, any chance this could be picked up on Q1?

Comment 4

[Armen [:armenzg]]

ping

Comment 5

[Armen [:armenzg]]

Is this something that could be picked up as Q2?

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting with Axel;

Axel really cares about http://l10n.mozilla.org/~axel/nightlies only when debugging problems with the nightly l10n repacks. 

If RelEng generates an email report, after all l10n nightly repacks are run, he'll be happy to stop using http://l10n.mozilla.org/~axel/nightlies, and hence remove l10n dependency on tinderbox.m.o. 

This report would be generated using data that we already have in https://secure.pub.build.mozilla.org/builddata/buildjson/builds-2013-03-04.js.gz. The exact format of the email report doesnt have to match the experimental page at http://l10n.mozilla.org/~axel/nightlies, so long as it can solve the need - being able to see if there is a locale-specific problem or an infrastructure-specific-problem causing problems for multiple locales. Please loop in Axel on early drafts for vetting.




There's several bugs around this same topic, so morphing the closest-similar to these requirements.

(In reply to Armen Zambrano G. [:armenzg] from comment #0)
...
> A proof of concept for that is http://l10n.mozilla.org/~axel/nightlies but
> it is based on tinderbox.
> 
> Currently the same data can be retrieved from our json dumps [2]
> 
> I want to answer these questions:
> - Did an L10n nightly get generated?
> - Did the job succeed/fail?
> - Where is the log?
> - Where is the nightly build?
...

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

also, there should be one per train:

mozilla-central (fewer locales, subset of locales from aurora, but still of interest)
mozilla-aurora (all locales)
mozilla-beta (subset of locales from aurora)

Comment 9

[Axel Hecht [:Pike]]

To quote the notes from the meeting:

1) L10N: axel + joduinn just met. Summary is:
1a) that RelEng will setup new service (to be designed) to display the
following information:
* l10n build status (red/yellow/green)
* link to build log on ftp.m.o
* link to downloadable binary on ftp.m.o
* nice to have: good to have changesets.
Once this is up, l10n no longer needs any l10n pages on tinderbox.m.o like:
http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n
http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n-km&hours=24
...as they are used mostly by Axel/Stas to help localiser community
debug if there is a problem with locale repack vs problem with
infrastructure.
1b) As far as axel/joduinn know, all other l10n
dashboards/infrastructure for localizer signoffs, release signoff, etc:
https://l10n.mozilla.org/shipping/dashboard?locale=km
https://l10n.mozilla.org/shipping/signoffs/km/fx20#01d2ad9564a8
https://l10n.mozilla.org/shipping/milestones
...do not rely on tinderbox.m.o. If any surprise bustages happen, it
will not block decommission of tinderbox.m.o, and can be handled after.
1c) If setting up this replacement is too slow for OpSec, it might be
possible for l10n group to have the existing tinderbox pages moved
behind VPN in the short term while the above replacement is built. This
is disruptive to l10n work, so is only listed for completeness.


Nothing here says that I get emails, and I strongly object to that.

Comment 11

[Chris AtLee [:catlee]]

Axel, would publishing a json file similar to how we do http://builddata.pub.build.mozilla.org/buildjson/ work for you?

Comment 12

[Axel Hecht [:Pike]]

I don't really know why this bug got reduced to nightlies, fyi.

There needs to be public reporting on both nightlies and depend builds, and the l10n team is neither on the hook nor resourced to design or implement that. At the end of the day, if someone files a bug on missing builds or other problems, releng needs to be able to answer questions on what happens on their infrastructure, and thus I see releng as the first level of customers of those reports.

Comment 13

[Chris AtLee [:catlee]]

It doesn't have to be specific to nightlies.

It sounds like you're not currently relying on tinderbox for any kind of automation, so what is the purpose of these reports? RelEng already has ways of debugging build failures, so we wouldn't be using these reports. If we wouldn't use them, and you wouldn't use them, then who are they for?

Comment 14

[Axel Hecht [:Pike]]

Sorry, but I don't have time to have this discussion again and again. If the outcome of my discussion with joduinn doesn't hold, then that's that.

Comment 15

[Chris AtLee [:catlee]]

Well, I'm trying to understand your requirements here. You imply in comment #12 that you wouldn't actually make use of this data.

If you do want this data published, would something along the lines of the json we publish to http://builddata.pub.build.mozilla.org/buildjson/ work for you?

Comment 16

[Chris Cooper [:coop] (he/him)]

(In reply to Axel Hecht [:Pike] from comment #14)
> Sorry, but I don't have time to have this discussion again and again. If the
> outcome of my discussion with joduinn doesn't hold, then that's that.

joduinn is on vacation this week, so we're trying to debug the requirements here without his input. We're not trying to be deliberately obtuse, we just don't have access to his side of the discussion right now.

Comment 17

[Axel Hecht [:Pike]]

(In reply to Axel Hecht [:Pike] from comment #9)
> To quote the notes from the meeting:
> 
> 1) L10N: axel + joduinn just met. Summary is:
> 1a) that RelEng will setup new service (to be designed) to display the
> following information:
....

"service" here means a setup that displays human readable information. John formulated that loosely to not tie that down to a particular implementation, regardless if that'd be a Firefox addon talking to a webservice for data, or a data driven website.

Point being that the l10n team doesn't have to code anything.

Comment 18

[Chris AtLee [:catlee]]

Attachment: l10n_report script

I'm not really sure what our requirements are, so I'm happy to change the output format as required. This is very human readable (to me at least!)

Comment 19

[Chris AtLee [:catlee]]

Attachment: deploy l10n report as part of buildapi

Comment 20

[Chris AtLee [:catlee]]

Attachment: sample report

Comment 21

[Chris AtLee [:catlee]]

report is available internally for now at http://buildapi01.build.scl1.mozilla.com/l10n_reports/latest.txt

once 855733 is fixed it will be public at http://builddata.pub.build.mozilla.org/l10n_reports/latest.txt

Comment 22

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Summary of mtg w/pike and joduinn on 21mar2013:

The disconnect here was because we were each solving two overlapping-yet-different problems. Axel was solving the "how to keep track of what l10n repacks are being generated/infra problems/etc". John was solving the "how to get off tinderbox server". We agreed to the following, which we think will solve both problems, and also quickly address needs of both groups:

1) RelEng or sheriff will check for any infra/bustage problems with l10n nightlies. 
1a) If there was an infra-related problem, RelEng or sheriff will file RelEng bug to fix and retrigger. 
1b) If there is a locale-specific problem, RelEng or sheriff will copy/paste error into l10n bug, and let Axel/Stas coordinate with localizers to fix.
1c) We discussed sending email/notification if things were green, but agreed these would quickly be filtered into spam, so not useful. Instead, we'll be silent if all is well, and file bugs for any problems.

2) Typically, Axel first hears of problems by word from localizer community. Historically, Axel has first looked at the tinderbox server page to see if this was locale-specific  or across-locales, in order to be able to file better quality bugs. 

Instead, going forward, now if Axel hears any reports of locale bustages from community, Axel will directly file bug in RelEng, or encourage localizers to do same. RelEng will investigate and resolve or redirect as needed. 


With these process changes, l10n no longer has a dependency on tinderbox.m.o server.

Comment 23

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Chris AtLee [:catlee] from comment #21)
> report is available internally for now at
> http://buildapi01.build.scl1.mozilla.com/l10n_reports/latest.txt
> 
> once 855733 is fixed it will be public at
> http://builddata.pub.build.mozilla.org/l10n_reports/latest.txt

Cool, thanks catlee!

Comment 24

[Chris AtLee [:catlee]]

This is now available at http://builddata.pub.build.mozilla.org/buildjson/l10n_reports/latest.txt