744619 - multi-result reporting solution for mozharness

Status: RESOLVED DUPLICATE of bug 747457

(Release Engineering :: General, defect, P3)

Description

[Aki Sasaki (not active)]

There are a number of places where we have multiple results to report, and no way or place to report them to.

We could:

1) Create separate tinderbox emails for each result.  This is fairly easily done, and already has the back end written and set up.  However, we have made a conscious effort to retire tinderbox usage, so creating new dependencies on tinderbox would not be ideal.

2) Add some way to inject status into http://build.mozilla.org/builds/builds-4hr.js.gz , which tbpl looks at.  Aiui, this currently assumes a 1-1 ratio of buildbot job and status.  We need to be able to have a 1-many ratio at the very least; a many-to-many may help us in abstracting results from how we split up jobs.

3) Something involving pulse or other message queues, and a lot of uninformed hand-waving on my part.

4) Something involving some post-job log file parsing ?

My preference is for 2 or a fleshed out, informed 3.  4 seems like a hacky approach but has the fewest dependencies on RelEng.  1 would be the fastest solution but seems like a step backwards.

Comment 1

[Aki Sasaki (not active)]

Concrete examples in case this bug is too vague:

Repack N number of locales in a single job.  Report repack status per locale.
(de is red but es-ES is green: should be able to see that easily on a dashboard.)

Run N number of test suites in a single job.  Report test status per test suite.
(Mochitest-1 orange but browser-chrome green.)

Build en-US and multilocale in the same job.  Report status per locale.
(en-US green, multilocale red.)

Comment 2

[Armen [:armenzg]]

We also need to upload the individual logs individually, no?
Should each individual log have the common part be part of the individual log? (en-US checkout, make unpack et all).

Comment 3

[Aki Sasaki (not active)]

I think uploading a single log, and pointing to it from the individually reported results, would be a fine first pass, as long as the log is readable and not overly huge.

I think if we get upload-from-testers functionality I'd upload an entire directory of logs, and you could look at the shared steps in another log if you wish.  But if certain steps failed, like checking out m-c, all of the results would be broken, not just one.  I think the locale-specific stuff in the individual logs, with the ability to easily find and read the full log, would hopefully be sufficient.

Comment 4

[Aki Sasaki (not active)]

I think I resolved this in bug 747457.