500755 - RelEng tasks to set up an "Electrolysis" project branch

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Benjamin Smedberg]

Questionnaire from https://wiki.mozilla.org/ReleaseEngineering:ProjectBranchPlanning

*do you want builds? Yes
** which o.s.? All
** incremental-build-on-checkin? Yes
** nightlies? Yes
** Are en-US builds enough, with no l10n? Correct, no l10n
* do you want unittests? Yes
* do you want talos? Not yet... I'll file a separate bug for that
*name of branch owner: bsmedberg
*timeline:
** when can we start project branch (any pre-req landings pending needed before
starting project branch out from mozilla-central?)

I'm not sure what this means. The branch already exists at http://hg.mozilla.org/projects/electrolysis and I'm doing the initial push now.

** approx expected life span of project branch - if known? Not known... at least nine months, and probably longer.

*misc:
** need any changes to toolchain used in m-c? Not yet
** need any changes to the compile/link/repack steps used in m-c? Not yet
** preference on tinderbox waterfall name? Electrolysis
** preference on where to put builds on ftp.m.o? not really
** preference on name of project branch in hg? already done

* other information: Because running the browser involves multiple processes, the testing infrastructure may require some changes in the future to make sure that processes get cleaned up. But we don't know exactly what that means yet, so we'll go with the default for now and modify things as appropriate.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(Tweaking summary, as there is more to a project branch then just the builds.)

(In reply to comment #1)
> Questionnaire from
> https://wiki.mozilla.org/ReleaseEngineering:ProjectBranchPlanning
> 
> *do you want builds? Yes
> ** which o.s.? All
All 5 o.s. or just 3 desktop o.s.? Unclear if you meant to request linux-arm and WinCE builds also?

> ** incremental-build-on-checkin? Yes
> ** nightlies? Yes
> ** Are en-US builds enough, with no l10n? Correct, no l10n
> * do you want unittests? Yes
Did you want unittests on all o.s. ?

> * do you want talos? Not yet... I'll file a separate bug for that
If its relatively soon, might as well include it here in this bug. If its months away, then ok a separate bug is fine.

> *name of branch owner: bsmedberg
> *timeline:
> ** when can we start project branch (any pre-req landings pending needed before
> starting project branch out from mozilla-central?)
> 
> I'm not sure what this means. The branch already exists at
> http://hg.mozilla.org/projects/electrolysis and I'm doing the initial push now.
heh, you just answered the question! :-) We've had cases where people dont want project branch started until some major pending fix is landed on m-c first. In this case, sounds like you have it totally covered.


> ** approx expected life span of project branch - if known? Not known... at
> least nine months, and probably longer.
> 
> *misc:
> ** need any changes to toolchain used in m-c? Not yet
> ** need any changes to the compile/link/repack steps used in m-c? Not yet
> ** preference on tinderbox waterfall name? Electrolysis
> ** preference on where to put builds on ftp.m.o? not really
> ** preference on name of project branch in hg? already done
> 
> * other information: Because running the browser involves multiple processes,
> the testing infrastructure may require some changes in the future to make sure
> that processes get cleaned up. But we don't know exactly what that means yet,
> so we'll go with the default for now and modify things as appropriate.
I guess this is about unittests. If I read this correctly, you expect unittests to pass right now, but we may have to rework testware a bit in the future in some not-yet-known way. Sounds fair, lets cross that bridge when we get to it.

Comment 2

[Benjamin Smedberg]

Yes, all desktop and mobile builds.

Yes, unittests on all OSes.

No Talos for now... it's unlikely we'll have anything measurable for months.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

I've done the patches, will test on staging and post when ready.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Running on staging - compile fails across all platforms including mobile.  Results of forced builds can be seen on the tinderbox MozillaTest page.

Comment 5

[Benjamin Smedberg]

Fennec+Windows are green now... Linux still has an odd issue that we're debugging. Expect Mac and WinMo builds to stay red for a while; that shouldn't delay putting this into production.

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Enable electrolysis in staging-master

This has been tested on staging.  From the build perspective it works, though the builds themselves are not all compiling properly at the moment.  Bsmedberg is aware of the issues with the current builds being produced.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Enable electrolysis in production-master

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Nagios monitoring file for electrolysis

This needs to be checked in to be able to file the bug for IT to turn on Nagios monitoring for Electrolysis.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 389031 [details] [diff] [review]
Enable electrolysis in staging-master

d5337aca0a5f

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 389032 [details] [diff] [review]
Enable electrolysis in production-master

1f223951e984

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add debug builds to Electrolysis production

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add debug builds to Electrolysis staging

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Also need graph post machine names added, which will happen in bug 504435 as part of a combined patch.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 389034 [details] [diff] [review]
Nagios monitoring file for electrolysis

Checking in Firefox_electrolysis.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_electrolysis.txt,v  <--  Firefox_electrolysis.txt
initial revision: 1.1
done

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Re-assigning to bhearsum who will get the debug builds turned on next week in the first available downtime.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 390052 [details] [diff] [review]
add debug builds to Electrolysis production

changeset:   1377:cdb592d1f311

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 390053 [details] [diff] [review]
add debug builds to Electrolysis staging

changeset:   1377:cdb592d1f311

Comment 18

[[e-mail] (:bhearsum)]

Results from the first run of the debug builds: OS X doesn't compile, of course, Windows and Linux went green. Linux had a failure first, which was due to an out-of-disk problem. I don't quite understand why yet, but the second Linux build used 1GB less space than the first. I'm going to kick some more debug builds to try and get a better idea of what's happening before marking this fixed.

Comment 19

[[e-mail] (:bhearsum)]

I ran 4 linux builds after the failed - all of them passed, and used the expected amount of disk space. I'm going to write the first one off as a fluke. If it happens again we can bump the disk space requirements for that builder. We're all done here AFAICT.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

We hit another out of disk error on linux debug, on moz2-linux-slave12 for the build starting at Sun Aug 2 18:02:20 2009 (#22 in buildbot). The disk cleanup step asked for 3G and got 3.15G at the start of the run. There is a 722M obj-firefox/_leaktest/sdleak.log, which was only about 5M for other builds. I've uploaded that to http://people.mozilla.org/~nthomas/misc/sdleak.log.bz2 (13M).

Also, of the 22 linux debug's we've done on Electrolysis since it started up, only this latest build and the initial one have failed like this. There are 6 other builds on the same revision that are green prior to this latest fail, all on other slaves. Intermittent code bug ?