387958 - Deploy buildbot patch uploader

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Tracking bug for the deployment of the Buildbot patch uploader.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: cleaned up master.cfg for patch uploader

This is a cleaned up version of the master.cfg from test1-linux-buildbot. It has builders for Linux, Mac, and Windows. It's valid according to checkconfig.py.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: mozbuild.py needed by master.cfg

This is the mozbuild.py for master.cfg. The win32-buildbotref-v4 environment _should_ be compatible with the current Win32 Buildbot ref platform.

Comment 3

[Robert Helmer [:rhelmer]]

Looks good, let's get these checked in. I suggest:
mozilla/tools/buildbot-configs/tryserver/

Any alternative suggestions?

Comment 4

[Robert Helmer [:rhelmer]]

Attachment: patch as landed

Comment 5

[Robert Helmer [:rhelmer]]

RCS file: /cvsroot/mozilla/tools/buildbot-configs/tryserver/master.cfg,v
done
Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/master.cfg,v  <--  master.cfg
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbot-configs/tryserver/mozbuild.py,v
done
Checking in mozbuild.py;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/mozbuild.py,v  <--  mozbuild.py
initial revision: 1.1
done

Comment 6

[[e-mail] (:bhearsum)]

The buildmaster is setup on staging-build-console.

Comment 7

[[e-mail] (:bhearsum)]

Some points that came up on IRC:
* How are people going to get results from their submission? Reporting to a Tinderbox or looking at the Waterfall page directly?
* How will access be secured? LDAP or despot?

Comment 8

[Benjamin Smedberg]

1) the builds should be given some sort of unique ID and then made available for a 24-hour window on some download site

2) Ask IT (reed and I were talking about it briefly)

Comment 9

[[e-mail] (:bhearsum)]

(In reply to comment #8)
> 1) the builds should be given some sort of unique ID and then made available
> for a 24-hour window on some download site
> 

You don't think having a Waterfall or Tinderbox display will be useful? I don't think everyone will be downloading builds all the time -- they'll just want to know "does my code burn the tree" or not.

Comment 10

[Benjamin Smedberg]

Oh, I kinda figured people would be able to see the buildbot results somewhere, but maybe I shouldn't have made that assumption. So yes, the results should be made available (both build and test results, when we get around to the testing part).

Comment 11

[[e-mail] (:bhearsum)]

As per discussion with joduinn this will be setup with the existing win32 ref platform for now.

Comment 12

[[e-mail] (:bhearsum)]

Just to be clear: The win32 buildslave will be re-done once the new windows ref platform is available.

Comment 13

[[e-mail] (:bhearsum)]

I'm pushing the ETA back to Friday. Reed says he'll have LDAP accounts for CVS commiters by then.
TODO list:
* Have the Buildmaster push completed builds back to dm-wwwbuild01
* Authenticate users via LDAP
* A bit more testing

Comment 14

[[e-mail] (:bhearsum)]

OK. Here's what I've done the past couple days:
* Push completed builds (in packages) back to build.m.o
* Have the Buildmaster report to the Tinderbox on MozillaTest
* Write a BuildStep that ensures only one change per Build is processed
* Setup cronjobs on staging-build-console and build.m.o that cleanup patches/binaries older than a week
* Run downloadpatches.sh in a cronjob every minute
* Add a short note at the top of Tinderbox log so people can easily identify what builds are theirs.

Still to do:
* Test with many builds queued.
* Attach final copies of master.cfg and mozbuild.py to this bug after testing

Comment 15

[[e-mail] (:bhearsum)]

Attachment: final version of configs

This is an update to try servers master.cfg and mozbuild.py. Barring any problems these are the configs it will use.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: minor tweaks to the last patch

These are the files that are in use on the tryserver. Can you land these after review?

Comment 17

[[e-mail] (:bhearsum)]

For everyone's information: I moved the master to build-console.b.m.o. I'll be using staging-build-console.b.m.o as a staging area for it.

Comment 18

[Reed Loden [:reed]]

Noticed this while I was fixing an issue on dm-wwwbuild01:
[error] uploadpatch.cgi: Bareword found in conditional at /var/www/html/build/uploadpatch.cgi line 181., referer: https://build.mozilla.org/uploadpatch.cgi

Comment 19

[[e-mail] (:bhearsum)]

Found the culprit: print PATCH || print("could not upload patch: $!"); The $! is causing the trouble.

I'm not sure who put the '|| print ...' in, it's not part of my code.

Comment 20

[[e-mail] (:bhearsum)]

Attachment: try configs, with mac UB support.

This version of the configs has support for Mac universal binaries. It should have all the tweaks from your patch too, rhelmer.

I'm going to attach the mozconfig files in a minute.

Comment 21

[[e-mail] (:bhearsum)]

Attachment: tryserver linux mozconfig

Comment 22

[[e-mail] (:bhearsum)]

Attachment: tryserver mac mozconfig

Comment 23

[[e-mail] (:bhearsum)]

Attachment: tryserver win32 mozconfig

Comment 24

[Robert Helmer [:rhelmer]]

Comment on attachment 275985 [details]
tryserver mac mozconfig

why does mac have:
mk_add_options MOZ_PACKAGE_NSIS=1

?

Comment 25

[[e-mail] (:bhearsum)]

Attachment: tryserver mac mozconfig, fixed

I must've got mixed up when I was making these.

Comment 26

[[e-mail] (:bhearsum)]

Attachment: tryserver configs, update + all in one

These are the configs that have been running on the try server for about 2 weeks. I'm confident that they are OK.

Sorry to bog you down with another review on this, Rob. Once this is checked in I think we can consider this FIXED.

Comment 27

[Robert Helmer [:rhelmer]]

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/master.cfg,v  <--  master.cfg
new revision: 1.2; previous revision: 1.1
done
Checking in mozbuild.py;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/mozbuild.py,v  <--  mozbuild.py
new revision: 1.2; previous revision: 1.1
done