597087 - Setup project branch for devtools repo

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P3)

Description

[Rob Campbell [:rc] (:robcee)]

We'd like some set of unittests and talos runs for this repo. Not sure if
we need the full suite or not, but definitely mochitest-browser-chrome in opt
and debug configurations

Comment 1

[Rob Campbell [:rc] (:robcee)]

bug 597008 is now closed and we have a populated repository at hg.mozilla.org/devtools. We're ready for some unittests!

Comment 2

[Chris AtLee [:catlee]]

Can you tell us how to run these?  What do we need to checkout / download / unpack, and where?  What commands then need to be run?  How are results reported?

Comment 3

[Rob Campbell [:rc] (:robcee)]

these are the same mochitests and browser-chrome tests you're running on mozilla-central. In fact, this is mozilla-central + a few changesets. It is a straight copy.

If time is an issue, we can probably get away with just mochitest-browser-chrome for starters. Then full mochitest+xpc_shell followed by talos.

checkout and build is the same, but you'll be pulling from hg.mozilla.org/devtools instead of /mozilla-central.

Comment 4

[Chris AtLee [:catlee]]

Do we need to do a whole new build for this?  Is what you really want a new project branch?

Comment 5

[Rob Campbell [:rc] (:robcee)]

Yes.

Comment 6

[Chris AtLee [:catlee]]

Then please fill out our handy dandy project branch questionnaire!

https://wiki.mozilla.org/ReleaseEngineering:ProjectBranchPlanning

What's the timeline for needing this up and running?

Comment 7

[Rob Campbell [:rc] (:robcee)]

* Name of hg repo for this branch (relative to hg.mozilla.com)

/devtools

** '''NOTE''': this name will be used for all things related to the branch, tinderbox, tbpl, ftp locations, etc.
** Has a bug already been filed to create the repo?
** Name of parent branch?

tip

* Do you want builds?

yes.

** All o.s. or subset of linux32, linux64, osx10.5, osx10.6, win32, win64

All, but only one flavor of OS X (10.6). Win7 only.

(except mobile, e.g., maemo, android, ... any others?)

** Incremental-build-on-checkin? y/N

yes

** Nightly builds? Y/n

yes

*** Nightly updates? y/N

no

* Do you want unittests?

yes

** All o.s. or subset of: WinXP, Win7, 10.5, 10.6, fedora12, fedora12-64, 

all (except as above, only 10.6, win7, no mobile)

* Do you need mobile builds?

no 

(note. maybe move this section up to the "Do you want builds" part to prevent people from repeating themselves in the future)

* Do you want talos?

yes

** Do you need talos on Tiger?

Tiger? The OS X release? No, Snow Leopard should be fine.

** All o.s. or subset of Talos platforms?

all (except as above, only 10.6, win7, no mobile)

* Name of the contact person for this branch who will:
** Be doing periodic refreshes from parent

Rob Campbell

** Be contact person for misc setup questions

Rob Campbell

** Decide when to land back project branch onto parent

Rob Campbell

** Decide when to terminate the project branch

Rob Campbell

* Timeline:
** Is this branch is needed for a specific hard calendar deadline? (we typically can do this in 2 weeks, and we will notify you if we think we'll miss that)

We need this for Q4 work, so sooner is better. If 2 weeks is how long it should take, that should be fine, plus any additional padding you require to clear current work.

** For changes in schedule, or status checks, please ping in bug.
** Approx expected life span of project branch - if known?

Ongoing.

* Additional requests:
** By default project branches do not have l10n:

no l10n.

** Need any changes to toolchain used in parent branch?

no

** Need any changes to the compile/link/repack steps used in parent branch?

no

** Do you expect to create any releases from this project branch?

no, builds are for test-only.

*** '''NOTE''': Currently we do not have nightly/alpha/beta/release updates on these project branches.
** Any other info that might be helpful to us?

This is mozilla-central + some set of front-end patches for the browser. Other than repo, it should be a straight copy.

Comment 8

[Rob Campbell [:rc] (:robcee)]

Ping. This is going to be needed soon.

Comment 9

[Rob Campbell [:rc] (:robcee)]

Re-ping! Can I get an ETA on test infrastructure?

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

I'll put together configs for review - aiming for going live by Friday

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add devtools branch to buildbot-configs

So a couple of changes had to be made in order to
a) take macosx out of the possible slave_platforms on talos 
b) turn off mobile and enable nightlies for this branch (not previously used on any other project branch yet)

I realize it's a bit hacky, but I also hope that some parts will be useful in bug 633054

Also, a Tinderbox page has been created for this branch in Devtools

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: data.sql update for devtools branch

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 519324 [details] [diff] [review]
add devtools branch to buildbot-configs

>+# Copy project branches into BRANCHES keys
>+for key, value in PROJECT_BRANCHES.items():
>+    BRANCHES[key] = value
>+    
>+# Look for project branch overrides of slave platforms
>+for branch, branch_config in PROJECT_BRANCHES.items():
>+    if branch_config.has_key('platforms'):
>+        for platform in branch_config['platforms']:
>+            if branch_config['platforms'][platform].has_key('slave_platforms') and '-debug' not in platform:
>+                PLATFORMS[platform]['slave_platforms'] = branch_config['platforms'][platform]['slave_platforms']
>+                

I think it would be safer to copy the list of slave_platforms here rather than just assigning the same list.

>diff --git a/mozilla/config.py b/mozilla/config.py
>--- a/mozilla/config.py
>+++ b/mozilla/config.py
>@@ -729,16 +729,20 @@
> for key, value in PROJECT_BRANCHES.items():
>     BRANCHES[key] = value
> 
> # Copy global vars in first, then platform vars
> for branch in BRANCHES.keys():
>     for key, value in GLOBAL_VARS.items():
>         # Don't override platforms if it's set
>         if key == 'platforms' and 'platforms' in BRANCHES[branch]:
>+            # Remove any talos-only platforms from the 'platforms'
>+            for platform in BRANCHES[branch]['platforms'].keys():
>+                if platform not in PLATFORM_VARS.keys():
>+                    del BRANCHES[branch]['platforms'][platform]
>             continue
>         elif key == 'mobile_platforms' and 'mobile_platforms' in BRANCHES[branch]:
>             continue
>         BRANCHES[branch][key] = deepcopy(value)

What's this chunk for?

Can you do a pass through dump_master to make sure you're only affecting the new devtools branch?

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add devtools branch to buildbot-configs as well as more handling for customization of project_branches

added deepcopying as per comments, also added an is_test check on platforms which are to be used in talos only.  also added slave_platforms[] customization (buildbotcustom patch accompanies this) in order to disable the OS 10.5 tests

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: adds checking for overriden slave_platforms in branch-level configs to generateTalosBranchObjects

this will check if a branch_config has ['slave_platforms'] set first before grabbing the ACTIVE_PLATFORMS ones so that we can customize which slave_platforms we want at the branch level, namely in project_branches.

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

OK so I ran dump_masters on an instance of a builder master, build-scheduler, test-scheduler, and test master -- all of them are only affected by having new devtools builders in them. Here's the example of the build-scheduler:

--- dump_master	2011-03-21 15:46:29.437419035 -0700
+++ dump_master_devtools	2011-03-21 15:46:14.663831380 -0700
@@ -21,6 +21,7 @@
 <buildbotcustom.changes.hgpoller.HgPoller object> 'http://hg.mozilla.org/services/services-central/json-pushes?full=1' {'branch': 'services/services-central',
  'hgURL': 'http://hg.mozilla.org/',
  'pollInterval': 60}
+<buildbotcustom.changes.hgpoller.HgPoller object> 'http://hg.mozilla.org/devtools/json-pushes?full=1' {'branch': 'devtools', 'hgURL': 'http://hg.mozilla.org/', 'pollInterval': 60}
 <buildbotcustom.changes.hgpoller.HgPoller object> 'http://hg.mozilla.org/projects/maple/json-pushes?full=1' {'branch': 'projects/maple',
  'hgURL': 'http://hg.mozilla.org/',
  'pollInterval': 60}
@@ -447,6 +448,51 @@
  'name': 'mobile-browser-services-central-build-mobile',
  'properties': {'scheduler': 'mobile-browser-services-central-build-mobile'},
  'treeStableTimer': 180}
+<buildbotcustom.scheduler.Scheduler-props> {'builderNames': ['Linux QT devtools build',
+                  'OS X 10.5.2 devtools build',
+                  'OS X 10.6.2 devtools build',
+                  'WINNT 5.2 devtools build',
+                  'Linux x86-64 devtools build',
+                  'Linux devtools build',
+                  'WINNT 5.2 devtools leak test build',
+                  'OS X 10.6.2 devtools leak test build',
+                  'Linux x86-64 devtools leak test build',
+                  'Linux devtools leak test build'],
+ 'change_filter': <ChangeFilter on branch == devtools>,
+ 'fileIsImportant': <function <lambda>>,
+ 'name': 'devtools',
+ 'properties': {'scheduler': 'devtools'},
+ 'propfuncs': [<function buildIDSchedFunc>,
+               <function buildUIDSchedFunc>],
+ 'treeStableTimer': 180}
+<buildbotcustom.scheduler.SpecificNightly-props> {'branch': 'devtools',
+ 'builderNames': ['OS X 10.5.2 devtools nightly',
+                  'OS X 10.6.2 devtools nightly',
+                  'WINNT 5.2 devtools nightly',
+                  'Linux x86-64 devtools nightly',
+                  'Linux devtools nightly'],
+ 'dayOfMonth': '*',
+ 'dayOfWeek': '*',
+ 'fileIsImportant': None,
+ 'hour': [4],
+ 'minute': [2],
+ 'month': '*',
+ 'name': 'devtools nightly',
+ 'onlyIfChanged': False,
+ 'properties': {'scheduler': 'devtools nightly'},
+ 'propfuncs': [<function buildIDSchedFunc>,
+               <function buildUIDSchedFunc>]}
+<buildbot.schedulers.timed.Nightly> {'branch': 'devtools',
+ 'builderNames': [],
+ 'dayOfMonth': '*',
+ 'dayOfWeek': 5,
+ 'fileIsImportant': None,
+ 'hour': [3],
+ 'minute': [2],
+ 'month': '*',
+ 'name': 'weekly-devtools',
+ 'onlyIfChanged': False,
+ 'properties': {'scheduler': 'weekly-devtools'}}
 <buildbotcustom.scheduler.Scheduler-props> {'builderNames': ['Linux QT maple build',
                   'OS X 10.6.2 maple build',
                   'WINNT 5.2 maple build',

Comment 17

[Lukas Blakk [:lsblakk] use ?needinfo]

This branch is now live in production.  Please start using it and reopen this bug if any issues arise regarding its setup.

Comment 18

[David Dahl :ddahl]

(In reply to comment #17)
> This branch is now live in production.  Please start using it and reopen this
> bug if any issues arise regarding its setup.

Hawt! Thank you.

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Looking into burning on the linux opt * and the mac10.5 nightly SDK issue.

Comment 20

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: fix devtools by adding to list of gcc bug fix branches, and removing uneeded macosx 10.5

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: fixes devtools (and any branch with overriden platforms) to not need to be in the PLATFORMS list

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 521025 [details] [diff] [review]
fixes devtools (and any branch with overriden platforms) to not need to be in the PLATFORMS list

landed on default http://hg.mozilla.org/build/buildbotcustom/rev/076e569f00bb

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 521024 [details] [diff] [review]
fix devtools by adding to list of gcc bug fix branches, and removing uneeded macosx 10.5

landed on default http://hg.mozilla.org/build/buildbot-configs/rev/341cc4cfabb5

Comment 24

[Lukas Blakk [:lsblakk] use ?needinfo]

FYI - because of release mechanics right now this isn't landing today but as soon as it does the burning on linuc & macosx 10.5 will go away.  Know that this is in our deployment queue and will go out on the next reconfig which should happen in the next day or two.

Comment 25

[Rob Campbell [:rc] (:robcee)]

sounds good. Thanks for your help!

Comment 26

[Rob Campbell [:rc] (:robcee)]

any estimate when we can land this? We have patches we'd like to land on this branch now and can't do so until we get some greenery.

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

the reconfig is happening as i write this, so within the next 20 minutes all production masters should have these updates.

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

looks like we're good to go now - closing again.

Comment 29

[Rob Campbell [:rc] (:robcee)]

yes!