561439 - Integrate mobile builders into try-as-branch and turn off mobile in old tryserver

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

Once try-as-branch is turned on we will be leaving the mobile builders on the old try master until the mobile builders can be generated with the build configs and mobile can be turned on in the try-as-branch configs.

This bug will track the integration and subsequent disabling of mobile on the old try master.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

Working against http://hg.mozilla.org/users/catlee_mozilla.com/buildbot-configs/rev/9e2726aa3f7f and the patches in bug 523946.  The mobile on 080 parts of the patch are duplicated from the other patch.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom changes needed to make mobile work on buildbot-0.8.0

Overrides that we need to do in config.py once it has the tryserver branch:
BRANCHES['tryserver']['mobile_tinderbox_tree'] = 'MozillaTry'

Things that need to added to the yet-to-be-landed generateMobileBranchObjects are:

factory_kwargs['try_mode'] = config.get('enable_try', False)
factory_kwargs['try_basefilename'] = 'fennec'

I am entirely unsure if the email in the body of the try message will line up to the actual location these builds are uploaded to.

What this patch does:
Create a 'try_mode' argument that changes the source and upload behaviour.  The factory default mode is False, which goes to regular mode which is 100% untouched other than being indented.

The source steps for try_mode use the Mercurial class to make sure that the revision from the change_source is what is built (the default action is to build tip of whichever branch the repository is on) using clobber mode on the to ensure there are no leftovers.  There are also some magic files checked ('mobile_repo' and 'mobile_rev') that if created in the root directory of a mozilla-central patch contain the path relative to hg.m.o and revision of mobile to build.

The upload steps use the old tryserver steps to upload the mobile builds to where (I hope) will be right beside new tryserver builds.  I have to test this on staging but I'd like to land generateMobileBranchObjects first.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 451739 [details] [diff] [review]
buildbotcustom changes needed to make mobile work on buildbot-0.8.0

sorry, this patch is invalid

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

specifically the following shouldn't be there

in the __init__ kwarg list,

+                 try_slavedir=None, try_slavesrcdir=None,
+                 try_scpstring=None, try_targetsubdir=None,

And in the __init__ body
+            assert try_basefilename
+            assert try_slavedir
+            assert try_slavesrcdir
+            assert try_scpstring
+            assert try_targetsubdir
+            self.try_basefilename = try_basefilename
+            self.try_slavedir = try_slavedir
+            self.try_slavesrcdir = try_slavesrcdir
+            self.try_scpstring = try_scpstring
+            self.try_targetsubdir = try_targetsubdir
+


Otherwise, this patch is valid

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v1

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom v1

These patches are yet to be staged and are WIP.

A couple things:
-forcing a rebuild will not work but because every try build is a clobber, we don't care about this.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 454719 [details] [diff] [review]
buildbot-configs v1

>+BRANCHES['tryserver']['stage_base_path_mobile'] = '/home/ftp/pub/firefox/tryserver-builds'

Looks like this is going to be an area for discussion.
Since we have a second config option for it, it will be easy to send these to mobile/tryserver-builds (my preference), but that will split up the full set of builds... which I'm find with.
Maybe eventually pub/mozilla.org/tryserver-builds/ID with subdirectories firefox/ and fennec/ ?

Not sure I have much else to say on a patch that hasn't even been checkconfiged.  I don't mind the if/else for checkouts/uploads at all.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: try server mozconfigs

This patch adds mozconfig files based on the current nightly mozconfig files.  The main difference between the nightly mozconfigs and these new mozconfigs is that we have 

if [ -f $topsrcdir/mozconfig-extra ] ; then
  . $topsrcdir/mozconfig-extra
fi
if [ -f $topsrcdir/mozconfig-extra-PLATFORM ] ; then
  . $topsrcdir/mozconfig-extra-PLATFORM
fi

where PLATFORM is one of maemo4, maemo5-gtk, maemo5-qt, android-r7, linux, macosx, win32.

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: mobile try mozconfigs

Uploading a new patch and attaching a script to recreate the patch
cd buildbot-configs/mozilla2/
cp -r mobile mobile-tryserver
cd mobile-tryserver
find -type d -name 'mobile-1.1' -exec rm -rf {} \;
find -type d -name 'mobile-e10s' -exec rm -rf {} \;
find -type d -name 'release' -exec rm -rf {} \;
for i in * ; do
  mv $i/mobile-browser/nightly/mozconfig $i/
done
find -type d -name 'mobile-browser' -exec rm -rf {} \;
for i in * ; do
  cat >> $i/mozconfig << EOF

if [ -f \$topsrcdir/mozconfig-extra ] ; then
  . \$topsrcdir/mozconfig-extra
fi
if [ -f \$topsrcdir/mozconfig-extra-$i ] ; then
  . \$topsrcdir/mozconfig-extra-$i
fi
EOF
done
find -type f -exec hg add {} \;
hg diff > ../../new-confs.diff
cd ../../mozilla2-staging/
patch -p2 < ../new-confs.diff
find mobile-tryserver -type f -exec hg add {} \;

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

verified that this does not merge pushes:
(Jun 29 20:34:14, waiting 10 mins, 26 secs)
[e-mail]
(Jun 29 20:34:14, waiting 10 mins, 26 secs)
[e-mail]

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v2

need to verify that the factory changes to android don't break normal android builds.  I am guessing that I am going to have to add the objdir to the package_glob.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom v2

This patch adds a 'try mode' to the MobileBuildFactory that understands how a try build needs to be different to a normal mobile build.  I have disabled the duplicate addUploadSteps command in AndroidBuildFactory as it is functionally equivalent to the one on MobileBuildFactory if we make a change to the packagegloblist (i think).  I have to test a non-try build in staging before I ask for review on this patch.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v3

Asking for review from Lukas for the try-serverness and Bear for the change to the packageGlobList

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 455579 [details] [diff] [review]
buildbot-configs v3

sorry, just noticed something wrong with the corresponding buildbotcustom patch.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs-v4

Instead of removing the duplicate upload function, I am going to override only try behaviour as I cannot seem to get a working slave and m-c/m-b revision to test modified upload for normal builds.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom-v4

Comment 17

[Aki Sasaki (not active)]

Comment on attachment 455717 [details] [diff] [review]
buildbot-configs-v4

>+BRANCHES['tryserver']['mobile_platforms']['maemo5-gtk']['mozconfig'] = 'mobile-tryserver/maemo5-gtk/'

Extremely minor nit: we had a little unhappiness with trailing/non-trailing slashes already, so let's just keep it uniform. r=me with this trailing slash removed to match the others.

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom-v4 (with showfunc=True)

(identical to buildbotcustom-v4 with the showfunc flag on in my hgrc)

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 455719 [details] [diff] [review]
buildbotcustom-v4

the try parts look good to me: setting who, got_revision, and uploading.

Comment 20

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 455717 [details] [diff] [review]
buildbot-configs-v4


>+#BRANCHES['tryserver']['mobile_platforms']['linux']['slaves'] = TRY_SLAVES['linux']
>+#BRANCHES['tryserver']['mobile_platforms']['win32']['slaves'] = TRY_SLAVES['win32']
>+#BRANCHES['tryserver']['mobile_platforms']['macosx']['slaves'] = TRY_SLAVES['macosx']
>+

I would take these out or put in a comment about why they are there but to just have commented out lines in the configs with no explanation seems unnecessary IMO. r+ with that addressed.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 455719 [details] [diff] [review]
buildbotcustom-v4

I don't see anything glaring.  Not really motivated to nit indenting your function args to line up with the '(', but if you do that I won't be unhappy.

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: turn symbols generation off for mobile builds

turn off symbol generation/uploading on try

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 455719 [details] [diff] [review]
buildbotcustom-v4

http://hg.mozilla.org/build/buildbotcustom/rev/b634f09d2940

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 455717 [details] [diff] [review]
buildbot-configs-v4

http://hg.mozilla.org/build/buildbot-configs/rev/97f7093ab99f

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 455792 [details] [diff] [review]
turn symbols generation off for mobile builds

We may want this at some point, but we can deal with getting that working then.

Comment 26

[John Ford [:jhford] CET/CEST Berlin Time]

in production and working :)