Last Comment Bug 469290 - Provide updates for nightly and release builds of Fennec on linux mobile.
: Provide updates for nightly and release builds of Fennec on linux mobile.
: mobile
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: P2 normal (vote)
: ---
Assigned To: Aki Sasaki [:aki]
: 475799 519748 527079 (view as bug list)
Depends on: 430200 527076
  Show dependency treegraph
Reported: 2008-12-12 02:02 PST by John O'Duinn [:joduinn] (please use "needinfo?" flag)
Modified: 2013-08-12 21:54 PDT (History)
9 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

wip patch (3.13 KB, patch)
2009-11-09 12:06 PST, Aki Sasaki [:aki]
no flags Details | Diff | Review
wip buildbot configs portion (10.06 KB, patch)
2009-11-09 12:08 PST, Aki Sasaki [:aki]
no flags Details | Diff | Review does most of the heavy lifting. (5.76 KB, patch)
2009-11-10 13:57 PST, Aki Sasaki [:aki]
nthomas: review+
Details | Diff | Review
debsign buildbot-configs (6.90 KB, patch)
2009-11-10 14:02 PST, Aki Sasaki [:aki]
nthomas: review-
Details | Diff | Review update (5.86 KB, patch)
2009-11-18 10:08 PST, Aki Sasaki [:aki]
nthomas: review+
aki: checked‑in+
Details | Diff | Review
debsign buildbot-configs (16.57 KB, patch)
2009-11-18 10:16 PST, Aki Sasaki [:aki]
nthomas: review+
aki: checked‑in+
Details | Diff | Review

Description John O'Duinn [:joduinn] (please use "needinfo?" flag) 2008-12-12 02:02:09 PST
It is important for Fennec nightly users to automatically get updates to the latest nightly build... just like we currently do for Firefox nightly users.

Some factors we know:
1) The builds need to be produced in deb format. This is already done.
2) There is no need to use AUS. The updating mechanism, and the download location logic, is provided within the deb packaging framework used by Fennec.

Some factors we dont know:
1) There's mobile signing logic which Stuart has in some scripts that we need to figure out, review and get integrated into the automation. There was also some discussion about specific text files, containing SHA1 checksums and GPG signatures. Format of these text files unclear.
2) Confirm that the location of builds on ftp.m.o, and the filenames used, are suitable for deb update mechanisms. For example, the xulrunner filenames need  buildid in the filename so that the deb packaging dependency checking would work correctly. In discussions today with Christian/Brad/Stuart, it seems like we are already doing all the right things. However, we need to verify if everything is ok?

What else do we need to figure out before we can start here?
Comment 1 Joel Maher (:jmaher) 2009-08-11 06:38:38 PDT
*** Bug 475799 has been marked as a duplicate of this bug. ***
Comment 2 Joel Maher (:jmaher) 2009-09-17 09:31:37 PDT
can we get this fixed sooner rather than later?  With the need for testing multiple branches every day, this is a critical piece.
Comment 3 Aki Sasaki [:aki] 2009-10-29 13:42:38 PDT
*** Bug 519748 has been marked as a duplicate of this bug. ***
Comment 4 Aki Sasaki [:aki] 2009-11-02 15:47:09 PST
Multiple branches?
So we need to have en-US nightlies, l10n nightlies for 1.9.2 + m-c?
Comment 5 Joel Maher (:jmaher) 2009-11-02 17:28:12 PST
if we had to pick and choose, I would say priority list is:
* 1.9.2 en-US nightlies
* 1.9.2 l10n nightlies
* m-c en-US nightlies
* m-c l10n nightlies

Also an option for windows mobile would be really nice!  I am sure others would argue, but I would like to propose adding windows mobile 1.9.2 nightly updates before m-c updates.
Comment 6 Aki Sasaki [:aki] 2009-11-02 17:32:34 PST
Noted. WinMo is a completely separate process, however, and has a couple blockers, so it's not going to be part of this bug.
Comment 7 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-11-02 18:07:58 PST
I think getting updates for multi-locale 1.9.2 nightly and release fennec on linux-arm should be highest priority. 

(Maybe this is assumed but as there was no mention of it in the discussion so far, I just want to be certain.)
Comment 8 Joel Maher (:jmaher) 2009-11-02 18:12:36 PST
I think that is my first two items in the list.  We are releasing from 1.9.2 and en-US/multi are the two that would be the most helpful right now.
Comment 9 Aki Sasaki [:aki] 2009-11-09 12:06:46 PST
Created attachment 411235 [details] [diff] [review]
wip patch

Here's a WIP patch for that also needs a lot of buildbot-side logic.

I'd like to get this all doable from the command-line side, so I'm working on a script that will do that, especially since this has the potential of getting really large (75+ locales x 2 build types (release + nightly) x 2 branches)
Comment 10 Aki Sasaki [:aki] 2009-11-09 12:08:23 PST
Created attachment 411237 [details] [diff] [review]
wip buildbot configs portion

This is the buildbot side that I have running tests.
If I keep working on this, I want to add the .install file creation in here and upload both, and test.
Comment 11 Armen Zambrano [:armenzg] - Engineering productivity 2009-11-10 06:16:30 PST
*** Bug 527079 has been marked as a duplicate of this bug. ***
Comment 12 Aki Sasaki [:aki] 2009-11-10 13:57:03 PST
Created attachment 411512 [details] [diff] [review] does most of the heavy lifting.
Comment 13 Aki Sasaki [:aki] 2009-11-10 14:02:44 PST
Created attachment 411515 [details] [diff] [review]
debsign buildbot-configs

Currently running on staging-mobile-master, til I get a new box to run this.

Need to add the other locales as they become available, and I won't be surprised if I need to move the factory at some point, but this is working atm.

This is a pretty light config right now, but I'm thinking 75+ locales x 2 branches x 2 build types will be a lot of pollers + builders, hence the separate buildbot master.
Comment 14 Axel Hecht [:Pike] 2009-11-10 15:13:11 PST
ftp poller? that'd be sad.

FWIW, I'd be surprised if the per-locale debs wouldn't end up as one per directory, at least that's how the build system produces them right now.

I'd favour a triggered scheduler, with a descent idle time, so that it grabs a bunch of l10n debs at a time once they'll get done.
Comment 15 Aki Sasaki [:aki] 2009-11-10 15:32:09 PST
It's got one slave, so I'm not sure what the difference is, really.
So I can sendchange through the firewall, or ftppoller.

I'm planning on the debs each being in their own dir.
Comment 16 Nick Thomas [:nthomas] 2009-11-11 19:48:16 PST
Comment on attachment 411515 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/ b/debsign/
>+BRANCHES['mozilla-central']['locales'] = {
>+  'en-US': {
>+    'ftpUrls': [''],
>+    'searchString': "fennec_.*.deb",
>+  },

If there are lots of locales and a multi-locale build to be added here later I'd guess there's going to be lots of repetition here. If a couple of ftpUrls and a searchstring are all that are required then I'd suggest setting them at the BRANCHES['mozilla-central'] level along with a list of locales, and sticking some logic into factory creation loop.

>diff --git a/debsign/master.cfg b/debsign/master.cfg
>+c['slaves'] = []
>+for platform, names in SLAVES.items():
>+    for name in names:
>+        c['slaves'].append(BuildSlave(name, 'd3bs1gn', max_builds=1))

Lets store the password outside of a public VCS. What we do on pm* is a good model.

>+class DebRepoSign(BuildFactory):
>+    def __init__(self, locale, branch, buildToolsRepo, nightly=False,

s/branch/branchNick/ in this class (and callers) to limit confusion.

>+        self.addStep(DownloadFile,
>+         url_fn=get_fennecUrl,
>+         filename_property='fennec_filename',
>+         haltOnFailure=True,
>+         name="download_fennec",
>+         timeout=60*60,
>+        )

Why's the timeout so long here ?
Comment 17 Nick Thomas [:nthomas] 2009-11-11 19:48:21 PST
Comment on attachment 411512 [details] [diff] [review] does most of the heavy lifting.

Seems like a sensible extension to me, r+ with consideration to the suggestions below.

>diff --git a/release/signing/ b/release/signing/

This seems like a pretty important variable so we should error out if it's not set.

>+WORKDIR			?= /scratchbox/users/cltbld/home/cltbld/sign-debs/$(BRANCH_NICK)_$(LOCALE)

You could define SBOX_WORKDIR first, then use
WORKDIR	?= /scratchbox/users/cltbld/home/cltbld/${SBOX_WORKDIR}

>+#XULRUNNER_VERSION	= 1.9.3a1pre-20091109081649
>+#NIGHTLY			= 1

Leftovers from testing ?

> upload:
>-	rsync -e "ssh -i $(SSH_KEY)" -azv $(WORKDIR)/dists/. \
>+	rsync -e "ssh -i $(SSH_KEY)" $(RSYNC_ARGS) -azv $(WORKDIR)/dists $(WORKDIR)/$(INSTALL_FILENAME) \

In the download the -avz is over-ridable by RSYNC_ARGS, but not the upload. Is that deliberate ?
Comment 18 Aki Sasaki [:aki] 2009-11-16 11:31:47 PST
Question, for anyone who knows the answer:

Do you know if we want to keep old debs around for any reason?

In the nightlies I'm definitely nuking all old debs before populating the new ones... not doing so is just asking for disk space issues.

For the releases I can keep old ones around, but I'm not sure if it's useful to do so.  I suppose if this will be the only way to get old debs we shouldn't clean 'em up.

Anyway, my guess was to only have one nightly per repo, and keep old release debs.
Comment 19 Mark Finkle (:mfinkle) (use needinfo?) 2009-11-16 13:03:56 PST
QA definitely uses nightlies for checking regression ranges. Currently, we can grab from the tinderbox build FTP site. Would we still be able to do that?

Joel - Is that the way QA gets old debs now?
Comment 20 Mark Finkle (:mfinkle) (use needinfo?) 2009-11-16 13:04:46 PST
(In reply to comment #19)

> Joel - Is that the way QA gets old debs now?

I meant, "will it be ok for us to continue to use the tinderbox FTP site?"
Comment 21 Joel Maher (:jmaher) 2009-11-16 13:09:08 PST
yes, testing old builds via ftp is just fine.  Testing daily builds via .install repository is awesome!
Comment 22 Aki Sasaki [:aki] 2009-11-18 10:08:38 PST
Created attachment 413104 [details] [diff] [review] update

I've addressed the above issues with; I've also added l10n support (BASE_XULRUNNER_URL) and the touch-repository target to update the directory timestamp for easier detection when a repo is being updated.

I'd like to detect whether the deb has changed since the last update (wget -S --spider? curl --head?) and exit 0 if there's nothing to be done, but that seems like an enhancement that can wait.

Re-attaching for review in case for diffing against the previous patch. I can carry over the r+ but figured there might be enough different to need another review.

This is running in staging currently...

Configs to come.
Comment 23 Aki Sasaki [:aki] 2009-11-18 10:16:18 PST
Created attachment 413105 [details] [diff] [review]
debsign buildbot-configs

Staging and production configs in one dir; we can softlink as needed.

I was able to get the deb sign builders down to one per branch via NoMergeScheduler and a locale property... currently detected from the url.

I also removed the ftppollers, which weren't working.  Afaik, sendchanges are somewhat dependent on deb uploads using |make upload|, which they aren't currently, and may not for some time.

I toyed with a small shell script in cron, then ended up adding a trigger builder that sends multiple sendchanges after building urls using all-locales.  This is far from ideal, but it works (see above links) and if we get sendchanges coming from pm* going we can just remove this chunk of code.  Also, the advantage over a cron/script solution is we can easily re-trigger an entire branch through buildbot.
Comment 24 Armen Zambrano [:armenzg] - Engineering productivity 2009-11-18 10:21:59 PST
It seems that we have multiple repos after all. I guess it is manageable after all.
I can also see multiple xulrunner debs across all of the repos or is that a symlink?
How long (roughly) does it takes the whole process?

What do you need specifically from |make upload|? I am currently waiting on a 192 approval for xulrunner's |make upload|
Comment 25 Aki Sasaki [:aki] 2009-11-18 10:53:10 PST
Nope, it's multiple debs... don't think I can sign a softlink.

It's fast. All 80? repos get updated in ~15 min from when the Nightly scheduler kicks off.

I'm kind of vague on how our sendchanges go. I'm under the impression it's dependent on and generateBranchObjects somehow, though we may be able to hack it into

However, one benefit of doing it this way is that currently this setup puts zero additional load onto pm*.
Comment 26 Nick Thomas [:nthomas] 2009-11-19 15:31:48 PST
Comment on attachment 413104 [details] [diff] [review] update

>diff --git a/release/signing/ b/release/signing/
>+RELEASE			?=
>+LOCALE			?=

I'd was going to suggest using this pattern
BRANCH_NICK = $(error BRANCH_NICK must be defined)
but yours has the advantage of reporting more than one missing variable. Your choice.
>+	@touch $(STAGE_PATH)

Don't follow this bit, that seems to be a remote path rather than a local one.
Comment 27 Nick Thomas [:nthomas] 2009-11-19 20:45:15 PST
Comment on attachment 413105 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/ b/debsign/
>+TRIGGER_CONFIG['sendchange_master'] = ""

Only have this for staging ?

>diff --git a/debsign/master-main.cfg b/debsign/master-main.cfg
>+        # I love this too.  It's this or cron or writing a new ftppoller or
>+        # depending on a bunch of (currently) non-existent |make upload|s
>+        # for debs.

Um, yes. Roll on sendchange from pm02.

>diff --git a/debsign/master-production.cfg b/debsign/master-production.cfg
>+# Will probably move the following out once the factory's all done
>+from buildbot.process.factory import BuildFactory
>+from import ShellCommand, WithProperties, SetProperty
>+from buildbotcustom.steps.misc import DownloadFile
>+import re

Leftovers for the trash.
Comment 28 Nick Thomas [:nthomas] 2009-11-19 20:46:14 PST
Comment on attachment 413104 [details] [diff] [review] update

(In reply to comment #26)
> >+touch-repository:
> >+	@touch $(STAGE_PATH)
> >+
> Don't follow this bit, that seems to be a remote path rather than a local one.

r+ to fix this on checkin.
Comment 29 Aki Sasaki [:aki] 2009-11-20 12:35:21 PST
Comment on attachment 413104 [details] [diff] [review] update
Comment 30 Aki Sasaki [:aki] 2009-11-20 12:39:01 PST
Comment on attachment 413105 [details] [diff] [review]
debsign buildbot-configs
Comment 31 Aki Sasaki [:aki] 2009-11-20 13:06:01 PST
whitespace fix in ... I think we're all done here.
Comment 32 alex_mayorga 2009-11-20 13:39:22 PST
Kudos on fixing this one! Now how do I jump into the nightly builds on my N800? would there be a follow-up bug on end user documentation?
Comment 33 Aki Sasaki [:aki] 2009-11-20 13:43:46 PST
There are *_nightly.install files in each of the updated repos in

1) Choose the branch you want (trunk == mozilla-central. 1.9.2 == mozilla-1.9.2)
2) Choose the locale you want. This is probably multi but you can choose a different one if you want.
3) Verify that that repo has been updated recently... the timestamp in should be from that morning if everything went well.
4) Click on the _nightly.install file in the appropriate repository, and open it.

If/when we have release debs in those repos, there will be a second .install file (with no _nightly in the name) that points to the release section rather than the extras section.  But if you install that, you won't get nightly updates for those.
Comment 34 Armen Zambrano [:armenzg] - Engineering productivity 2009-11-23 08:21:25 PST
Verified that the catalog and Fennec get installed by clicking on this:
Comment 35 Armen Zambrano [:armenzg] - Engineering productivity 2009-11-25 07:20:58 PST
I got an update. It works as expected.
Comment 36 Aakash Desai [:aakashd] 2009-12-01 10:32:14 PST
I wish we have a in-litmus? flag on this :(. QA will get this into litmus and our wiki as well.

Note You need to log in before you can comment on or make changes to this bug.