Closed Bug 469290 Opened 12 years ago Closed 11 years ago

Provide updates for nightly and release builds of Fennec on linux mobile.


(Release Engineering :: General, defect, P2)



(Not tracked)



(Reporter: joduinn, Assigned: aki)



(Keywords: mobile)


(2 files, 4 obsolete files)

It is important for Fennec nightly users to automatically get updates to the latest nightly build... just like we currently do for Firefox nightly users.

Some factors we know:
1) The builds need to be produced in deb format. This is already done.
2) There is no need to use AUS. The updating mechanism, and the download location logic, is provided within the deb packaging framework used by Fennec.

Some factors we dont know:
1) There's mobile signing logic which Stuart has in some scripts that we need to figure out, review and get integrated into the automation. There was also some discussion about specific text files, containing SHA1 checksums and GPG signatures. Format of these text files unclear.
2) Confirm that the location of builds on ftp.m.o, and the filenames used, are suitable for deb update mechanisms. For example, the xulrunner filenames need  buildid in the filename so that the deb packaging dependency checking would work correctly. In discussions today with Christian/Brad/Stuart, it seems like we are already doing all the right things. However, we need to verify if everything is ok?

What else do we need to figure out before we can start here?
Priority: -- → P3
Keywords: mobile
Duplicate of this bug: 475799
can we get this fixed sooner rather than later?  With the need for testing multiple branches every day, this is a critical piece.
Assignee: nobody → aki
Duplicate of this bug: 519748
Component: Release Engineering: Future → Release Engineering
Multiple branches?
So we need to have en-US nightlies, l10n nightlies for 1.9.2 + m-c?
Priority: P3 → P2
if we had to pick and choose, I would say priority list is:
* 1.9.2 en-US nightlies
* 1.9.2 l10n nightlies
* m-c en-US nightlies
* m-c l10n nightlies

Also an option for windows mobile would be really nice!  I am sure others would argue, but I would like to propose adding windows mobile 1.9.2 nightly updates before m-c updates.
Noted. WinMo is a completely separate process, however, and has a couple blockers, so it's not going to be part of this bug.
I think getting updates for multi-locale 1.9.2 nightly and release fennec on linux-arm should be highest priority. 

(Maybe this is assumed but as there was no mention of it in the discussion so far, I just want to be certain.)
I think that is my first two items in the list.  We are releasing from 1.9.2 and en-US/multi are the two that would be the most helpful right now.
Attached patch wip patch (obsolete) — Splinter Review
Here's a WIP patch for that also needs a lot of buildbot-side logic.

I'd like to get this all doable from the command-line side, so I'm working on a script that will do that, especially since this has the potential of getting really large (75+ locales x 2 build types (release + nightly) x 2 branches)
Attached patch wip buildbot configs portion (obsolete) — Splinter Review
This is the buildbot side that I have running tests.
If I keep working on this, I want to add the .install file creation in here and upload both, and test.
Duplicate of this bug: 527079
Depends on: 527076
Attachment #411235 - Attachment is obsolete: true
Attachment #411512 - Flags: review?(nthomas)
Attached patch debsign buildbot-configs (obsolete) — Splinter Review
Currently running on staging-mobile-master, til I get a new box to run this.

Need to add the other locales as they become available, and I won't be surprised if I need to move the factory at some point, but this is working atm.

This is a pretty light config right now, but I'm thinking 75+ locales x 2 branches x 2 build types will be a lot of pollers + builders, hence the separate buildbot master.
Attachment #411237 - Attachment is obsolete: true
Attachment #411515 - Flags: review?(nthomas)
ftp poller? that'd be sad.

FWIW, I'd be surprised if the per-locale debs wouldn't end up as one per directory, at least that's how the build system produces them right now.

I'd favour a triggered scheduler, with a descent idle time, so that it grabs a bunch of l10n debs at a time once they'll get done.
It's got one slave, so I'm not sure what the difference is, really.
So I can sendchange through the firewall, or ftppoller.

I'm planning on the debs each being in their own dir.
Comment on attachment 411515 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/ b/debsign/
>+BRANCHES['mozilla-central']['locales'] = {
>+  'en-US': {
>+    'ftpUrls': [''],
>+    'searchString': "fennec_.*.deb",
>+  },

If there are lots of locales and a multi-locale build to be added here later I'd guess there's going to be lots of repetition here. If a couple of ftpUrls and a searchstring are all that are required then I'd suggest setting them at the BRANCHES['mozilla-central'] level along with a list of locales, and sticking some logic into factory creation loop.

>diff --git a/debsign/master.cfg b/debsign/master.cfg
>+c['slaves'] = []
>+for platform, names in SLAVES.items():
>+    for name in names:
>+        c['slaves'].append(BuildSlave(name, 'd3bs1gn', max_builds=1))

Lets store the password outside of a public VCS. What we do on pm* is a good model.

>+class DebRepoSign(BuildFactory):
>+    def __init__(self, locale, branch, buildToolsRepo, nightly=False,

s/branch/branchNick/ in this class (and callers) to limit confusion.

>+        self.addStep(DownloadFile,
>+         url_fn=get_fennecUrl,
>+         filename_property='fennec_filename',
>+         haltOnFailure=True,
>+         name="download_fennec",
>+         timeout=60*60,
>+        )

Why's the timeout so long here ?
Attachment #411515 - Flags: review?(nthomas) → review-
Comment on attachment 411512 [details] [diff] [review] does most of the heavy lifting.

Seems like a sensible extension to me, r+ with consideration to the suggestions below.

>diff --git a/release/signing/ b/release/signing/

This seems like a pretty important variable so we should error out if it's not set.

>+WORKDIR			?= /scratchbox/users/cltbld/home/cltbld/sign-debs/$(BRANCH_NICK)_$(LOCALE)

You could define SBOX_WORKDIR first, then use
WORKDIR	?= /scratchbox/users/cltbld/home/cltbld/${SBOX_WORKDIR}

>+#XULRUNNER_VERSION	= 1.9.3a1pre-20091109081649
>+#NIGHTLY			= 1

Leftovers from testing ?

> upload:
>-	rsync -e "ssh -i $(SSH_KEY)" -azv $(WORKDIR)/dists/. \
>+	rsync -e "ssh -i $(SSH_KEY)" $(RSYNC_ARGS) -azv $(WORKDIR)/dists $(WORKDIR)/$(INSTALL_FILENAME) \

In the download the -avz is over-ridable by RSYNC_ARGS, but not the upload. Is that deliberate ?
Attachment #411512 - Flags: review?(nthomas) → review+
Question, for anyone who knows the answer:

Do you know if we want to keep old debs around for any reason?

In the nightlies I'm definitely nuking all old debs before populating the new ones... not doing so is just asking for disk space issues.

For the releases I can keep old ones around, but I'm not sure if it's useful to do so.  I suppose if this will be the only way to get old debs we shouldn't clean 'em up.

Anyway, my guess was to only have one nightly per repo, and keep old release debs.
QA definitely uses nightlies for checking regression ranges. Currently, we can grab from the tinderbox build FTP site. Would we still be able to do that?

Joel - Is that the way QA gets old debs now?
(In reply to comment #19)

> Joel - Is that the way QA gets old debs now?

I meant, "will it be ok for us to continue to use the tinderbox FTP site?"
yes, testing old builds via ftp is just fine.  Testing daily builds via .install repository is awesome!
I've addressed the above issues with; I've also added l10n support (BASE_XULRUNNER_URL) and the touch-repository target to update the directory timestamp for easier detection when a repo is being updated.

I'd like to detect whether the deb has changed since the last update (wget -S --spider? curl --head?) and exit 0 if there's nothing to be done, but that seems like an enhancement that can wait.

Re-attaching for review in case for diffing against the previous patch. I can carry over the r+ but figured there might be enough different to need another review.

This is running in staging currently...

Configs to come.
Attachment #411512 - Attachment is obsolete: true
Attachment #413104 - Flags: review?(nrthomas)
Staging and production configs in one dir; we can softlink as needed.

I was able to get the deb sign builders down to one per branch via NoMergeScheduler and a locale property... currently detected from the url.

I also removed the ftppollers, which weren't working.  Afaik, sendchanges are somewhat dependent on deb uploads using |make upload|, which they aren't currently, and may not for some time.

I toyed with a small shell script in cron, then ended up adding a trigger builder that sends multiple sendchanges after building urls using all-locales.  This is far from ideal, but it works (see above links) and if we get sendchanges coming from pm* going we can just remove this chunk of code.  Also, the advantage over a cron/script solution is we can easily re-trigger an entire branch through buildbot.
Attachment #411515 - Attachment is obsolete: true
Attachment #413105 - Flags: review?(nrthomas)
It seems that we have multiple repos after all. I guess it is manageable after all.
I can also see multiple xulrunner debs across all of the repos or is that a symlink?
How long (roughly) does it takes the whole process?

What do you need specifically from |make upload|? I am currently waiting on a 192 approval for xulrunner's |make upload|
Nope, it's multiple debs... don't think I can sign a softlink.

It's fast. All 80? repos get updated in ~15 min from when the Nightly scheduler kicks off.

I'm kind of vague on how our sendchanges go. I'm under the impression it's dependent on and generateBranchObjects somehow, though we may be able to hack it into

However, one benefit of doing it this way is that currently this setup puts zero additional load onto pm*.
Comment on attachment 413104 [details] [diff] [review] update

>diff --git a/release/signing/ b/release/signing/
>+RELEASE			?=
>+LOCALE			?=

I'd was going to suggest using this pattern
BRANCH_NICK = $(error BRANCH_NICK must be defined)
but yours has the advantage of reporting more than one missing variable. Your choice.
>+	@touch $(STAGE_PATH)

Don't follow this bit, that seems to be a remote path rather than a local one.
Comment on attachment 413105 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/ b/debsign/
>+TRIGGER_CONFIG['sendchange_master'] = ""

Only have this for staging ?

>diff --git a/debsign/master-main.cfg b/debsign/master-main.cfg
>+        # I love this too.  It's this or cron or writing a new ftppoller or
>+        # depending on a bunch of (currently) non-existent |make upload|s
>+        # for debs.

Um, yes. Roll on sendchange from pm02.

>diff --git a/debsign/master-production.cfg b/debsign/master-production.cfg
>+# Will probably move the following out once the factory's all done
>+from buildbot.process.factory import BuildFactory
>+from import ShellCommand, WithProperties, SetProperty
>+from buildbotcustom.steps.misc import DownloadFile
>+import re

Leftovers for the trash.
Attachment #413105 - Flags: review?(nrthomas) → review+
Attachment #413104 - Flags: review?(nrthomas) → review+
Comment on attachment 413104 [details] [diff] [review] update

(In reply to comment #26)
> >+touch-repository:
> >+	@touch $(STAGE_PATH)
> >+
> Don't follow this bit, that seems to be a remote path rather than a local one.

r+ to fix this on checkin.
whitespace fix in ... I think we're all done here.
Closed: 11 years ago
Resolution: --- → FIXED
Kudos on fixing this one! Now how do I jump into the nightly builds on my N800? would there be a follow-up bug on end user documentation?
There are *_nightly.install files in each of the updated repos in

1) Choose the branch you want (trunk == mozilla-central. 1.9.2 == mozilla-1.9.2)
2) Choose the locale you want. This is probably multi but you can choose a different one if you want.
3) Verify that that repo has been updated recently... the timestamp in should be from that morning if everything went well.
4) Click on the _nightly.install file in the appropriate repository, and open it.

If/when we have release debs in those repos, there will be a second .install file (with no _nightly in the name) that points to the release section rather than the extras section.  But if you install that, you won't get nightly updates for those.
Verified that the catalog and Fennec get installed by clicking on this:
I got an update. It works as expected.
I wish we have a in-litmus? flag on this :(. QA will get this into litmus and our wiki as well.
Product: → Release Engineering
You need to log in before you can comment on or make changes to this bug.