469290 - Provide updates for nightly and release builds of Fennec on linux mobile.

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

It is important for Fennec nightly users to automatically get updates to the latest nightly build... just like we currently do for Firefox nightly users.

Some factors we know:
1) The builds need to be produced in deb format. This is already done.
2) There is no need to use AUS. The updating mechanism, and the download location logic, is provided within the deb packaging framework used by Fennec.

Some factors we dont know:
1) There's mobile signing logic which Stuart has in some scripts that we need to figure out, review and get integrated into the automation. There was also some discussion about specific text files, containing SHA1 checksums and GPG signatures. Format of these text files unclear.
2) Confirm that the location of builds on ftp.m.o, and the filenames used, are suitable for deb update mechanisms. For example, the xulrunner filenames need  buildid in the filename so that the deb packaging dependency checking would work correctly. In discussions today with Christian/Brad/Stuart, it seems like we are already doing all the right things. However, we need to verify if everything is ok?

What else do we need to figure out before we can start here?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

can we get this fixed sooner rather than later?  With the need for testing multiple branches every day, this is a critical piece.

Comment 4

[Aki Sasaki (not active)]

Multiple branches?
So we need to have en-US nightlies, l10n nightlies for 1.9.2 + m-c?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

if we had to pick and choose, I would say priority list is:
* 1.9.2 en-US nightlies
* 1.9.2 l10n nightlies
* m-c en-US nightlies
* m-c l10n nightlies

Also an option for windows mobile would be really nice!  I am sure others would argue, but I would like to propose adding windows mobile 1.9.2 nightly updates before m-c updates.

Comment 6

[Aki Sasaki (not active)]

Noted. WinMo is a completely separate process, however, and has a couple blockers, so it's not going to be part of this bug.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I think getting updates for multi-locale 1.9.2 nightly and release fennec on linux-arm should be highest priority. 

(Maybe this is assumed but as there was no mention of it in the discussion so far, I just want to be certain.)

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I think that is my first two items in the list.  We are releasing from 1.9.2 and en-US/multi are the two that would be the most helpful right now.

Comment 9

[Aki Sasaki (not active)]

Attachment: wip signdebs.mk patch

Here's a WIP patch for signdebs.mk that also needs a lot of buildbot-side logic.

I'd like to get this all doable from the command-line side, so I'm working on a script that will do that, especially since this has the potential of getting really large (75+ locales x 2 build types (release + nightly) x 2 branches)

Comment 10

[Aki Sasaki (not active)]

Attachment: wip buildbot configs portion

This is the buildbot side that I have running tests.
If I keep working on this, I want to add the .install file creation in here and upload both, and test.

Comment 12

[Aki Sasaki (not active)]

Attachment: signdebs.mk does most of the heavy lifting.

Comment 13

[Aki Sasaki (not active)]

Attachment: debsign buildbot-configs

Currently running on staging-mobile-master, til I get a new box to run this.

Need to add the other locales as they become available, and I won't be surprised if I need to move the factory at some point, but this is working atm.

This is a pretty light config right now, but I'm thinking 75+ locales x 2 branches x 2 build types will be a lot of pollers + builders, hence the separate buildbot master.

Comment 14

[Axel Hecht [:Pike]]

ftp poller? that'd be sad.

FWIW, I'd be surprised if the per-locale debs wouldn't end up as one per directory, at least that's how the build system produces them right now.

I'd favour a triggered scheduler, with a descent idle time, so that it grabs a bunch of l10n debs at a time once they'll get done.

Comment 15

[Aki Sasaki (not active)]

It's got one slave, so I'm not sure what the difference is, really.
So I can sendchange through the firewall, or ftppoller.

I'm planning on the debs each being in their own dir.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 411515 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/config.py b/debsign/config.py
>+BRANCHES['mozilla-central']['locales'] = {
>+  'en-US': {
>+    'ftpUrls': ['http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/'],
>+    'searchString': "fennec_.*.deb",
>+  },

If there are lots of locales and a multi-locale build to be added here later I'd guess there's going to be lots of repetition here. If a couple of ftpUrls and a searchstring are all that are required then I'd suggest setting them at the BRANCHES['mozilla-central'] level along with a list of locales, and sticking some logic into factory creation loop.

>diff --git a/debsign/master.cfg b/debsign/master.cfg
>+c['slaves'] = []
>+for platform, names in SLAVES.items():
>+    for name in names:
>+        c['slaves'].append(BuildSlave(name, 'd3bs1gn', max_builds=1))

Lets store the password outside of a public VCS. What we do on pm* is a good model.

>+class DebRepoSign(BuildFactory):
>+    def __init__(self, locale, branch, buildToolsRepo, nightly=False,

s/branch/branchNick/ in this class (and callers) to limit confusion.

>+        self.addStep(DownloadFile,
>+         url_fn=get_fennecUrl,
>+         filename_property='fennec_filename',
>+         haltOnFailure=True,
>+         name="download_fennec",
>+         timeout=60*60,
>+        )

Why's the timeout so long here ?

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 411512 [details] [diff] [review]
signdebs.mk does most of the heavy lifting.

Seems like a sensible extension to me, r+ with consideration to the suggestions below.

>diff --git a/release/signing/signdebs.mk b/release/signing/signdebs.mk
>+FENNEC_FILEURL	?=

This seems like a pretty important variable so we should error out if it's not set.

>+XULRUNNER_VERSION	?=
>+WORKDIR			?= /scratchbox/users/cltbld/home/cltbld/sign-debs/$(BRANCH_NICK)_$(LOCALE)
>+SBOX_WORKDIR	?= sign-debs/$(BRANCH_NICK)_$(LOCALE)

You could define SBOX_WORKDIR first, then use
WORKDIR	?= /scratchbox/users/cltbld/home/cltbld/${SBOX_WORKDIR}

>+#FENNEC_FILEURL	= http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/fennec_1.0b5_armel.deb
>+#XULRUNNER_VERSION	= 1.9.3a1pre-20091109081649
>+#NIGHTLY			= 1

Leftovers from testing ?

> upload:
>-	rsync -e "ssh -i $(SSH_KEY)" -azv $(WORKDIR)/dists/. \
>+	rsync -e "ssh -i $(SSH_KEY)" $(RSYNC_ARGS) -azv $(WORKDIR)/dists $(WORKDIR)/$(INSTALL_FILENAME) \
> 	  $(STAGE_USERNAME)@$(STAGE_SERVER):$(STAGE_PATH)/.

In the download the -avz is over-ridable by RSYNC_ARGS, but not the upload. Is that deliberate ?

Comment 18

[Aki Sasaki (not active)]

Question, for anyone who knows the answer:

Do you know if we want to keep old debs around for any reason?

In the nightlies I'm definitely nuking all old debs before populating the new ones... not doing so is just asking for disk space issues.

For the releases I can keep old ones around, but I'm not sure if it's useful to do so.  I suppose if this will be the only way to get old debs we shouldn't clean 'em up.

Anyway, my guess was to only have one nightly per repo, and keep old release debs.

Comment 19

[Mark Finkle (:mfinkle) (use needinfo?)]

QA definitely uses nightlies for checking regression ranges. Currently, we can grab from the tinderbox build FTP site. Would we still be able to do that?

Joel - Is that the way QA gets old debs now?

Comment 20

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to comment #19)

> Joel - Is that the way QA gets old debs now?

I meant, "will it be ok for us to continue to use the tinderbox FTP site?"

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

yes, testing old builds via ftp is just fine.  Testing daily builds via .install repository is awesome!

Comment 22

[Aki Sasaki (not active)]

Attachment: signdebs.mk update

I've addressed the above issues with signdebs.mk; I've also added l10n support (BASE_XULRUNNER_URL) and the touch-repository target to update the directory timestamp for easier detection when a repo is being updated.

I'd like to detect whether the deb has changed since the last update (wget -S --spider? curl --head?) and exit 0 if there's nothing to be done, but that seems like an enhancement that can wait.

Re-attaching for review in case for diffing against the previous patch. I can carry over the r+ but figured there might be enough different to need another review.

This is running in staging currently...
http://staging-mobile-master.mv.mozilla.com/debtest/
http://staging-mobile-master.mv.mozilla.com:8011/waterfall

Configs to come.

Comment 23

[Aki Sasaki (not active)]

Attachment: debsign buildbot-configs

Staging and production configs in one dir; we can softlink as needed.

I was able to get the deb sign builders down to one per branch via NoMergeScheduler and a locale property... currently detected from the url.

I also removed the ftppollers, which weren't working.  Afaik, sendchanges are somewhat dependent on deb uploads using |make upload|, which they aren't currently, and may not for some time.

I toyed with a small shell script in cron, then ended up adding a trigger builder that sends multiple sendchanges after building urls using all-locales.  This is far from ideal, but it works (see above links) and if we get sendchanges coming from pm* going we can just remove this chunk of code.  Also, the advantage over a cron/script solution is we can easily re-trigger an entire branch through buildbot.

Comment 24

[Armen [:armenzg]]

It seems that we have multiple repos after all. I guess it is manageable after all.
I can also see multiple xulrunner debs across all of the repos or is that a symlink?
How long (roughly) does it takes the whole process?

What do you need specifically from |make upload|? I am currently waiting on a 192 approval for xulrunner's |make upload|

Comment 25

[Aki Sasaki (not active)]

Nope, it's multiple debs... don't think I can sign a softlink.

It's fast. All 80? repos get updated in ~15 min from when the Nightly scheduler kicks off.

I'm kind of vague on how our sendchanges go. I'm under the impression it's dependent on post_upload.py and generateBranchObjects somehow, though we may be able to hack it into mobile_master.py.

However, one benefit of doing it this way is that currently this setup puts zero additional load onto pm*.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 413104 [details] [diff] [review]
signdebs.mk update

>diff --git a/release/signing/signdebs.mk b/release/signing/signdebs.mk
>+RELEASE			?=
>+BRANCH_NICK		?=
>+LOCALE			?=

I'd was going to suggest using this pattern
BRANCH_NICK = $(error BRANCH_NICK must be defined)
but yours has the advantage of reporting more than one missing variable. Your choice.
 
>+touch-repository:
>+	@touch $(STAGE_PATH)
>+

Don't follow this bit, that seems to be a remote path rather than a local one.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 413105 [details] [diff] [review]
debsign buildbot-configs

>diff --git a/debsign/config-staging.py b/debsign/config-staging.py
>+TRIGGER_CONFIG['sendchange_master'] = "staging-mobile-master.mv.mozilla.com:9011"

Only have this for staging ?

>diff --git a/debsign/master-main.cfg b/debsign/master-main.cfg
>+        # I love this too.  It's this or cron or writing a new ftppoller or
>+        # depending on a bunch of (currently) non-existent |make upload|s
>+        # for debs.

Um, yes. Roll on sendchange from pm02.

>diff --git a/debsign/master-production.cfg b/debsign/master-production.cfg
>+# Will probably move the following out once the factory's all done
>+from buildbot.process.factory import BuildFactory
>+from buildbot.steps.shell import ShellCommand, WithProperties, SetProperty
>+from buildbotcustom.steps.misc import DownloadFile
>+import re

Leftovers for the trash.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 413104 [details] [diff] [review]
signdebs.mk update

(In reply to comment #26)
> >+touch-repository:
> >+	@touch $(STAGE_PATH)
> >+
> 
> Don't follow this bit, that seems to be a remote path rather than a local one.

r+ to fix this on checkin.

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 413104 [details] [diff] [review]
signdebs.mk update

http://hg.mozilla.org/build/tools/rev/037990e025b0

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 413105 [details] [diff] [review]
debsign buildbot-configs

http://hg.mozilla.org/build/buildbot-configs/rev/afddb3e375ea

Comment 31

[Aki Sasaki (not active)]

whitespace fix in http://hg.mozilla.org/build/buildbot-configs/rev/e3748c85f72f

http://ftp.mozilla.org/pub/mozilla.org/mobile/repos/ ... I think we're all done here.

Comment 32

[alex_mayorga]

Kudos on fixing this one! Now how do I jump into the nightly builds on my N800? would there be a follow-up bug on end user documentation?

Comment 33

[Aki Sasaki (not active)]

There are *_nightly.install files in each of the updated repos in http://ftp.mozilla.org/pub/mozilla.org/mobile/repos/

1) Choose the branch you want (trunk == mozilla-central. 1.9.2 == mozilla-1.9.2)
2) Choose the locale you want. This is probably multi but you can choose a different one if you want.
3) Verify that that repo has been updated recently... the timestamp in http://ftp.mozilla.org/pub/mozilla.org/mobile/repos/ should be from that morning if everything went well.
4) Click on the _nightly.install file in the appropriate repository, and open it.

If/when we have release debs in those repos, there will be a second .install file (with no _nightly in the name) that points to the release section rather than the extras section.  But if you install that, you won't get nightly updates for those.

Comment 34

[Armen [:armenzg]]

Verified that the catalog and Fennec get installed by clicking on this:
http://ftp.mozilla.org/pub/mozilla.org/mobile/repos/1.9.2_multi/1.9.2_multi_nightly.install

Comment 35

[Armen [:armenzg]]

I got an update. It works as expected.

Comment 36

[Aakash Desai [:aakashd]]

I wish we have a in-litmus? flag on this :(. QA will get this into litmus and our wiki as well.