456082 - Maemo releases need to be signed

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Brad Lassey [:blassey] (use needinfo?)]

The binary deb installers and associated meta data ("Release" files) need to be signed for non-developers to install fennec on their devices. The debsign helper app should do the trick.

More info can be found here:
http://www.debian-administration.org/articles/174

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Do you have a list of the instructions we're to automate?  Also, our current signing machine is win32 based. Is this something that needs to be done in scratchbox or on a debian machine?

Comment 2

[Stuart Parmenter]

John: You'll probably need a Linux machine at least, a debian box will be easiest but it should be possible to install dpkg/debsign stuff on a redhat-based box.

Comment 3

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: make signed distro script

you only need gpg to do the actual signing, so a win32 box would work.  I've attached the script that I use to make the distros and sign them.  

The dpkg-scanpackages and apt-ftparchibe utilities are to make the distro and those you'll probably need a linux box for them.

Comment 4

[Chris AtLee [:catlee]]

What URL are the devices currently looking at to get new packages?  Are they expecting a complete debian repository layout?

If we put the Packages[.gz], Release and Release.gpg files at http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-browser-linux-arm/, will that be sufficient?

Comment 5

[Chris AtLee [:catlee]]

Attachment: Add steps to mobile_master to create debian metadata files

Comment 6

[Chris AtLee [:catlee]]

Attachment: Public key that Release file is signed with in staging

Comment 7

[Chris AtLee [:catlee]]

So right now the builds in staging are also generating the Packages.gz, Release and Release.gpg files and uploading them to:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-browser-linux-arm/

The public part of the key that the Release file is signed with is attached to this bug.  It's currently only on the staging machines, and expires in a month.  We'll have to figure out if/how we want the nightly builds signed, and possibly have several keypairs for automated nightly signing vs. official releases signing.

Comment 8

[Chris AtLee [:catlee]]

We need to have a separate upload directory for each build so that machines aren't stepping on each other's toes.

We should also have a nightly builder that uploads a signed build to an appropriate location.

Comment 9

[Chris AtLee [:catlee]]

We may be able to distribute alpha / beta and final releases via maemo.org as well.  See bug 457307.

Comment 10

[Chris AtLee [:catlee]]

(In reply to comment #8)
> We should also have a nightly builder that uploads a signed build to an
> appropriate location.

This is bug 469294

Comment 11

[Chris AtLee [:catlee]]

(In reply to comment #10)
> (In reply to comment #8)
> > We should also have a nightly builder that uploads a signed build to an
> > appropriate location.
> 
> This is bug 469294

Sorry, I meant the part about separate upload directories for each build is bug 469294

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #7)
> So right now the builds in staging are also generating the Packages.gz, Release
> and Release.gpg files and uploading them to:
> http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-browser-linux-arm/
> 
> The public part of the key that the Release file is signed with is attached to
> this bug.  It's currently only on the staging machines, and expires in a month.


Stuart/Brad: Can you have a look at this and confirm if the signatures work as expected?

Comment 13

[Brad Lassey [:blassey] (use needinfo?)]

What ever key were going to use to sign release builds we need to register with maemo.org so users stop getting the warning about an untrusted installer

Comment 14

[[e-mail] (:bhearsum)]

I really feel like everything except the gpg command should be done as part of the 'make deb' target. Buildbot is not the place to do it IMHO.

Ted, what do you think about that?

Comment 15

[(not currently active) Ted Mielczarek]

I can't say I really understand what's happening there, but as long as it's something that can happen for every nightly build, yeah, that makes sense.

Comment 16

[Chris AtLee [:catlee]]

(In reply to comment #14)
> I really feel like everything except the gpg command should be done as part of
> the 'make deb' target. Buildbot is not the place to do it IMHO.

Debian's package management tools rely on a few meta-data files to get their job done: Packages and Release.  Release is mostly just a meta-meta-data file that can have some information about the package repository that it's in, as well as listing out all the Package files, and their hashes.

The Packages file lists out all the packages in the package repository, along with all their version information and dependencies.

Both are very specific to the particular package repository in which they are resident.  I don't think that they belong in the 'make deb' target, as they're not related to building or packaging the code, but are related directly with the release of the packages to our users.

Also, if we wanted to distribute other .deb files, then 'make deb' would need to know about those as well in order to generate valid Packages and Release files.

If this code doesn't belong in 'make deb', and doesn't belong in buildbot (which I would agree with - it's pretty ugly in there), then maybe it should go into the release automation code?

Comment 17

[(not currently active) Ted Mielczarek]

The "make deb" target is already specific per-app, and we have a bunch of files related to it in the installer dir:
http://hg.mozilla.org/mobile-browser/file/0361cd36d50d/installer/debian/

Comment 18

[Brad Lassey [:blassey] (use needinfo?)]

IMO, this code doesn't belong in make deb.  I don't know the separation of the buildbot code and release automation code, but it would seem like it belongs in one of those.

Comment 19

[[e-mail] (:bhearsum)]

Ok - Brad and Chris know more about debian packaging than I so if you guys don't think 'make deb is a good place, that's fine.

I'm pretty uncomfortable with so much bash in the buildbot config, though. Is there a better place in the build system for this to live? If not, lets put it in a wrapper script in hg.m.o/build/tools.

Comment 20

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #12)
> Stuart/Brad: Can you have a look at this and confirm if the signatures work as
> expected?

From irc with blassey:

1) The various meta data files generated are all valid, so thats good.

2) We need to get the GPG key registered with maemo.org. This will prevent installer-time warnings on the devices. I'd like to register our existing GPG key, which we already use when signing Firefox and Thunderbird releases. This seems better then creating and registering a new different GPG key just for mobile builds - as we'd then need to support two GPG keys. Sound ok?

3) Like bhearsum, I am also uncomfortable putting too many scripts directly in buildbot. The more we can keep build logic in makefiles, the easier it is to do buildbot upgrades, for developers to see/reproduce what automation does, and, if needed, to switch to another Continuous Integration Framework in the future. However, lets not bikeshed on this. For now, we'll not put it in "make deb", and we'll get something working. However, I'd like to revisit this later, as I agree with bhearsum that this logic should be in the makefiles, not in framework, and not in misc script files. Worst case, we could even just call this script manually - after all, its no worse then what we still do when manually signing Firefox/Thunderbird releases.

Comment 21

[[e-mail] (:bhearsum)]

(In reply to comment #20)
> However, lets not bikeshed on this. For now, we'll not put it in "make deb",
> and we'll get something working. However, I'd like to revisit this later, as I
> agree with bhearsum that this logic should be in the makefiles, not in
> framework, and not in misc script files. Worst case, we could even just call
> this script manually - after all, its no worse then what we still do when
> manually signing Firefox/Thunderbird releases.

If we want to use our "real" GPG key for this we'll need to put it on all of the Linux slaves. We should probably discuss that before deciding to...

With regard to not bikeshedding, I guess that's fine. I suspect we won't "revisit" this for a long time, though, and end up maintaining it inside of buildbotcustom.

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 354262 [details] [diff] [review]
Add steps to mobile_master to create debian metadata files

Catlee reminded me that we're planning to generate a new GPG key for nightlies. As long as you promise to move this somewhere better (script, makefile, w/e) this is fine for now.

Comment 23

[Chris AtLee [:catlee]]

From my pre-vacation discussions on IRC, I believe the plan was something like this:

- Nightly mobile builds get signed by a new "Nightly" GPG key, and will be available from mozilla.org.  We'll need code somewhere to create the debian files.

- Releases (betas, finals, etc.) will be uploaded to maemo for distribution.  This needs more research to figure out how to get this working.

Comment 24

[Aki Sasaki (not active)]

Attachment: create nightly source debs

Ok.  (disclaimer) I might be confused, so correct me if I'm wrong.

It appears that the new process is to create a source deb and upload it for maemo to build for the chinook and/or diablo extras repositories.  That upload can be done via the web or through ssh.

(found here:
  https://garage.maemo.org/extras-assistant/index.php?step=2
but you may have to log in.  Also referred to here:
  http://wiki.maemo.org/Uploading_to_Extras )

This patch copies the installer Makefile to debian/rules and runs the dpkg-buildpackage command and uploads the changes/source deb in nightly builds.  I had to remove the |GARBAGE += debian/changelog| from the xulrunner installer Makefile for it to work.

Tested in staging, though I haven't really done anything with the resulting files.

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 365389 [details] [diff] [review]
create nightly source debs

nit:

>-    def addUploadSteps(self, platform):
>+    def addUploadSteps(self, platform, packageGlob=None):
>+        if (packageGlob == None):
>+            packageGlob = self.packageGlob

()'s aren't required around conditions in python.  Also, 'if packageGlob is None:' is more pythonic.

>+    def addSourcePackageSteps(self):
>+        if (self.nightly):

same here

>+            self.addStep(ShellCommand,
>+                command="echo '#!/usr/bin/make -f' > rules",
>+                workdir="%s/%s/%s/mobile/mobile/installer/debian" %
>+                        (self.baseWorkDir, self.branchName, self.objdir),
>+                haltOnFailure=True
>+            )
>+            self.addStep(ShellCommand,
>+                command="cat ../Makefile >> rules",
>+                workdir="%s/%s/%s/mobile/mobile/installer/debian" %
>+                        (self.baseWorkDir, self.branchName, self.objdir),
>+                description=['create', 'mobile', 'debian/build'],
>+                haltOnFailure=True
>+            )
>+            self.addStep(ShellCommand,
>+                command=[self.scratchboxPath, '-p', '-d',
>+                         'build/%s/%s/mobile/mobile/installer' %
>+                         (self.branchName, self.objdir),
>+                         "dpkg-buildpackage -r'fakeroot' -sa -S"],
>+                description=['make', 'mobile', 'source', 'package'],
>+                haltOnFailure=True
>+            )
>+            self.addStep(ShellCommand,
>+                command="echo '#!/usr/bin/make -f' > rules",
>+                workdir="%s/%s/%s/xulrunner/xulrunner/installer/debian" %
>+                        (self.baseWorkDir, self.branchName, self.objdir),
>+                haltOnFailure=True
>+            )
>+            self.addStep(ShellCommand,
>+                command="grep -v 'GARBAGE += debian/changelog' Makefile " +
>+                        ">> debian/rules",
>+                workdir="%s/%s/%s/xulrunner/xulrunner/installer" %
>+                        (self.baseWorkDir, self.branchName, self.objdir),
>+                description=['create', 'xulrunner', 'debian/build'],
>+                haltOnFailure=True
>+            )
>+            self.addStep(ShellCommand,
>+                command=[self.scratchboxPath, '-p', '-d',
>+                         'build/%s/%s/xulrunner/xulrunner/installer' %
>+                         (self.branchName, self.objdir),
>+                         "dpkg-buildpackage -r'fakeroot' -sa -S"],
>+                description=['make', 'xulrunner', 'source', 'package'],
>+                haltOnFailure=True
>+            )

A better place for these steps may be in a script in the build-tools repo?  Ok here for now though.

r+ with the style nits above fixed.

Comment 26

[Aki Sasaki (not active)]

Comment on attachment 365389 [details] [diff] [review]
create nightly source debs

rev 208:459245ff1a56

Comment 27

[Aki Sasaki (not active)]

Ok, it looks like we need a real debian/rules and/or make deb-source.
I'll file another bug.

Comment 28

[Aki Sasaki (not active)]

I'm trying to find a quick workaround, but according to http://wiki.maemo.org/Uploading_to_Extras#.22non-free.22_packages :

  "Direct upload of binary packages is allowed for non-free packages only."

Changing the summary accordingly.

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 365389 [details] [diff] [review]
create nightly source debs

Backed out so people don't get confused by the broken source debs.

Comment 30

[Aki Sasaki (not active)]

Ok.

I've successfully run the signing steps and created a test repository, from
which I successfully installed b1.

Some observations:

1) The signing doesn't change the deb at all, just other files in the
repository.
2) It doesn't seem to matter which key you use; any non-Nokia-approved key is
equivalent and will throw a warning.

I would guess when we're added to Maemo Extras we just need to send the
debs as built.

It's simple to install nightly debs, as well, so I'm assuming we want a nightly
repository for ease of updating devices (is this the case?).  I'm thinking

- rsync repository from ftp
- clean local repository (we need to decide which old builds stick around, if
any.  if none, we can probably skip the 1st step)
- add new build
- rsync up to ftp with --delete

This may cause issues if there are multiple builds pushing concurrently. Not
sure how to fix this... lock file? Special builder that just does these?

Comment 31

[Aki Sasaki (not active)]

Attachment: makefile to update mozilla deb repository

Sounds like a nightly repository isn't in the scope of this bug.

I'm using your makefile from bug 409479 as a model, and adding the steps from the "make signed distro script" above.  This should only be needed at release time, and may likely become obsolete when we have source debs and are distributed through Maemo.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 368185 [details]
makefile to update mozilla deb repository

>echo:
>	@echo STAGE_USERNAME: $(STAGE_USERNAME)
>	@echo SSH_KEY: $(SSH_KEY)
>	@echo WORKDIR: $(WORKDIR)
>	@echo SBOX_WORKDIR: $(SBOX_WORKDIR)

Please echo out all of the variables here.

This looks good otherwise, functionality wise.


I'm assuming that we want to sign the repository with the same GPG key we use for Firefox. In that case, we'll need either a Linux machine alongside keymaster, or a Linux VM on keymaster. The only place our GPG key is (AFAIK) is on a USB key connected to that machine. I'm paranoid about putting it on build machines since we have been known to give developers access to them on occasion, because they live in the colo as opposed to the office, etc.

Come to think of it, even if we use a different key we should be just as paranoid about where it lives.

I'm not entirely certain where this should live. I was thinking of putting the Makefile in bug 409479 in hg.m.o/build/tools/release/signing. If that sounds OK this can probably live there too.

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 368185 [details]
makefile to update mozilla deb repository

rev 247:6c5805d0da84

Comment 34

[Aki Sasaki (not active)]

Ok.
We have a working repository, a makefile to automate the process, and a bug to create source debs (bug 481508) that may make this all obsolete. I'm going to call this fixed.