831433 - Create script for imaging releng panda build artifacts

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Mark Côté [:mcote]]

The releng builds are distributed in the form of build artifacts (tarballs) that need to be installed in a preseed image.  Mozpool does this via pxe booting with the help of some scripts that are run on the panda itself.  It would be convenient to have something similar that could be run on a linux machine to image an SD card directly.

Comment 1

[Mark Côté [:mcote]]

Attachment: Script to fetch and install preseed image and build artifacts

Okay this has successfully installed a b2g build to my SD card.  I've tried to make it as simple as possible such that you only have to specify the device.  The script can figure out the latest preseed image and b2g build, but you can also specify specific versions of either or both, either via URLs or local paths.

At the moment it always flashes the preseed.  Do you think it's worth putting in an option to only format and install the b2g build?

It can also prompt for username and password for the pvtbuilds server, or it can use an existing wgetrc file.

This is a patch against mozpool, which seems like the most logical location for this script.  I will link directly to it from one or two relevant wiki pages.

Feedback most welcome.  I have spent too long on this and am getting tunnel vision. :)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 703518 [details] [diff] [review]
Script to fetch and install preseed image and build artifacts

I think this looks fine.  Flashing the preseed is slow and will burn write cycles on the sdcard, so I think the *better* way to do this would be to embed a basic Linux install into a "flashing image".  The image would have five partitions: system, cache, userdata, media, and linux.  Its bootloader would be configured to let the user hit a key to boot to linux, and otherwise boot to the system partition.  The linux partition would contain a script like this one, somewhere that's easy to edit without creating a new squashfs, etc.

Then re-flashing would just require a serial connection to a panda.

Anyway, that's a lot of work, and quite different work from what you've been doing.  What you have is great :)

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 703518 [details] [diff] [review]
Script to fetch and install preseed image and build artifacts

Review of attachment 703518 [details] [diff] [review]:
-----------------------------------------------------------------

Very cool, thanks for writing this! Just a few suggestions.

::: scripts/flash-panda-sd.sh
@@ +2,5 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +#
> +# Flashes a rel-eng-style build to an attached disk.

Should probably mention "consisting of tarball artifacts from a B2G or Android build".

@@ +91,5 @@
> +        echo "${WGET_OUTPUT}"
> +        WGET_OUTPUT=$(wget -nv --directory-prefix=${BUILD_DIR} ${BUILD_ARG}/userdata.tar.bz2 2>&1)
> +        echo "${WGET_OUTPUT}"
> +        WGET_OUTPUT=$(wget -nv --directory-prefix=${BUILD_DIR} ${BUILD_ARG}/boot.tar.bz2 2>&1)
> +        echo "${WGET_OUTPUT}"

I think we mentioned on IRC, but I'd just not redirect wget here and let it spew to stdout.

@@ +166,5 @@
> +echo "Decompressing preseed"
> +bzip2 -d ${PRESEED_DIR}/${PRESEED_ZIP_NAME}
> +
> +echo "Writing to ${DEVICE} (requires root)"
> +sudo dd if=${PRESEED_DIR}/${PRESEED_NAME} of=${DEVICE} bs=1M

You should definitely pipeline these two commands to save disk I/O:
bzip2 -dc ${PRESEED_DIR}/${PRESEED_ZIP_NAME} | sudo dd of=${DEVICE} bs=1M

Comment 4

[William Lachance (:wlach)]

Comment on attachment 703518 [details] [diff] [review]
Script to fetch and install preseed image and build artifacts

I'm pretty happy with this. I've tried it and it works well. Code looks clean too. My main complaint is the constant prompts for sudo while the script is executing. It would be nice if somehow it could just ask for that once on startup and not again. Not sure how difficult that would be to do though. :)

Comment 5

[Mark Côté [:mcote]]

Yeah I think the only way to do that would be to execute the whole script as sudo. Might be possible to bootstrap it somehow. Agreed that it's annoying to leave the script for a while only to return and find it sitting at a sudo prompt.

Comment 6

[William Lachance (:wlach)]

(In reply to Mark Côté ( :mcote ) from comment #5)
> Yeah I think the only way to do that would be to execute the whole script as
> sudo. Might be possible to bootstrap it somehow. Agreed that it's annoying
> to leave the script for a while only to return and find it sitting at a sudo
> prompt.

Maybe you could embed the script inside another one, cat that to a file, then execute that with sudo? A bit ridiculous, but maybe worth it for the convenience factor.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

If you do that, you might as well run the whole thing with sudo..

Comment 8

[Mark Côté [:mcote]]

Yeah, dustin is right.  The easiest way to avoid multiple sudo prompts is to run the whole thing under sudo.  I added a note about that to the usage.  Also addressed ted's comments.

https://github.com/djmitche/mozpool/commit/5b8e9240e6bc4dd4468f2e5a52b1e933e6702359

Comment 9

[Mark Côté [:mcote]]

Basic instructions and link up on mana at https://mana.mozilla.org/wiki/display/IT/Building+and+Installing+B2G+to+a+Panda+Board.  If at some point we can distribute panda images (or at least the preseed) publicly, we can move all this to MDN.