Closed Bug 556099 Opened 13 years ago Closed 12 years ago

update n900 image to new qt-capable image (PR1.2)

Categories

(Release Engineering :: General, defect, P2)

x86
Maemo
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jhford, Assigned: jhford)

References

Details

(Whiteboard: [fennec][talos])

Attachments

(6 files, 2 obsolete files)

We need a new reference image for the N900s in order to run the QT builds on device.  We are currently using:
-RX-51_2009SE_1.2009.41-1.VANILLA_PR_EMMC_MR0_ARM.bin
-RX-51_2009SE_2.2009.51-1_PR_COMBINED_MR0_ARM.bin

This bug is to track creating a new rootfs image.  This should be flashing a new image and installing the existing tools but who knows what else will be changed in the new version of the rom
this depends on getting a QT capable rom that has been signed off as good by mobile team
Priority: -- → P5
Whiteboard: [fennec][talos]
things that need to be fixed:
-install unzip
-use the DHCP provided dns server
We probably also need ntpdate run at boot; getting a lot of "26373226 s in the future" warnings.
Instead of using the dropbear client on device, we should look at using openssh-client.
(In reply to comment #1)
> this depends on getting a QT capable rom that has been signed off as good by
> mobile team

Stuart, DougT: any ETA on this?
OS: Mac OS X → Linux (embedded)
(In reply to comment #5)
> (In reply to comment #1)
> > this depends on getting a QT capable rom that has been signed off as good by
> > mobile team
> 
> Stuart, DougT: any ETA on this?

met with Stuart, DougT: QT release delayed, lets wait another week and see revisit.
Still no word on an updated firmware image
From Stuart:  We are going to wait for the 1.2 released ROM.
The rom has been released:

http://tablets-dev.nokia.com/nokia_N900.php

enter your IMEI number.
We should be using specifically RX-51_2009SE_10.2010.19-1.002_PR_COMBINED_002_ARM.bin (PR 1.2 version 10.2010.19-1 Latest Maemo 5 USA release for Nokia N900)
Doug, 

Can you please confirm that the known good firmware images have the following SHA1 checksums

John


$ openssl sha1 RX-51_2009SE_10.2010.19-1.002_PR_COMBINED_002_ARM.bin RX-51_2009SE_10.2010.13-2.VANILLA_PR_EMMC_MR0_ARM.bin
SHA1(RX-51_2009SE_10.2010.19-1.002_PR_COMBINED_002_ARM.bin)= a4a22b3c92407c293ab197a8acb7480058d52a20
SHA1(RX-51_2009SE_10.2010.13-2.VANILLA_PR_EMMC_MR0_ARM.bin)= a90cd06ef46e3a02d4feb33d4ec0ca190ab0ead4
SHA1(RX-51_2009SE_10.2010.19-1.002_PR_COMBINED_002_ARM.bin)= a4a22b3c92407c293ab197a8acb7480058d52a20

Looks good.
Ok, I will start working on this.

Hopefully the majority of our previous image custom tools will tranfer well to the new firmware image.
Assignee: nobody → jhford
Priority: P5 → P2
John --
It sounds like there's an issue with this rom (/proc/cpuinfo doesn't list thumb).
You should check with blassey about this before rolling 1.2 out -- there's a possibility of a new image coming with that fix.
not supporting thumb shouldn't block rolling out these roms. We don't currently produce thumb code for maemo 5 builds because we need to support the n810s anyway.
What is the time frame for the fixed rom?

What is the time frame for producing thumb builds?
There is no time frame for the "fixing" the rom.  The change to remove thumb support was made on purpose.  Maybe future kernels will support the thumbee.  Maybe not.  Either way, not really our decisions.  However, we really do not understand why anyone would purposely disable thumb mode.

Please, do not block on the thumb issue.

The thumb issue has nothing to do with getting Qt builds.
for more information about the thumb issue (which has nothing to do with this bug), see:

https://bugs.maemo.org/show_bug.cgi?id=10333
(In reply to comment #17)
> There is no time frame for the "fixing" the rom.  The change to remove thumb
> support was made on purpose.  Maybe future kernels will support the thumbee. 
> Maybe not.  Either way, not really our decisions.  However, we really do not
> understand why anyone would purposely disable thumb mode.

I understand :)  I was under the impression from IRC conversations that we wanted thumb and that its removal was unintentional.  I will continue with the work i have begun on the new reference image.

> Please, do not block on the thumb issue.

still not blocked

> The thumb issue has nothing to do with getting Qt builds.

We do have QT builds already but they are not currently tested as this bug blocks the testing. Nightly trunk builds are at http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mobile-trunk-maemo5-qt/
Status: NEW → ASSIGNED
starting on this.  The last version of the former reference image doc is located at

https://wiki.mozilla.org/index.php?title=ReferencePlatforms/Test/N900&oldid=226891
this is a non-trivial project as nokia has changed the boot process.  Previously we were fine to take a snapshot of a running device but we are no longer able to do that as the first boot behaviour has been changed.  Right now the best option (best being relative) would be to unpack the official ubi image, modify files, repack and flash with that.  The problem with this is that I am not sure how we are going to set up things like wifi.

There is information on unpacking the ubifs images here:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_extract
this is also going to require building a custom kernel image with support for ubi
and so i don't loose it,


Comment from IRC about finding wifi settings so we can set them up before firstboot
find it from gconftool -R | grep IAP or so, gconftool --dump /that/gconf/path, inject it somewhere, e.g. with an init / rc script or such with gconftool --load
turns out that Fedora 12 does actually have the ubi/ubifs kernel modules but it still doesn't work

[root@localhost n900-imaging]# dmesg | grep ubifs
UBIFS error (pid 1986): ubifs_get_sb: cannot open "/dev/ubi0_0", error -22
UBIFS error (pid 2002): ubifs_get_sb: cannot open "/dev/ubi0", error -22
Attachment #449767 - Flags: review?(nrthomas)
Comment on attachment 449767 [details] [diff] [review]
imaging tool enhancements

*stamp*
Attachment #449767 - Flags: review?(nrthomas) → review+
Script the image extraction/generation process.
Attachment #452002 - Flags: review?(aki)
also,

doc on creating images:
https://wiki.mozilla.org/ReferencePlatforms/Test/N900-PR1.2

Reference Platform for creating images:
https://wiki.mozilla.org/ReferencePlatforms/mobile-imaging-n900-images


The section to be written for the modify image steps is my task for today.
Comment on attachment 452002 [details] [diff] [review]
more imaging enhancements

*stamp*
Attachment #452002 - Flags: review?(aki) → review+
Because I cannot get a live snapshot of this system, I am going to have to compile a couple more packages than I had thought:

-dropbear
-rsync
-bzip2
-wget
-zip
-unzip
-python
-libpcre3
Because of the new requirements for generating N900 images, we have to install a bunch of applications from source that we didn't have to before.  One of them is our entire python stack, which is a pain to cross-compile

All software has been rebuilt and tested in scratchbox.  The rootfs device has 75mb free even with all of our utilities baked into the firmware image.  This confirms that the change in vol_size=210MiB from 200 does work (this is a ubinize setting).

This image has the following issues:
buildbot zip and unzip weren't baked into the image

dropbear doesn't have a host key and dropbearkey isn't installed.  I am goign to need to install dropbearkey and/or generate hostkey on a different machine and/or generate a working key with the installed openssh-keygen

python based scripts (twistd, hg) have the wrong #! line because the setup.py script is run with the cross-compiled python interpreter which has a path of /home/maemo/build/buildroot/usr/local/bin/python instead of the system path of /usr/local/bin/python.  Will run a script to fix the #! lines programatically after installation if i can't force setup.py to use a specific interpreter location.  All modules are installed and import correctly in the python interpreter


The following custom compiled applications launch:
-tar
-bzip2
-nginx
-xrestop
-cvs
-ntpdate
-dropbear server
-openssh's ssh, scp, ssh-keygen
-wget


I am going to recompile the broken applications, fix the python scripts and generate a new image.
(In reply to comment #31)
> dropbear doesn't have a host key and dropbearkey isn't installed.  I am going
> to need to install dropbearkey and/or generate hostkey on a different machine
> and/or generate a working key with the installed openssh-keygen

It turns out that they were actually installed

> python based scripts (twistd, hg) have the wrong #! line because the setup.py
> script is run with the cross-compiled python interpreter which has a path of
> /home/maemo/build/buildroot/usr/local/bin/python instead of the system path of
> /usr/local/bin/python.  Will run a script to fix the #! lines programatically
> after installation if i can't force setup.py to use a specific interpreter
> location.  All modules are installed and import correctly in the python
> interpreter

The reason buildbot wasn't installed was because I wasn't passing --prefix to the setup.py scripts.  I was using rsync to correct the issue with the DESTDIR make variable for python2.6 (python issue 2233).  Once that patch is applied, and --prefix is given to setup.py it works.

I was able to fix the python script #! lines with

find -exec grep -l "$ROOTDIR" {} \; | xargs sed -i -e "s|$ROOTDIR||"

I am going to generate a new image and test that it is able to run our tests.

I have ensured that zip and unzip are in the bin directory.  I have also changed the prefix i use to /usr as I don't want to change /etc/profile.
I have tested that this new rom is able boot and run buildbot.  I have not been able to get a test run in because HG.m.o is down right now and the device isn't able to clone hg.m.o/build/tools.  I am going to make some final corrections to my imaging doc then move on to the device initialization script and the startup scripts.

There is 65mb of free space on the root partition.
Traceback (most recent call last):
  File "/builds/fennec/xpcshell/runxpcshelltests.py", line 525, in <module>
    main()
  File "/builds/fennec/xpcshell/runxpcshelltests.py", line 521, in main
    if not xpcsh.runTests(args[0], testdirs=args[1:], **options.__dict__):
  File "/builds/fennec/xpcshell/runxpcshelltests.py", line 434, in runTests
    stdout=pStdout, stderr=pStderr, env=self.env, cwd=testdir)
  File "/builds/fennec/xpcshell/runxpcshelltests.py", line 298, in launchProcess
    env=env, cwd=cwd)
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

python /builds/fennec/xpcshell/runxpcshelltests.py --xre-path=/builds/fennec/xulrunner --manifest=/builds/fennec/xpcshell/tests/all-test-dirs.list --test-path=/builds/fennec/xpcshell/tests/test_extensionmanager/xpcshell/test_bug564030.js /builds/fennec/xulrunner/xpcshell


That path (/xulrunner) doesn't exist anymore as far as I understand
Attachment #452838 - Flags: review?(aki)
I can verify that crashtests and reftests do run on the new image.  I am going to test that GTK isn't broken on the new image.  The crashtests and reftests don't look like they run to completion but they are being launched.
Comment on attachment 452838 [details] [diff] [review]
we no longer use a separate xulrunner build.

Joel: does this look like it'll work?
Attachment #452838 - Flags: review?(jmaher)
Attached patch qt branches (obsolete) — Splinter Review
If this page is ok, can you please run:

insert into branches values (NULL,"mobile-qt");
insert into branches values (NULL,"mobile-tracemonkey-qt");
insert into branches values (NULL,"mobile-1.9.2-qt");
insert into branches values (NULL,"mobile-lorentz-qt");
insert into branches values (NULL,"mobile-electrolysis-qt");
insert into branches values (NULL,"mobile-places-qt");
Attachment #452845 - Flags: review?
Attached patch graphs changesSplinter Review
previous patch had a stray line
Attachment #452845 - Attachment is obsolete: true
Attachment #452856 - Flags: review?
Attachment #452845 - Flags: review?
this patch goes along with https://wiki.mozilla.org/ReferencePlatforms/Test/N900-PR1.2

This document is not intended to be the final flashing process document.  This document is to explain how to go from nothing to a working n900 for PR1.2
Attachment #452857 - Flags: review?(aki)
Attachment #452856 - Flags: review? → review?(anodelman)
Comment on attachment 452857 [details] [diff] [review]
root filesystem changes for qt testing

*STAMP*
Attachment #452857 - Flags: review?(aki) → review+
Attachment #452857 - Flags: checked-in? → checked-in+
This image is currently running in staging
Attachment #452856 - Flags: review?(anodelman) → review+
Attached file sql to run!
pleaes run this on staging and production graph server
Attachment #452921 - Attachment mime type: application/octet-stream → text/plain
Comment on attachment 452838 [details] [diff] [review]
we no longer use a separate xulrunner build.

can't hurt? as jhford pointed out, we're already busted on all of these.
Attachment #452838 - Attachment description: we no longer use a seperate xulrunner build. → we no longer use a separate xulrunner build.
Attachment #452838 - Flags: review?(aki) → review+
Comment on attachment 452838 [details] [diff] [review]
we no longer use a separate xulrunner build.

Joel,  I think this is a big part of why xpcshell tests are failing right now.  I have pushed to the repository because /builds/fennec/xulrunner doesn't exist anymore on our slaves.  If this is incorrect please let me know :)
Depends on: 573650
Comment on attachment 452838 [details] [diff] [review]
we no longer use a separate xulrunner build.

hmm, my understanding is xre path should be more like this:
xre-path = /builds/fennec

not like the appname:
xre-path = /builds/fennec/fennec

If you have ran with this above change to the cfg file, then I am wrong.


In reading over this bug, I see some information regarding compiling python from scratch.  Have we tested unittests and talos?  I know python on the n810 had some limitations and I don't if this custom compiled version would be better or worse.  We also have remote testing options!!!!
Attachment #452838 - Flags: review?(jmaher) → review-
Summary: update n900 image to new qt-capable rom → update n900 image to new qt-capable image (PR1.2)
(In reply to comment #46)
> (From update of attachment 452838 [details] [diff] [review])
> hmm, my understanding is xre path should be more like this:
> xre-path = /builds/fennec
> 
> not like the appname:
> xre-path = /builds/fennec/fennec
> 
> If you have ran with this above change to the cfg file, then I am wrong.

You are right.

> 
> 
> In reading over this bug, I see some information regarding compiling python
> from scratch.  

The python versions that are distributed as a binary by maemo.org are totally messed up.  This version is more solid from what I can tell.

> Have we tested unittests and talos?  I know python on the n810
> had some limitations and I don't if this custom compiled version would be
> better or worse.  

So far, it is no worse.

> We also have remote testing options!!!!

This seems to be working apart from this issue to xre-path.

I will upload a new patch
Attachment #452838 - Attachment is obsolete: true
Attachment #453183 - Flags: review?(jmaher)
Comment on attachment 453183 [details] [diff] [review]
use a better xre path

this looks good, thanks!
Attachment #453183 - Flags: review?(jmaher) → review+
These devices are in production.  Marking as fixed, if there are any further problems with these phones please file a new bug.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.