802317 - Need Gaia UI tests in buildbot automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[cmtalbert]

There are two new B2G emulator based frameworks that we need to turn on for Ash and Cedar (and aurora and central too). These are the Gaia "Integration" and Gaia "Web QA" tests. Both are Gaia UI automation tests. The Integration tests work using JavaScript, the Web QA ones work via Python.  They are complimentary at this point and both should be run.

Current items needing resolution to getting these to run (feel free to file dependent bugs here):
* change the releng mirror of the gaia tree to the "nightly" branch.
* ensure we can perform a build step of gaia on a test slave (looks like any linux test slave can do this) steps are here: https://github.com/mozilla-b2g/gaia/blob/master/README.md
** We can probably perform the "build" step of gaia (because it merely moves directories around and puts things in their expected places) in the mozharness script for running these test frameworks.
* Create mozharness script to run the two frameworks
* Deploy buildbot changes to run the mozharness script.

We need releng to verify that the build step of gaia can be completed on a test slave and that the Gaia nightly branch is what is being mirrored to the releng Gaia mirror. This should be completed by the end of the week if we're to make our targets.

We hope to deploy at least one of these two frameworks (if not both) to cedar/ash by November 6.

Comment 1

[cmtalbert]

Catlee, both these frameworks are going to depend on having panda builds available with gaia as we talked about yesterday.

We've discovered that we can't run these tests in the emulator builds because the emulators are too slow, so we need to run these on pandas.  Catlee is there a bug for fixing the gaia story on pandas that can block this bug, or should I file it?

Comment 2

[Chris AtLee [:catlee]]

Bug 807373 handles getting up-to-date gaia mirrored inside mozilla.

Comment 3

[cmtalbert]

And continuing on from comment 1, I'm adding in the dependent bugs so that we properly block based on the availability of a flashing mechanism so we can reliably flash the panda boards to new builds so that we can run gaia tests on the pandaboards.

Comment 4

[Chris AtLee [:catlee]]

Ok, it sounds like we need at least 2 different things here:

* "Up-to-date" gaia in panda builds so we can run these tests when gecko changes. I recommend pinning the gaia version used here to a fixed revision or maybe to the nightly branch. The purpose of these tests is to make sure that gecko changes don't break gaia.

* per-gaia push, run tests on pre-existing panda builds. The purpose of this is to make sure that gaia changes work with a known good panda build.

Comment 5

[Jonathan Griffin (:jgriffin)]

(In reply to Chris AtLee [:catlee] from comment #4)
> Ok, it sounds like we need at least 2 different things here:
> 
> * "Up-to-date" gaia in panda builds so we can run these tests when gecko
> changes. I recommend pinning the gaia version used here to a fixed revision
> or maybe to the nightly branch. The purpose of these tests is to make sure
> that gecko changes don't break gaia.

We'll probably want to use the nightly branch.  There are too many interactions between gaia and gecko to pin gaia to a known good version.

> 
> * per-gaia push, run tests on pre-existing panda builds. The purpose of this
> is to make sure that gaia changes work with a known good panda build.

Yes.  This means we'll also need to figure out which panda build to use; the most recent?  Or ...?

Comment 6

[Chris AtLee [:catlee]]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> (In reply to Chris AtLee [:catlee] from comment #4)
> > Ok, it sounds like we need at least 2 different things here:
> > 
> > * "Up-to-date" gaia in panda builds so we can run these tests when gecko
> > changes. I recommend pinning the gaia version used here to a fixed revision
> > or maybe to the nightly branch. The purpose of these tests is to make sure
> > that gecko changes don't break gaia.
> 
> We'll probably want to use the nightly branch.  There are too many
> interactions between gaia and gecko to pin gaia to a known good version.

Ok, let's use nightly.

> 
> > 
> > * per-gaia push, run tests on pre-existing panda builds. The purpose of this
> > is to make sure that gaia changes work with a known good panda build.
> 
> Yes.  This means we'll also need to figure out which panda build to use; the
> most recent?  Or ...?

Latest nightly? (there currently aren't panda nightlies...) Manually chosen "good" build?

Comment 7

[Jonathan Griffin (:jgriffin)]

I think latest nightly is as good as anything.

Comment 8

[cmtalbert]

updating summary since these are called the "smoketests" generally.

Comment 9

[cmtalbert]

Sorry for the spam. I'm going to split the Gaia Integration tests into their own bug because the dependencies for these two are different. While much of the setup work for gaia integration is going to be necessary for the gaia smoketests as well (i.e. all the stuff related to flashing/finding/scheduling pandas with buildbot will be shared by both these test suites) but some of the test harness specific dependencies may wind up being different. For example, there is now evidence that the integration tests may be able to run in emulators whereas the smoketests cannot.

Comment 10

[Aki Sasaki (not active)]

Armen will be driving this.

Comment 11

[Armen [:armenzg]]

For now, I'm trying to figure out if I can run locally a simple test.
I'm trying to figure out how to image my panda board.
By either creating my own build or determine how to flash it from something in:
https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-panda/20121120102534

mdas says that she was expecting .img files.

######################
http://hg.mozilla.org/mozilla-central/file/a9d59073400b/testing/marionette/client/marionette/tests/unit/test_simpletest_pass.js

The gaia tests are just special Marionette tests, and I don't think we should worry about how well they work on the panda right now; they are pretty unstable right now anyway, and it's not a stretch to think they may need some work to get running on a panda.  Instead, you can just work with the simplest Marionette test, which I've confirmed does work locally.

This test is in m-c, under testing/marionette/client/marionette/tests/unit/test_simpletest_pass.js.

To run this yourself, after flashing a panda build:

adb forward tcp:2828 tcp:2828
cd testing/marionette/client
python setup.py develop
cd marionette
python runtests.py --address localhost:2828 tests/unit/test_simpletest_pass.js

Comment 12

[Armen [:armenzg]]

As a status of this bug:
* bug 810045 prevents us from being able to image a b2g panda
* bug 800138 prevents running the test remotely on the colo

My plan is:
1) get things running locally while the other bugs unblock

A) as soon as we can image, I will try to add fake jobs that all they do is test the imaging process (this depends on the mozpool support - bug 810439)
A.1) if mozpool is not there, I will ask a chassis for myself and start with a fixed image

B) as soon as we can run the tests remotely, I will try to make my board to be "remote" rather than "local" and setup my machine to talk to it through a foopy


########################
For 1)

It seems that I need a Linux OS to follow any imaging process.
I'm downloading Ubuntu 12.10 64-bit.

I will be following the steps from:
https://mana.mozilla.org/wiki/display/IT/Building+and+Installing+B2G+to+a+Panda+Board

I have an 8GB SD card (not a 16GB one).


It seems that copying the artifacts is all that it is needed:
- Format system, cache, userdata and media partitions
    sudo mkfs -t ext4 -L "System" /dev/mmcblk0p2
    sudo mkfs -t ext4 -L "Cache" /dev/mmcblk0p3
    sudo mkfs -t ext4 -L "Userdata" /dev/mmcblk0p5
    sudo mkfs -t ext4 -L "Media" /dev/mmcblk0p6

- Mount SD card partitions
    mkdir -p /tmp/boot /tmp/system /tmp/userdata
    sudo mount -t fat /dev/mmcblk0p1 /tmp/boot
    sudo mount -t ext4 /dev/mmcblk0p2 /tmp/system
    sudo mount -t ext4 /dev/mmcblk0p5 /tmp/userdata

- Untar artifacts to partitions
    sudo tar -jxpsf system.tar.bz2 -C /tmp/system --strip=1 system
    sudo tar -jxpsf userdata.tar.bz2 -C /tmp/userdata --strip=1 data

- Copy uImage and uInitrd to Boot partition
    sudo cp uImage uInitrd /tmp/boot

- Unmount sdcard
    sudo umount /tmp/boot /tmp/system /tmp/userdata
    Put SD card in Panda Board and BOOT IT

Comment 13

[Armen [:armenzg]]

On another note, I was trying to build b2g for pandas and I got to here:
sed: 1: "/^INSERT$/ { :l; n; p;  ...": unexpected EOF (pending }'s)
  GEN     util-linux/volume_id/Kbuild
target thumb C: libnl_2 <= system/core/libnl_2/genl/family.c
  HOSTCC  scripts/kconfig/conf.o
target thumb C: libnl_2 <= system/core/libnl_2/handlers.c
  HOSTCC  scripts/kconfig/kxgettext.o
target thumb C: libnl_2 <= system/core/libnl_2/msg.c
  HOSTCC  scripts/kconfig/mconf.o
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c: In function ‘exec_conf’:
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c:469: error: ‘SIGWINCH’ undeclared (first use in this function)
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c:469: error: (Each undeclared identifier is reported only once
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c:469: error: for each function it appears in.)
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c: In function ‘main’:
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c:1051: warning: statement with no effect
/Users/armenzg/repos/B2G/external/busybox/scripts/kconfig/mconf.c:1052: warning: statement with no effect
make[3]: *** [scripts/kconfig/mconf.o] Error 1
make[2]: *** [android_defconfig] Error 2
make[1]: *** [android_defconfig] Error 2
make: *** [out/target/product/panda/obj/EXECUTABLES/busybox_intermediates//.config] Error 2
make: *** Waiting for unfinished jobs....

real    11m28.109s
user    17m51.913s
sys     2m20.579s

> Build failed! <

Build with |./build.sh -j1| for better messages
If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.

Comment 14

[Jonathan Griffin (:jgriffin)]

Are you building on a mac?  See https://github.com/mozilla-b2g/android-device-panda/issues/1

Comment 15

[Armen [:armenzg]]

(In reply to Jonathan Griffin (:jgriffin) from comment #14)
> Are you building on a mac?  See
> https://github.com/mozilla-b2g/android-device-panda/issues/1

Yep!
I will move it to Ubuntu if I can't get it to work the "copy artifacts" method.

Am I going to be able to make it work by following the steps on comment 12?
I just want to know if someone has done it successfully since we switched away from img files to the tar balls.

Comment 16

[Jonathan Griffin (:jgriffin)]

FWIW, one of the environments I build in is an Ubuntu 11.10 VM on my Mac using VMWare Fusion, and it works well.

I have not "flashed" a panda build using the steps in comment 12.

Comment 17

[Armen [:armenzg]]

Change of plans:
ted: armenzg: there's not a good way to get those tarballs onto the pandaboard right now except for using BMM
12:05 armenzg: ted: what about if I build b2g locally?
12:05 ted: then it's fairly straightforward to flash it

I'm looking at https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/Pandaboard

I'm having build problems.
hub pointed me to https://github.com/mozilla-b2g/B2G/issues/33

My steps:
https://etherpad.mozilla.org/install-ubuntu-on-OSX-and-image-a-panda

I'm tried to downgrading to gcc 4.6 but I have not been able to:
sudo apt-get install gcc=4.4.6
I thought ^ that is the way. I'm currently on 4.7.2

I'm using now his local patch and it seems to be moving forward:
diff --git a/include/utils/KeyedVector.h b/include/utils/KeyedVector.h
index 6bcdea4..5e62755 100644
--- a/include/utils/KeyedVector.h
+++ b/include/utils/KeyedVector.h
@@ -190,7 +190,7 @@ DefaultKeyedVector<KEY,VALUE>::DefaultKeyedVector(const VALUE& defValue)

 template<typename KEY, typename VALUE> inline
 const VALUE& DefaultKeyedVector<KEY,VALUE>::valueFor(const KEY& key) const {
-    ssize_t i = indexOfKey(key);
+    ssize_t i = this->indexOfKey(key);
     return i >= 0 ? KeyedVector<KEY,VALUE>::valueAt(i) : mDefault;
 }

I wish I could setup things the same way as with our panda builds and not have to use a local patch:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17274239&tree=Firefox&full=1
03:37:46 INFO -  Package gcc-4.4.6-3.el6.i686 already installed and latest version

Comment 18

[Armen [:armenzg]]

Should the milestone of this bug be "B2G C2 (20nov-10dec)"?

Comment 19

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] from comment #18)
> Should the milestone of this bug be "B2G C2 (20nov-10dec)"?

Yes.
https://bugzilla.mozilla.org/show_bug.cgi?id=810439#c21

Comment 20

[Armen [:armenzg]]

I'm going to try to downgrade:
sudo apt-get install gcc-4.6 g++-4.6 g++-4.6-multilib


####################################
(if [ -d ./.git ]; then \
          git log -1 --format="%H%n%at" HEAD > apps/settings/resources/gaia_commit.txt; \
        else \
          echo 'Unknown Git commit; build date shown here.' > apps/settings/resources/gaia_commit.txt; \
          date +%s >> apps/settings/resources/gaia_commit.txt; \
        fi)
Packaged webapps
run-js-command  webapp-zip
Done
Generating extensions...
Done
python build/settings.py --console --homescreen app://homescreen.gaiamobile.org/manifest.webapp --ftu app://communications.gaiamobile.org/manifest.webapp --wallpaper build/wallpaper.jpg --output profile/settings.json
Profile Ready: please run [b2g|firefox] -profile /home/armenzg/repos/B2G/gaia/profile
make[1]: Leaving directory `/home/armenzg/repos/B2G/gaia'
cd /home/armenzg/repos/B2G/gaia/profile && tar cfz /home/armenzg/repos/B2G/gaia/profile.tar.gz webapps

real    25m13.280s
user    18m57.995s
sys     2m32.898s

> Build failed! <

Build with |./build.sh -j1| for better messages
If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.

Comment 21

[Armen [:armenzg]]

and this:
echo "export CC=gcc-4.6 CXX=g++-4.6" >> ~/.userconfig

Comment 22

[Armen [:armenzg]]

I'm not being able to build.

I'm going to try to use mozharness.
scripts/scripts/b2g_build.py --no-clobber --target panda --config b2g/releng.py

I have tried deploying hgtool.py but I get the following:
armenzg@armenzg-VirtualBox:~/repos$ hgtool.py
Traceback (most recent call last):
  File "/usr/local/bin/hgtool.py", line 13, in <module>
    from util.hg import mercurial, out, remove_path
ImportError: No module named util.hg

I don't see PYTHONPATH setup on the tbpl log.

Comment 23

[Aki Sasaki (not active)]

You need the special hgtool with all the required libs base64 encoded at the top of the file, like http://people.mozilla.org/~catlee/hgtool.py

Comment 24

[Armen [:armenzg]]

Trying this now:
sudo wget -O /usr/local/bin/hgtool.py http://people.mozilla.org/~catlee/hgtool.py
sudo chmod 755 /usr/local/bin/hgtool.py 
sudo mkdir /builds
sudo chown armenzg /builds
scripts/scripts/b2g_build.py --no-clobber --target panda --config b2g/releng.py

Let's see how far I get.

Comment 25

[Armen [:armenzg]]

Attachment: wip - add b2g-panda platform with gaia-mochitest

Change of plans: I will be focusing on the buildbot code changes where I know I will make progress.

I chatted with aki and he will try to do the builds with his unagi and see where he can get to.

Beware: The patch is very raw and does nothing as it is.

Comment 26

[Ed Morley [:emorley]]

Would it be ok if we s/b2g-panda/b2g_panda/g to be consistent with the other panda_android and also the panda builds?

Comment 27

[Armen [:armenzg]]

Hey Ed! Sure I can do that.

On a low priority, I tried running the mozharness script over the weekend and I got to the point that I have to setup the BuildVPN on my VM or something else since my host being connected to the BuildVPN is not sufficient. Any alternative to setting up the BuildVPN on my VM?

* Add this to ~/.hgrc
[extensions]
share =
############################
sudo mkdir /tools
sudo chown armenzg /tools
wget -O /tools/tooltool.py http://mxr.mozilla.org/build/source/puppet/modules/packages/templates/tooltool.py?raw=1
chmod 755 /tools/tooltool.py
* Replace in /tools/tooltool.py for this:
#! <%= python %>
for
#!/usr/bin/python

############################
armenzg@armenzg-VirtualBox:~/repos/build$ /tools/tooltool.py --url http://runtime-binaries.pvt.build.mozilla.org/tooltool/ fetch -m /home/armenzg/repos/build/gecko/b2g/config/panda/releng-pandaboard.tt -o
ERROR - failed to fetch 'gonk.tar.xz': <urlopen error [Errno -2] Name or service not known>
ERROR - The following files failed: 'gonk.tar.xz'

Comment 28

[Aki Sasaki (not active)]

I think you can pass --gonk-snapshot with a different url.
You can't put gonk.tar.xz anywhere public aiui, but you can potentially put it on localhost somewhere.  (Not sure if tooltool.py will allow a file:// url, but if you have a simple webserver that would definitely work.)

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 684791 [details] [diff] [review]
wip - add b2g-panda platform with gaia-mochitest

This should go in mozilla-tests/b2g_config.py .

Comment 30

[Armen [:armenzg]]

Attachment: wip - add b2g-panda platform with gaia-mochitest (v1)

I have addressed the comments with this bug.

I'm now working on a dummy mozharness script until the other pieces are ready.
I will be able to post it tomorrow.

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 685282 [details] [diff] [review]
wip - add b2g-panda platform with gaia-mochitest (v1)

I think it's valid to have a nonexistent reboot_command (or a reboot_command of None), though the ls is harmless.

Comment 32

[Armen [:armenzg]]

I have written a simple script and config but they don't do anything useful:
http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness/file/default/configs/b2g/panda_releng.py
http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness/file/default/scripts/b2g_panda.py

I wonder what I could do next to get moving in advance.

Maybe I could:
A) get my current patches reviewed and enabled on Cedar even if they do nothing
** I could have one/two buildbot slaves enabled on one of the foopies (w/o a panda)
B) figure out what is the range of the pandas we're going to use and setup the foopies
** blocked on having actual devices up (since start_cp.sh does some verification with them)
C) keep on hammering the local build until I have a working local panda
** try to run a simple test locally through USB/ADB

Once we have the mozpool library ready:
* we can have it running on Cedar even if the b2g images don't yet get networking

For now I will lean towards C until I hear other suggestion or any of the blocking bugs get cleared up.

Comment 33

[Armen [:armenzg]]

I've placed gonk in my local server and adjusted the scripts to grab it from there.
Let's see how it goes.

I'm going to be looking at the code of jhopkins as well as looking if we should start things with clientproxy.py or an alternative.

Comment 34

[Armen [:armenzg]]

I got this:
17:38:38     INFO - #####
17:38:38     INFO - ##### Running build step.
17:38:38     INFO - #####
17:38:38     INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-i386', '--init']
17:38:38     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 --init
17:38:38    FATAL - caught OS error 2: No such file or directory while running ['mock_mozilla', '-r', 'mozilla-centos6-i386', '--init']
17:38:38    FATAL - Exiting -1

Comment 35

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] from comment #34)
> 17:38:38    FATAL - caught OS error 2: No such file or directory while
> running ['mock_mozilla', '-r', 'mozilla-centos6-i386', '--init']
> 17:38:38    FATAL - Exiting -1

This probably means mock_mozilla is not in your PATH.
You need to either set 'exes': {'mock_mozilla': '/path/to/mock_mozilla'} in your config file, or add mock_mozilla to your PATH, or something of the like.

It's on my TODO list to improve this error message =\

Comment 36

[Armen [:armenzg]]

I don't have mock_mozilla in my VM and I have not found a way to build it.
catlee suggested to copy build/gecko/b2g/config/panda/config.json somewhere else and remove the mock lines.
I then called the script with --gecko-config /path/to/new.config.json

I will be reading the mozpool/mozharness code and sync up with jhopkins.

Comment 37

[Armen [:armenzg]]

jhopkins and I have done quite some progress today and IT has placed a staging pool for us in perfect timing.

We have managed to talk with bmm through mozharness.

We're now looking into imaging pandas with the image in http://people.mozilla.com/~jwatkins/bmm/artifacts/b2g-2012112700

I'm going to be looking into setting up foopies to pandas.
I will be pairing devices from SCL1 - 201-1 - SCL1 in https://releng.etherpad.mozilla.org/foopy-pandas-mapping

Comment 38

[Armen [:armenzg]]

Status update:
* bug 800138 - "cannot run gaia smoketests remotely"
** jgriffin says that a fix might land between today and tomorrow
* bug 810045 - "b2g image does not have working network"
** dividehex has done a pull request in https://github.com/mozilla-b2g/android-device-panda/pull/12 that is still being discussed
* bug 810439 - "mozpool support"
** jhopkins and I have made a lot of progress and can request devices through mozharness
* bug 816237 - "setup staging pool of pandas"
** I'm making lots of progress
** hopefully between today and tomorrow I will have everything set up and have automated ways for future racks
* bug 816345 - "stop buildbot when bad image" - might not be a blocker since we have the same problem with tegras. It might be easy to fix though

WORRIES:
* no one has run gaia smoketests on a panda board
* releng to still test dividehex's image (it should be done today)

Comment 39

[Aki Sasaki (not active)]

The new image comes up networked with SUT (takes a little while for SUT to come up).
Once in, I'm able to

cat /system/b2g/application.ini

which would hopefully give us something to verify.  I'm going to guess comparing the Version+BuildID would be easiest:

[App]
Vendor=Mozilla
Name=B2G
Version=19.0a1
BuildID=20121108162613
ID={3c2e2abc-06d4-11e1-ac3b-374f68613e61}

[Gecko]
MinVersion=19.0a1
MaxVersion=19.0a1

[XRE]
EnableExtensionManager=1

[Crash Reporter]
ServerURL=https://crash-reports.mozilla.com/submit?id={3c2e2abc-06d4-11e1-ac3b-374f68613e61}&version=19.0a1&buildid=20121108162613

Comment 40

[Armen [:armenzg]]

Tomorrow I will be hacking the panda builds to produce sendchanges for the gaia mochitest builders (which have landed but are not yet in production - note that those jobs would not do anything useful until mozharness changes would land).

It seems that I would have to make the panda b2g builds trigger them:
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_cedar_panda

aki, do we have any mozharness jobs that have sendchanges already being used?
I thought the gecko builds were one of them:
http://buildbot-master35.build.mozilla.org:8001/builders/b2g_cedar_ics_armv7a_gecko%20build/builds/96

Comment 41

[Aki Sasaki (not active)]

Hm, looks like we don't.

We have a self.info("Upload successful: %s" % upload_url) at the end of b2g_build.py.  We may have to either add sendchange masters to the configs/b2g/releng.py config file or somehow send them down in the properties file.

Then we can

# this may be self.config['sendchange_masters'] or self.buildbot_config['sendchange_masters'] depending on whether we get this from commandline/configfile or buildbot properties

if self.config.get("sendchange_masters"):
    buildbot = self.query_exe("buildbot", return_type="list")
    self.run_command(buildbot + ['sendchange', ...], error_list=BuildbotErrorList)

We probably have to create the BuildbotErrorList.

Comment 42

[Justin Wood (:Callek)]

Attachment: foo.py

So this is the py script to get the contents of the ini as described in https://bugzilla.mozilla.org/show_bug.cgi?id=802317#c39 as requested.

It is ugly, but it works :-) I went with simplistic rather than try and setup the logging module and to move forward on this easier. We can adjust anything as needed.

Basic assumption:
* We pass 'panda-NNNN' as a --device (passing 'panda-NNNN.pN.releng.scl1.mozilla.com' should also be just fine)
* We want to print the raw file contents and mozharness will just parse this output
* We don't need to push the error messages to stderr
* The default permissions on the ini are readable by the SUTAgent process [if not I can switch to manual commands + exec, but I'll find out when I test this later]
* someone else decides what the final .py name is and checks it into whatever repo.

Comment 43

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #42)

> Basic assumption:

One more:
* SUTAgent is already running on the device when we start this script. (I can implement retry for the connection, over [x] timeframe, in this script itself - if needed)

Comment 44

[Armen [:armenzg]]

Comment on attachment 688166 [details]
foo.py

We probably would be using this inside of the mozharness script.
Nevertheless, there are rumors that gaia tests can now be run (at least locally).
We need to sync up with the A-team and see if we can run them remotely.

Comment 45

[Justin Wood (:Callek)]

Comment on attachment 688166 [details]
foo.py

>    parser.add_option("-d", "--devie", type="string", dest="device",
>                      help="Device to connect to")

Typo --device  fixed locally

Also test passed!

[cltbld@foopy45 ~]$ PYTHONPATH=/builds/sut_tools python foo.py --device panda-0160
Connecting to: panda-0160
reconnecting socket
Read of file (/system/b2g/application.ini) follows
===========================
[App]
Vendor=Mozilla
Name=B2G
Version=19.0a1
BuildID=20121108162613
ID={3c2e2abc-06d4-11e1-ac3b-374f68613e61}

[Gecko]
MinVersion=19.0a1
MaxVersion=19.0a1

[XRE]
EnableExtensionManager=1

[Crash Reporter]
ServerURL=https://crash-reports.mozilla.com/submit?id={3c2e2abc-06d4-11e1-ac3b-374f68613e61}&version=19.0a1&build
id=20121108162613
[cltbld@foopy45 ~]$

Comment 46

[Aki Sasaki (not active)]

Comment on attachment 688166 [details]
foo.py

Ty.
We should be able to parse and retry in mozharness, so this works.

Comment 47

[Armen [:armenzg]]

Attachment: add b2g_panda to buildbot-master29

Comment 48

[Armen [:armenzg]]

Attachment: [wip] add sendchanges to the panda builds

Any ideas on how to limit this to the cedar branch?

This allowed me to generate a sendchange like this:
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange-unittest --branch cedar-b2g_panda-opt-unittest --revision cfdaace0736e2b07009dcd477efd8ee1001bca3a --who [e-mail] --comments "merge m-c -> cedar" http://pvtbuilds2.dmz.scl3.mozilla.com//pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121204112802

Comment 49

[Aki Sasaki (not active)]

Comment on attachment 688350 [details] [diff] [review]
[wip] add sendchanges to the panda builds

This is fine for staging purposes, but shouldn't land as-is.

For production, we should

a) not have dev-master01 in there
b) maybe wrap the entire b2g_build chunk in an |if self.config.get('sendchange_masters'):| block
c) I think you can get the 'cedar' branch in the sendchange branch from the buildbot_config.
d) we may want an error_list for the run_command() to detect sendchange errors.  Or we can look at the return value from run_command() to see if it exited 0 or not.

> Any ideas on how to limit this to the cedar branch?

If you're sendchanging to the right branch ('cedar-b2g_panda-opt-unittest' isn't hardcoded, and we get the 'cedar' part of that string from the buildbot_config) then we don't have to limit it.  If we sendchange for mozilla-central-b2g_panda-opt-unittest and there are no panda test builders for m-c, then nothing happens.

Comment 50

[Armen [:armenzg]]

I was having trouble with slavealloc since we did not have the password on slave_passwords table [1]. I did the following to fix:

insert into distros value(null, 'panda');
update slaves set distroid=23  where name='panda-0169';
insert into slave_passwords value(29,23,'passwd'); # real password excluded

update slaves set distroid=23 where name like 'panda-%' and distroid=13;
Query OK, 859 rows affected (0.06 sec)
Rows matched: 859  Changed: 859  Warnings: 0

Later on we might want to change things to discriminate between panda_b2g, panda_android or tegra. For now, only one of the three is getting setup to use slavealloc.

The first job has been run on production:
http://buildbot-master29.build.scl1.mozilla.com:8201/builders/b2g_b2g_panda%20cedar%20opt%20test%20gaia-mochitest

pandas-016[0-9] are now connected to buildbot-master29.

[1] http://mxr.mozilla.org/build/source/tools/lib/python/slavealloc/data/queries.py#110

Comment 51

[Armen [:armenzg]]

Attachment: add sendchanges to the panda builds

Comment 52

[Aki Sasaki (not active)]

Comment on attachment 688463 [details] [diff] [review]
add sendchanges to the panda builds

In order of importance:

* You may have to add a dummy zipfile to the sendchange until we have a gaia smoketests zip.

The code that looks at the buildbot_config is here:
http://hg.mozilla.org/build/mozharness/file/6f2b9bf381f6/mozharness/mozilla/testing/testbase.py#l91

We can stop expecting the test zip in the sendchange by turning this pref to False:
http://hg.mozilla.org/build/mozharness/file/6f2b9bf381f6/scripts/b2g_panda.py#l70

* If buildbot isn't in PATH, the fix is to change the config:

     "exes": {
         "tooltool.py": "/tools/tooltool.py",
     },

to add
        "buildbot": "/path/to/buildbot",

we can also use the format
        "buildbot": ["/path/to/python", "/path/to/buildbot"],

if we have to specify a specific python to call buildbot with.

* I'm not a big fan of the if self.config["target"] == "panda" but this may be the fastest way to success here.

Comment 53

[Armen [:armenzg]]

Attachment: wip - b2g_panda.py - start re-imaging boards

This patch starts getting the re-imaging working.

Instead of trying to image from pvtbuilds I decided to hard-code to dev-stage (where I downloaded jwatkin's working image) until we can download from pvtbuilds directly.
http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/b2g_temporary/b2g-2012112900/

Feel free to not let me land this until I have this 204 error code resolved.

BTW, what is this 204 error code?
"17:25:38    ERROR - Bad return status from http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/return/: 204!"
This happens after closing a request:
http://hg.mozilla.org/build/mozharness/file/tip/mozharness/mozilla/testing/mozpool.py#l308

2012-12-04T17:22:59 syslog Imaging complete. Rebooting
2012-12-04T17:25:00 statemachine entering state b2g_pinging
2012-12-04T17:25:00 bmm ping of panda-0093.p1.releng.scl1.mozilla.com complete: ok
2012-12-04T17:25:00 statemachine entering state ready
2012-12-04T17:25:38 statemachine entering state free

17:23:38     INFO - Status code: 200
17:23:38     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
^@17:24:38     INFO - Request GET http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/status/...
17:24:38     INFO - Status code: 200
17:24:38     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
^@17:25:38     INFO - Request GET http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/status/...
17:25:38     INFO - Status code: 200
17:25:38     INFO - #####
17:25:38     INFO - ##### Running run-test step.
17:25:38     INFO - #####
17:25:38     INFO - #####
17:25:38     INFO - ##### Running close-request step.
17:25:38     INFO - #####
17:25:38     INFO - Request POST http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/return/...
17:25:38    ERROR - Bad return status from http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/return/: 204!
Request 'http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/145/' deleted on cleanup

Comment 54

[Armen [:armenzg]]

The panda b2g builds are trying to run buildbot but buildbot does not exist on mock slaves:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17611817&tree=Cedar&full=1
17:53:48     INFO - Running command: ['buildbot', 'sendchange', ..
17:53:48     INFO - Copy/paste: buildbot sendchange --master buildbot-master36 ...
17:53:48    ERROR - caught OS error 2: No such file or directory while running ['buildbot', ...

This does not cause the panda builds to fail (expected).

I will fix this tomorrow.

Comment 55

[Aki Sasaki (not active)]

If buildbot does exist on the slaves, but isn't in PATH, you can fix with the exes fix in comment 52.

Comment 56

[Armen [:armenzg]]

Attachment: add buildbot path to panda builds

Weehee!
I triggered the first job on tbpl:
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_b2g_panda%20cedar%20opt%20test%20gaia-mochitest&noignore=1

I also landed this change to fix the sendchanges with r=bustage.

Comment 57

[Chris AtLee [:catlee]]

I think specifying this exact version-specific path to buildbot is fragile...Can we find a better way to find buildbot?

Comment 58

[Justin Wood (:Callek)]

(In reply to Chris AtLee [:catlee] from comment #57)
> I think specifying this exact version-specific path to buildbot is
> fragile...Can we find a better way to find buildbot?

Good point!

/tools/buildbot --> /tools/buildbot-0.8.4-pre-moz2 symlink exists, lets just use the generic loc.

rs+=me for that switch if you [armen] agree!

Comment 59

[Armen [:armenzg]]

Fixed buildbot path in:
http://hg.mozilla.org/build/mozharness/rev/fc1dca7f63cf

Comment 60

[JP Rosevear [:jpr]]

This doesn't look like it technically blocks, but is very, very useful - please ask for uplift approval asap.

Comment 61

[Armen [:armenzg]]

Attachment: fix naming

Comment 62

[Armen [:armenzg]]

Attachment: improvements to manage_buildslave.sh

This improves usage of manage_buildslave.sh.
Before, we required needing to get inside of the slaves' directory.

It will now only be symlinked from /builds.

Comment 63

[Justin Wood (:Callek)]

Comment on attachment 688881 [details] [diff] [review]
improvements to manage_buildslave.sh

Review of attachment 688881 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/mobile/manage_buildslave.sh
@@ +16,5 @@
> +    fi
> +    DEVICE=$2
> +    DEVICE_PATH=/builds/${DEVICE}
> +    if [ ! -d "$DEVICE_PATH" ]; then 
> +        echo $DEVICE_PATH does not exist. Try again with the right device name.

nit EOL whitespace on the if line
bikeshed-nit: english language potential confusion, change to "Try again with the correct device name."

@@ +21,5 @@
> +        exit 1
> +    fi
> +fi
> +
> +if [ "$opt" = "restart" ]; then

for historic sake -- the use case of this is for humans (for now)

@@ +22,5 @@
> +    fi
> +fi
> +
> +if [ "$opt" = "restart" ]; then
> +    set -ex

nit please either use this at the start of the script, or reset it to what we would normally expect when your done with the need to change.

@@ +34,4 @@
>  elif [ "$opt" = "start" ]; then
>      echo "We want to always start buildbot through twistd"
>      echo "We will run with the twistd command instead of calling buildslave"
> +    cd $DEVICE_PATH

nit pushd since then we can popd instead of `cd -` later (and even run from ~ if we call the script with an abspath)

@@ +38,5 @@
>      ${BB_PYTHON} ${BB_TWISTD} --no_save \
> +        --rundir=$DEVICE_PATH \
> +        --pidfile=$DEVICE_PATH/twistd.pid \
> +        --python=$DEVICE_PATH/buildbot.tac
> +    tail $DEVICE_PATH/twistd.log

possible simple improvement I just thought of -- pause 1 before the tail, so that we give twistd/buildbot a slight bit of time to start :-)

@@ +44,2 @@
>  else
> +    ${BB_PATH} $opt $DEVICE_PATH

Q: does *every* buildbot command [that we care about supporting] take the slavepath as the 2'nd arg?

Comment 64

[Armen [:armenzg]]

Attachment: improvements to manage_buildslave.sh

I've removed the case to call buildslave and pass a parameter.

Comment 65

[Armen [:armenzg]]

Attachment: check_b2g_application_ini.diff

I managed to call this from mozharness with self.runcommand().
I can't place it on external scripts.

Comment 66

[Justin Wood (:Callek)]

Comment on attachment 688923 [details] [diff] [review]
improvements to manage_buildslave.sh

Review of attachment 688923 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/mobile/manage_buildslave.sh
@@ +43,5 @@
> +        --pidfile=$DEVICE_PATH/twistd.pid \
> +        --python=$DEVICE_PATH/buildbot.tac
> +    sleep 1
> +    tail $DEVICE_PATH/twistd.log
> +    popd 

nit: EOL whitespace

Comment 67

[Jonathan Griffin (:jgriffin)]

Comment on attachment 688878 [details] [diff] [review]
fix naming

lgtm!

Comment 68

[Armen [:armenzg]]

Attachment: do something more interesting with b2g gaia test jobs

Boards won't be able to reach pvt though.
Maybe in an hour.

Comment 69

[Aki Sasaki (not active)]

Comment on attachment 688991 [details] [diff] [review]
do something more interesting with b2g gaia test jobs

Hm, we may want per-request good statuses, but this works for now.

If the query_env()["HOSTNAME"] doesn't work there are other tricks like

    from socket import gethostname
    print gethostname()

that may work.

Hoping this gets us to installing + verifying the image buildid (via log)... :)

Comment 70

[Armen [:armenzg]]

Attachment: b2g_build.py - download from pvtbuilds.pvt for now

This is a hack while bug 818329 gets resolved.

This patch plus attachment 688991 [details] [diff] [review] will get our boards to start testing the imaging process.

Unfortunately the b2g snapshot has not yet made into cedar.
Do we have to wait for the newer b2g snapshot to make it into mozilla-central and then merge to cedar? or can we land the b2g snaptshot on cedar as well?

Comment 71

[Aki Sasaki (not active)]

Landed the b2g snapshot fix on Cedar.

Comment 72

[Aki Sasaki (not active)]

Comment on attachment 689023 [details] [diff] [review]
b2g_build.py - download from pvtbuilds.pvt for now

Ok for now, we may want to unhardcode this later.

Comment 73

[Armen [:armenzg]]

A build is running to test my last two patches.
I will land if I see it succeed in the morning.

Comment 74

[Armen [:armenzg]]

Attachment: use pvtbuilds plus improve sendchange

Comment 75

[Rail Aliiev [:rail]]

In production since 9:00PT

Comment 76

[Armen [:armenzg]]

Attachment: do something more interesting with b2g gaia test jobs + sendchange improvements

This is the final patch for now (I had to make sure that is was a true dry run).

New improvements like getting logcat output will come in a following patch.

Comment 77

[Armen [:armenzg]]

Anyone know if we have everything that we need on tinderbox-builds?

http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121206153204/
b2g-20.0a1.en-US.android-arm.crashreporter-symbols.zip	06-Dec-2012 13:12 25M	 
b2g-20.0a1.en-US.android-arm.tar.gz			06-Dec-2012 13:12 14M	 
boot.tar.bz2						06-Dec-2012 13:12 3.8M	 
build.prop						06-Dec-2012 13:12 1.7K	 
logs/							06-Dec-2012 13:12 - 	 
sources.xml						06-Dec-2012 13:12 16K
system.tar.bz2						06-Dec-2012 13:12 38M
userdata.tar.bz2					06-Dec-2012 13:12 50M	 

From dividehex's artifacts:
boot.tar.bz2		29-Nov-2012 17:15 	3.9M	 
kernel			29-Nov-2012 16:43 	3.7M	 
ramdisk.img		29-Nov-2012 17:12 	201K	 
system.tar.bz2		29-Nov-2012 16:43 	38M	 
userdata.tar.bz2	29-Nov-2012 16:43 	51M	 

I've requested boards 130-138 to be re-imaged with Jake's image and boards 140-149 to get the image from tinderbox-builds.

Comment 78

[Armen [:armenzg]]

It seems that both images are working :S
Maybe we're doing things wrong b2g_panda.py...
http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_panda.py
I will look into it tomorrow.

FTR not all of the pandas came back up.
* came back up: 13{1,2,4,5,6,7,8}
* came back up: 14{0,1,4,5,6,7,9}

It seems that both images manage to come back with similar success ratio.
All of them are in the "free" state.

Jake's image:
################################
Armens-MacBook-Air:~ armenzg$ telnet panda-0131 20701
Trying 10.12.128.114...
Connected to panda-0131.p1.releng.scl1.mozilla.com.
Escape character is '^]'.
$>exec cat /system/b2g/application.ini
[App]
Vendor=Mozilla
Name=B2G
Version=19.0a1
BuildID=20121108162613
ID={3c2e2abc-06d4-11e1-ac3b-374f68613e61}

[Gecko]
MinVersion=19.0a1
MaxVersion=19.0a1

[XRE]
EnableExtensionManager=1

[Crash Reporter]
ServerURL=https://crash-reports.mozilla.com/submit?id={3c2e2abc-06d4-11e1-ac3b-374f68613e61}&version=19.0a1&buildid=20121108162613
return code [0]
$>quit

tbpl's image:
################################
Armens-MacBook-Air:~ armenzg$ telnet panda-0143 20701
Trying 10.12.128.126...
Connected to panda-0143.p1.releng.scl1.mozilla.com.
Escape character is '^]'.
$>exec cat /system/b2g/application.ini                           
[App]
Vendor=Mozilla
Name=B2G
Version=20.0a1
BuildID=20121206153204
SourceRepository=http://hg.mozilla.org/projects/cedar
SourceStamp=12b77bada560
ID={3c2e2abc-06d4-11e1-ac3b-374f68613e61}

[Gecko]
MinVersion=20.0a1
MaxVersion=20.0a1

[XRE]
EnableExtensionManager=1

[Crash Reporter]
ServerURL=https://crash-reports.mozilla.com/submit?id={3c2e2abc-06d4-11e1-ac3b-374f68613e61}&version=20.0a1&buildid=20121206153204
return code [0]
$>quit

Comment 79

[Armen [:armenzg]]

I will track mozpool-releng issues in bug 819492.
Not all of the bugs under that bug will be blocking this work but it will improve the reliability of our new imaging process with mozpool.

Comment 80

[Armen [:armenzg]]

We're now testing the plumbing and I wanted to announce it:
http://armenzg.blogspot.ca/2012/12/b2g-test-jobs-running-on-panda-boards.html

We will now have to integrate the testing once we fix it and meanwhile I will be fixing our imaging issues in bug 819492.

Comment 81

[Armen [:armenzg]]

bug 816345 does not really block. It will be needed for improving bug 819492 though.

Comment 82

[Armen [:armenzg]]

Attachment: add b2g tests to the sendchanges

Comment 83

[Aki Sasaki (not active)]

Comment on attachment 690565 [details] [diff] [review]
add b2g tests to the sendchanges

I was worried this would break in postflight_read_buildbot_config(), but it ends with tests.zip, so we're good :)

I think you can remove the download_and_extract() method here and fall back to using TestingMixin.download_and_extract().

Comment 84

[Armen [:armenzg]]

Attachment: sendchanges for gaia-tests.zip plus start using it

You are right :)

We got the tests!

[cltbld@foopy40 ~]$ cd /builds/panda-0099/test/build
[cltbld@foopy40 build]$ ls gaia-tests.zip 
gaia-tests.zip
[cltbld@foopy40 build]$ ls tests/gaiatest/
gaiatest  MANIFEST.in  README.md  setup.py
[cltbld@foopy40 build]$ find tests/gaiatest/
tests/gaiatest/
tests/gaiatest/gaiatest
tests/gaiatest/gaiatest/mocks
tests/gaiatest/gaiatest/mocks/mock_contact.py
tests/gaiatest/gaiatest/mocks/__init__.py
tests/gaiatest/gaiatest/atoms
tests/gaiatest/gaiatest/atoms/gaia_data_layer.js
tests/gaiatest/gaiatest/atoms/gaia_apps.js
tests/gaiatest/gaiatest/atoms/remote_date.js
tests/gaiatest/gaiatest/atoms/gaia_lock_screen.js
tests/gaiatest/gaiatest/atoms/screenshot.js
tests/gaiatest/gaiatest/.gitignore
tests/gaiatest/gaiatest/gaia_test.py
tests/gaiatest/gaiatest/__init__.py
tests/gaiatest/gaiatest/tests
tests/gaiatest/gaiatest/tests/test_gallery.py
tests/gaiatest/gaiatest/tests/test_music.py
tests/gaiatest/gaiatest/tests/marketplace
tests/gaiatest/gaiatest/tests/marketplace/test_search_marketplace_and_install_app.py
tests/gaiatest/gaiatest/tests/marketplace/manifest.ini
tests/gaiatest/gaiatest/tests/marketplace/test_marketplace_login.py
tests/gaiatest/gaiatest/tests/test_lockscreen.py
tests/gaiatest/gaiatest/tests/test_browser_cell_data.py
tests/gaiatest/gaiatest/tests/test_call_log.py
tests/gaiatest/gaiatest/tests/test_video_player.py
tests/gaiatest/gaiatest/tests/unit
tests/gaiatest/gaiatest/tests/unit/test_killall.py
tests/gaiatest/gaiatest/tests/unit/test_settings.py
tests/gaiatest/gaiatest/tests/unit/test_lock_screen.py
tests/gaiatest/gaiatest/tests/unit/test_wifi.py
tests/gaiatest/gaiatest/tests/unit/manifest.ini
tests/gaiatest/gaiatest/tests/unit/test_launch_entrypoint.py
tests/gaiatest/gaiatest/tests/test_cards_view.py
tests/gaiatest/gaiatest/tests/test_camera.py
tests/gaiatest/gaiatest/tests/test_updater.py
tests/gaiatest/gaiatest/tests/test_dialer.py
tests/gaiatest/gaiatest/tests/test_clock.py
tests/gaiatest/gaiatest/tests/test_browser_wifi.py
tests/gaiatest/gaiatest/tests/contacts
tests/gaiatest/gaiatest/tests/contacts/test_add_new_contact.py
tests/gaiatest/gaiatest/tests/contacts/test_edit_contact.py
tests/gaiatest/gaiatest/tests/contacts/test_call_contact.py
tests/gaiatest/gaiatest/tests/contacts/manifest.ini
tests/gaiatest/gaiatest/tests/contacts/test_sms_contact.py
tests/gaiatest/gaiatest/tests/test_calculator.py
tests/gaiatest/gaiatest/tests/manifest.ini
tests/gaiatest/gaiatest/tests/test_calendar.py
tests/gaiatest/gaiatest/tests/test_sms.py
tests/gaiatest/gaiatest/runtests.py
tests/gaiatest/gaiatest/testvars_template.json
tests/gaiatest/README.md
tests/gaiatest/setup.py
tests/gaiatest/MANIFEST.in

Comment 85

[Aki Sasaki (not active)]

Comment on attachment 690625 [details] [diff] [review]
sendchanges for gaia-tests.zip plus start using it

>+                if os.path.join(dirs['abs_upload_dir'], "gaia-tests.zip"):
>+                     sendchange.append("%s/%s" % (download_url, "gaia-tests.zip"))
>+

a) this will always be true; did you mean os.path.exists(os.path.join(...)) ?
b) if we *don't* append the tests zip to the sendchange, b2g_panda.py will go red since it expects a test zip and won't find one.

We might as well just sendchange if we're going to sendchange.

Comment 86

[Armen [:armenzg]]

(In reply to Aki Sasaki [:aki] from comment #85)
> Comment on attachment 690625 [details] [diff] [review]
> sendchanges for gaia-tests.zip plus start using it
> 
> >+                if os.path.join(dirs['abs_upload_dir'], "gaia-tests.zip"):
> >+                     sendchange.append("%s/%s" % (download_url, "gaia-tests.zip"))
> >+
> 
> a) this will always be true; did you mean os.path.exists(os.path.join(...)) ?
>
Yes, I did.

> b) if we *don't* append the tests zip to the sendchange, b2g_panda.py will
> go red since it expects a test zip and won't find one.
> 
You're right.

> We might as well just sendchange if we're going to sendchange.

I was assuming that not all branches will be generating the gaia-tests.zip since they don't all have the config.json changes.
I guess we would only enable panda tests on branches that would already have the patch in it.

I will remove the if statement.

Comment 87

[Armen [:armenzg]]

== Status ==
* Until bug 820617 is fixed there is nothing that can be done to advance this bug

Meanwhile, I'm adding tooling to deal with the pandas - bug 821380.
This will also give me knowledge on how to handle the SUT agent being down - bug 815785.

Comment 88

[Armen [:armenzg]]

mdas recommends that we update the b2g snapshot to improve reliability.
I will keep track of it so we can uplift it.

Comment 89

[Armen [:armenzg]]

Uplift story:
* bug 818264 got uplifted in bug 819979
* bug 819342 got uplifted in bug 819979
* bug 821398 got landed but needs to be uplifted (will do in bug 822444)
* bug 822444 needs to land and uplifted

Comment 90

[Armen [:armenzg]]

Attachment: put b2g test code inside of b2g_panda.py

I want to put the logic on the same place.

Comment 91

[Armen [:armenzg]]

Attachment: check_b2g_application_ini should not live on tools anymore

Comment 92

[Aki Sasaki (not active)]

Comment on attachment 694567 [details] [diff] [review]
put b2g test code inside of b2g_panda.py

I think this works.

a) since we're in mozharness, this is duplicating DeviceMixin code.
However, I'm ok with this for the time being.
b) we're trying to stick to PEP8 in mozharness; could you s,fileContents,file_contents, ?  I know it's confusing when you throw in the camelcase devicemanager code, but aiui A-Team is pushing towards PEP8 as well.
c) Enhancement: for more automated goodness, we can actually verify the version + buildid in the file_contents and change buildbot status based on that.  Optional; might help avoid having to dig through lots of logs to verify them, though.

Comment 93

[Armen [:armenzg]]

Comment on attachment 694567 [details] [diff] [review]
put b2g test code inside of b2g_panda.py

Adjusted the style. 69a398af05cc

I will look as well into DeviceMixin.

Comment 94

[Armen [:armenzg]]

Comment on attachment 694568 [details] [diff] [review]
check_b2g_application_ini should not live on tools anymore

http://hg.mozilla.org/build/tools/rev/ef614e6bea2e

Comment 95

[Armen [:armenzg]]

Attachment: Fix previous landing

Whoever gets sooner to the patch. Thanks!

Comment 96

[Justin Wood (:Callek)]

Comment on attachment 694857 [details] [diff] [review]
Fix previous landing

Review of attachment 694857 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_panda.py
@@ +108,5 @@
>          """
>          Run the Panda tests
>          """
> +        mph = self.query_mozpool_handler()
> +        #self.request_url = "http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/958/"

I'm not a fan of commented-out code in final patches, is there a special reason to have this here?

Comment 97

[Armen [:armenzg]]

Comment on attachment 694857 [details] [diff] [review]
Fix previous landing

The comment was not to be there!

Landed and fixed :)

Comment 98

[Armen [:armenzg]]

I will gone next week so I better explain where things stand.
In short: releng is waiting on dependent bugs
I will leave a patch that would pretend to do what it is expected to be done as best as I know.

Status
######
Biggest blocker:
* bug 820617: Connecting to an nsIServerSocket on panda fails with NS_ERROR_OFFLINE 
** philikon is investigating

NOTE: As soon as bug 820671 gets fixed we might be able to run the tests (or a subset of them). Some of the dependent bugs are not blockers and/or only block a subset of the tests.

Other blockers:
* bug 819603: Booting on Panda often fails with wpa_driver_nl80211_driver_cmd: ...
** I'm not sure if this is just to improve the likeability that a board boots up
** I assume that mozpool would keep on trying until it manages to get the board up
* bug 820833: "No memory card found" in gaia apps on pandaboard
** This seems to help some specific tests to work rather than blocking the whole project
* bug 821379: Pandaboard will become unresponsive after idling
** IIUC this does not hard block us
* bug 822444: Update the B2G panda snapshot to include marionette on gaia-tests.zip 
** this has landed on m-c and cedar
** we're waiting on approvals to uplift
* bug 823122: Add settime support to Negatus (aka SUT agent version of b2g)
** IIUC this does not block us but it is needed for some tests to pass

Comment 99

[Armen [:armenzg]]

Attachment: patch to try to run tests

This patch gets us as far as I can without fixing bug 824094.

Comment 100

[Aki Sasaki (not active)]

Comment on attachment 694975 [details] [diff] [review]
patch to try to run tests

I think you'll have to import TBPL_WARNING as well (python will probably warn you about this when you try to run this).

We may want to use an error_list for code = self.run_command(cmd) to detect issues, but if the exit code is reliable that may be sufficient.

I haven't tried running this or anything, but it looks like it might work :)

Comment 101

[Armen [:armenzg]]

I managed to telnet to the 2828 port.
Tomorrow morning I will try to run the tests.

Blockers:
* bug 825899
** Reset the cedar repo. We only run panda jobs on Cedar.
* bug 824094
** This is where I have put my efforts to figure out how to run the tests correctly

Dep bugs:
* bug 820833: "No memory card found" in gaia apps on pandaboard
** This seems to help some specific tests to work rather than blocking the whole project
* bug 821379: Pandaboard will become unresponsive after idling
** I understand this a little better. This could actually affect test results if we don't keep the board busy enough while running test jobs
* bug 823122 & bug 825259: Add settime support to Negatus (aka SUT agent version of b2g)
** IIUC this does not block us but it is needed for some tests to pass

Comment 102

[Armen [:armenzg]]

We're pretty much done.
I want bug 826901 to be fixed before we enable these Gaia UI tests wherever is needed.
I will work on that patch meanwhile. I hope we can green out the tests by today/tomorrow (not in my hands though).

SUMMARY:
########
* bug 820833: "No memory card found" in gaia apps on pandaboard
** Specific tests 
* bug 821379: Pandaboard will become unresponsive after idling
** This could cause problems in the future. I don't think it blocks this bug
* bug 823122: Add settime support to Negatus
** I think this is fixed. I'm just waiting for the bug to be closed
* bug 826901: Mirror gaia-ui-tests to gaia
** This will remove tests that are keeping the jobs currently perma-orange
* bug 827867: Add OutputParser
** This will print the summary for tbpl to show it

Comment 103

[Armen [:armenzg]]

Attachment: add b2g gaia ui tests

The patch adds b2g panda gaia ui tests to all the remaining BRANCHES in b2g_config.py

We can add this once we have green tests.

Armens-MacBook-Air:master armenzg$ diff old new 
736a737,738
> b2g_panda try opt test gaia-ui-test ScriptFactory
> b2g_panda mozilla-aurora opt test gaia-ui-test ScriptFactory
737a740,742
> b2g_panda mozilla-central opt test gaia-ui-test ScriptFactory
> b2g_panda services-central opt test gaia-ui-test ScriptFactory
> b2g_panda mozilla-b2g18 opt test gaia-ui-test ScriptFactory
738a744,745
> b2g_panda mozilla-inbound opt test gaia-ui-test ScriptFactory
> b2g_panda fx-team opt test gaia-ui-test ScriptFactory

Comment 104

[Armen [:armenzg]]

SUMMARY:
########
* bug 820833: "No memory card found" in gaia apps on pandaboard
** Some specific tests will fail
* bug 826901: Mirror gaia-ui-tests to gaia
** This will remove the calculator tests which are keeping the jobs perma-orange

Comment 105

[Armen [:armenzg]]

Landing bug 829053 added a bunch of tests and we're now failing all tests in a similar way which might need a fix in marionette (JavascriptException: TypeError: settings is null).

Landing bug 829053 has removed all the Calculator tests that were failing.

Once that issue is fixed we will be able to see how many tests are still orange.

Comment 106

[Armen [:armenzg]]

Unfortunately there is nothing for me to do until the dep bugs get fixed.
Lowering to P3 in hopes that in the next week or two there will be more I can actually do.

The click events one is the one that will give the more headaches.

SUMMARY
#######
* bug 820833: "No memory card found" in gaia apps on pandaboard
** Some specific tests will fail
* bug 827783: Gaia UI tests failing with JavascriptException
* bug 829053: click events sent by marionette are discarded in gaia
** This is quite major and it was added on 2013-01-05 in bug 823619

Comment 107

[Armen [:armenzg]]

We are now passing 6 tests after a fix from bug 830431 landed.
We are now triggering two separate jobs for gaia and gaia-central (bug 829099)
Latest results in here:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18933827&tree=Cedar&full=1

tl;dr;
######
We are waiting for all tests suites to green before enabling across the board.
The two releng bugs at the bottom are blockers for doing things the right way.

SUMMARY
#######
* bug 820833: "No memory card found" in gaia apps on pandaboard
** Some specific tests will fail
* bug 827783: Gaia UI tests failing with JavascriptException: TypeError: settings is null
* bug 829053: click events sent by marionette are discarded in gaia
* bug 830431: socket.timeout or error: [Errno 113] No route to host
** might be done now
* bug 831820: run logcat at the end of gaia UI tests
** this helps when tests go wrong
* bug 832459: Pandaboards need to have the date set for some tests to run
** this will help with a test

Bugs that are more easily fixed by releng:
* bug 831788: gaia-central and gaia b2g builds are uploading to the same location
* bug 829145: Reveal Gaia changesets for b2g builds
** This helps with sheriffing

Comment 108

[Armen [:armenzg]]

A bunch of tests will be fixed by mirroring gaia-ui-tests (see bug 832492).

Comment 109

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #108)
> A bunch of tests will be fixed by mirroring gaia-ui-tests (see bug 832492).

30 test passes and 6 failing tests:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18960013&tree=Cedar&full=1

09:04:32 INFO - TEST-UNEXPECTED-FAIL | test_initial_state.py TestInitialState.test_state_after_reset | DMError: DeviceManager: Error reading file to push
09:06:10 INFO - TEST-UNEXPECTED-FAIL | test_search_marketplace_and_install_app.py TestSearchMarketplaceAndInstallApp.test_search_and_install_app | TimeoutException: Element .header-button.icon.search.right not visible before timeout
09:09:23 INFO - TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | clearElement@chrome://marionette/content/marionette-listener.js:855
09:10:29 INFO - TEST-UNEXPECTED-FAIL | test_gallery.py TestGallery.test_gallery_view | DMError: DeviceManager: Error reading file to push
09:11:37 INFO - TEST-UNEXPECTED-FAIL | test_radio.py TestRadio.test_radio | AssertionError: Antenna (headphones) not plugged in
09:11:39 INFO - TEST-UNEXPECTED-FAIL | test_video_player.py TestVideoPlayer.test_play_video | DMError: DeviceManager: Error reading file to push

Comment 110

[Malini Das [:mdas] - Away, not checking bugmail]

landed as:
https://hg.mozilla.org/build/mozharness/rev/39120c50f89b

Comment 111

[Malini Das [:mdas] - Away, not checking bugmail]

posted to the wrong bug.

Comment 112

[Armen [:armenzg]]

Down to 1 or 2 test failures! Looking into it with mdas.

Gaia Nightly:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18989486&tree=Cedar&full=1
07:43:36     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | 	clearElement@chrome://marionette/content/marionette-listener.js:855

Gaia Central:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18989504&tree=Cedar&full=1
07:44:36     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | 	clearElement@chrome://marionette/content/marionette-listener.js:855
07:46:56     INFO -  TEST-UNEXPECTED-FAIL | test_wallpaper.py TestWallpaper.test_change_wallpaper | AssertionError: Wallpaper has not changed from default.

Comment 113

[Armen [:armenzg]]

Attachment: add b2g gaia ui tests for every B2G branch

Comment 114

[Aki Sasaki (not active)]

Comment on attachment 704557 [details] [diff] [review]
add b2g gaia ui tests for every B2G branch

These look to still be orange -- we'll end up running these on all branches, but hidden everywhere.  Is that what we want?

Comment 115

[Armen [:armenzg]]

(In reply to Aki Sasaki [:aki] from comment #114)
> Comment on attachment 704557 [details] [diff] [review]
> add b2g gaia ui tests for every B2G branch
> 
> These look to still be orange -- we'll end up running these on all branches,
> but hidden everywhere.  Is that what we want?

I wanted to have the patch ready and reviewed for when it is green.
We could be enabling it everywhere today or tomorrow.

Comment 117

[Jonathan Griffin (:jgriffin)]

Note that this bug deals with getting Gaia UI tests running on pandas in TBPL.  Vivien, do you think we should be running them on b2g desktop builds as well?

Comment 118

[Armen [:armenzg]]

tl;dr;
######
One test left to green out (the calendar test on bug 833416) before we can enable everywhere.

SUMMARY
#######
* bug 820833: "No memory card found" in gaia apps on pandaboard
** It does not block this bug specifically
** It does not fail any of the currently running tests
** Adding new tests should be taken care on a per-case basis
** Removing dependency
* bug 830431: socket.timeout or error: [Errno 113] No route to host
** it happens intermittently
** Removing from dependency
* bug 831820: run logcat at the end of gaia UI tests
** waiting on reviews
* bug 831788: gaia-central and gaia b2g builds are uploading to the same location
** waiting on reviews
* bug 829145: Reveal Gaia changesets for b2g builds
** waiting on reviews
* bug 833416 - calendar test fails

Comment 119

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to Jonathan Griffin (:jgriffin) from comment #117)
> Note that this bug deals with getting Gaia UI tests running on pandas in
> TBPL.  Vivien, do you think we should be running them on b2g desktop builds
> as well?

Having them running on Panda sounds good enough to me. All my wishes will become true once bug 813301 will be resolved too - We have around a thousand of tests there waiting to be visible on tbpl ;)

Thanks for the work done here.

Comment 120

[Armen [:armenzg]]

Comment on attachment 704557 [details] [diff] [review]
add b2g gaia ui tests for every B2G branch

Merged to production and live on the masters.

Comment 121

[Armen [:armenzg]]

It seems that between yesterday green and today's merge a gecko/marionette change has landed that has made most tests to fail.

bug 834266 is our newest VIP bug.

Comment 122

[Armen [:armenzg]]

SUMMARY
#######
* bug 831788: gaia-central and gaia b2g builds are uploading to the same location
** testing new patch. waiting on reviews
* bug 834266: it prevents us from running green
** patch being tested and reviewed

Comment 123

[Armen [:armenzg]]

Almost there, we have one last intermittent test - bug 836510.

Comment 124

[Armen [:armenzg]]

One new perma orange :( - bug 836745
It only affects gaia-central and we have a range of changesets.

Comment 125

[Armen [:armenzg]]

We determined last week that without having gaia nightly based builds we cannot determine if a test failure would be due to gaia changes or gecko changes.

bug 834354 has now become critical if we want to keep this job to be sheriff-manageable.

jgriffin fixed bug 836772 which makes it easier to maintain the test jobs up-to-date.

Comment 126

[Armen [:armenzg]]

SUMMARY
#######
* bug 834354 - Add Gaia-Master tbpl tree
** This has landed and will be live by EOD
** A heads-up has been sent to mozilla.dev.gaia
* bug 838236 - 4 failing orange tests
** needs an owner

Comment 127

[Armen [:armenzg]]

SUMMARY
#######
** A heads-up has been sent to mozilla.dev.gaia
* bug 838236 - 4 failing orange tests
** needs an owner

We now have per Gaia commit coverage:
https://tbpl-dev.allizom.org/?tree=Gaia-Master

Is there any left before we go ahead and clear the last 4 test failures?

Comment 128

[Jonathan Griffin (:jgriffin)]

Nice job on the per-gaia-commit work!  When will this go live on production TBPL?

Comment 129

[Armen [:armenzg]]

(In reply to Jonathan Griffin (:jgriffin) from comment #128)
> Nice job on the per-gaia-commit work!  When will this go live on production
> TBPL?

Oops! I forgot I was pasting tbpl-dev.
edmorley told me to wait for him to test some deploy technique. I will ask him on the bug.

Comment 130

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #129)
> (In reply to Jonathan Griffin (:jgriffin) from comment #128)
> > Nice job on the per-gaia-commit work!  When will this go live on production
> > TBPL?
> 
> Oops! I forgot I was pasting tbpl-dev.
> edmorley told me to wait for him to test some deploy technique. I will ask
> him on the bug.

It seems that the changes got deployed without me knowing:
https://tbpl.mozilla.org/?tree=Gaia-Master

Comment 131

[Jonathan Griffin (:jgriffin)]

Nice! We've been fighting a few other fires wrt the Gaia UI tests this week, so won't have a chance to deal with the current oranges until next.

Comment 132

[Armen [:armenzg]]

jgriffin, anyone available from the A-team to work on greening this out?

Comment 133

[Jonathan Griffin (:jgriffin)]

So, the current plan is to get these running with B2G desktop builds, since it's much easier for people to debug/fix problems on that build.  We'll then see how much relative value we get out of the panda runs.

Comment 134

[Armen [:armenzg]]

Nothing left to be done in here.
If we find the b2g desktop builds not good enough we will come back and visit using the pandas.