776728 - perform acceptance test of panda chassis configuration

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Hal Wine [:hwine] use NI!]

+++ This bug was initially created as a clone of Bug #769429 +++

(In reply to Clint Talbert ( :ctalbert ) from bug 769429 comment #10)
> Great, Armen correct me if I'm wrong, but our next step should be to get the
> agent on these and try running tests?

Yes - we need to simulate production usage of this chassis. At a minimum (ideally), that means:
 a) impose max load on all devices simultaneously until temp stabilizes (demonstrate thermal & power supply capacities)
 b) test all impact of turning on RF devices on adjacent boards (demonstrate isolation of each board)
 c) demonstrate control of power to each Panda board via SNMP

I'm hoping some of the existing tests be re-purposed to help with (a) & (b)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

First off, thanks for filing this bug.  This is what we need!

These devices won't be using wifi, only ethernet, and should just have an sdcard (os+data) and power supply.  That should minimize the interfaces we access.

I would say lets build a suite using sut_tools (maybe mozharness) that does a simple smoke test (a really small test suite that will pass- i.e. a 5 minute mochitest dom-level-1 run), and put this in a loop with reboots, uninstall, install.

we have mozharness for talos that could be configured to run a smaller version of the test, or we could get sut_tools and a foopy lined up.  Either way we need a machine to control this.

In doing this, I would recommend about two days of monitoring (probably a lot of hands on fiddling) until we have it working in a continuous loop on 1 board.  Then a day of close monitoring while we scale to 10 boards.  I say a full weekend (60+ hours) of non stop automation with no failures would be ideal.

This should be a good starting point, lets get agreement on what we need to test first before writing custom scripts for this.

Comment 2

[Corey Shields [:cshields]]

Hal, who from releng or ATeam is on point for writing these tests (Joel gave some good recommendations)?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Sign me up for coordinating or doing the script writing.  There are a few ateam members who are working on panda boards, so it might be a split task or handed off if my skills are deemed too in need or not good enough.

Comment 4

[cmtalbert]

This sounds good. Both Joel and Will can help write the scripts to ensure the pandas are working.  We will need releng support to bring up a foopy to control the chassis of 10 pandas.

Comment 5

[Hal Wine [:hwine] use NI!]

(In reply to Joel Maher (:jmaher) from comment #1)
> These devices won't be using wifi, only ethernet, and should just have an
> sdcard (os+data) and power supply.  That should minimize the interfaces we
> access.

Just asking based on being burned elsewhere - is wifi disabled on board or in firmware, or it's just not convention to write a test that turns wifi/phone/bluetooth on? (Will the same also hold for b2g?)

I don't believe we need to take any action, necessarily, but would like to have radio usage documented as part of our spec for the chassis.

Comment 6

[(not currently active) Ted Mielczarek]

We haven't explicitly disabled the wifi, we're just not activating it. If necessary, we can look into disabling it more thoroughly.

Comment 7

[Hal Wine [:hwine] use NI!]

(In reply to Ted Mielczarek [:ted] from comment #6)
> We haven't explicitly disabled the wifi, we're just not activating it. If
> necessary, we can look into disabling it more thoroughly.

Good enough for now - the documentation is key - thanks!

Can you confirm same holds for phone/bluetooth (or that hardware doesn't exist on panda boards)?

Comment 8

[Armen [:armenzg]]

Adjusting dependencies of:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=725544&hide_resolved=1

Comment 9

[Armen [:armenzg]]

Sorry for the noise again!

Comment 10

[William Lachance (:wlach)]

I am looking at writing such a script with mozharness, currently trying to unwind how it works. It looks like scripts/device_talosrunner.py might be just the ticket to writing a simple smoketest. Will do more looking into it tomorrow!

Comment 11

[William Lachance (:wlach)]

I just barely managed to get talos's ts running against my phone using mozharness today, though I found some minor issues: bug 779983, bug 779980, bug 779979. There is still the issue of parsing/using the results. 

To be honest I am not sure if Talos/MozHarness are the ideal tools to use here as it can be very difficult to see what's going on if things go wrong, given the present state of things. It does seem to have some nice features, but I think it might need some more time to bake (or for me to just understand it better).

I am strongly thinking of just writing a small utility tomorrow to just do the equivalent of Talos (start the browser repeatedly and make sure it's working). I'll sleep on it.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Good writeup Will.  I found that in talos the getInfo.html is all that works on the panda right now.  I have gone back a whole month in builds with no luck in changing the results.  Also reftests work, but that seems a bit too complex for what we are looking to do.

Maybe a stripped down getInfo.html from talos will be all we need!

Comment 13

[William Lachance (:wlach)]

So I found that the smoketest in the autophone framework already did *almost* exactly what we wanted to do here, so I decided to piggy-back on top of it by enhancing it support running it an arbitrary number of times and searching for failures. Basically it just reboots the machine and starts Fennec repeatedly, which is really all we want to test. Yay for not writing yet another piece of code with its own deployment story.

Anyway, I'm going to be gone on Monday, but I'll try to have something that people can pull and use (along with appropriate instructions) sometime Tuesday.

Comment 14

[William Lachance (:wlach)]

Filed bug 781341 to explicitly track the smoketest.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

this is done.