1020366 - Adapt Mozpool to B2G

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We currently use Mozpool to manage device flashing and provisioning for Android panda tests:  https://wiki.mozilla.org/ReleaseEngineering/Mozpool

We'd like to adapt this system to work with B2G Flame devices in a different environment (Jenkins vs buildbot).

In this environment, Mozpool will manage pools of devices, each pool being physically connected via USB to a host machine, and each pool being accessible via a similar number of Jenkins slaves hosted on the same machine.

The Jenkins test slave will request a device from mozpool, along with a build url.  Mozpool will identify a device in its pool that is running, will flash the device, verify that the device comes up after rebooting and that Marionette is available on it, and will return to the Jenkins instance an ADB serial number and a Marionette port number.  The Jenkins slave will then run tests on the device, and when the tests are done, will signal Mozpool that the device is no longer in use.

Mozpool will have the ability to remotely power cycle the phone, courtesy of jhylands power harness, so may be able to recover from some flashing and rebooting errors.

If a device dies while flashing or rebooting and Mozpool cannot recover it, it will mark the device as inactive, and will somehow alert us of this for manual investigation.  It will then locate another device (or wait until one is available) to satisfy the original request.

The number of devices in a pool is not known; it will depend on how many devices we can connect simultaneously to the slave and how well ADB can handle these concurrently.

Other features we should plan on implementing:
- the ability to request a specific device by serial number
- the ability to segment a set of devices for use by specific jobs (i.e., set aside two devices in a 'b2gperf' pool, and use these to satisfy all b2gperf requests)

Comment 1

[Andrew Halberstadt [:ahal]]

The powertool can't tell which ammeters are associated with which phones in the event that multiple phones are connected to the same host, there's simply no way to get that information.

To get around this, we need to power on/off each ammeter one at a time and examine which adb serial number disappears from 'adb devices' and maintain an internal mapping (one per host) of ammeter -> phone. I guess if a new ammeter/phone pair is added/removed to a host, we can see which ammeter/phone is new or missing and update the internal mapping appropriately.

This seems like it could be pretty fragile.. but I don't think there is any other way to manage it. I started the ability to do this here:
https://github.com/ahal/fxos-powertool/tree/device_pool

That work is blocked on me getting a battery harness from jhylands for testing. Also, it might make sense for that logic to live in mozpool rather than powertool. I don't have strong opinions about it either way.

Comment 2

[Andrew Halberstadt [:ahal]]

More specifically, this method is what builds the internal mapping:
https://github.com/ahal/fxos-powertool/blob/device_pool/powertool/device/pool.py#L37

Note this doesn't yet support adding or removing phones to an existing pool.. but that can be added.

Comment 3

[(not currently active) Ted Mielczarek]

Is this something we could simply put in a configuration file? For the Android Pandas we have the relay bank/number in the IT inventory system for the very same reason. See 'system.relay.0' in inventory:
https://inventory.mozilla.org/en-US/systems/show/6981/

Comment 4

[Andrew Halberstadt [:ahal]]

Possibly.. Flames have a unique static serial we can use. For the ammeters I've been using their path under /dev/ as a serial which obviously can change if they are disconnected and re-connected again. I don't see a unique serial for them when running lsusb.. but maybe there's some other way to obtain one? Jhylands would be the better person to ask about that.

Comment 5

[Jon Hylands [:jhylands]]

You can get the unique serial number from an ammeter using the GET_SERIAL command, as shown in this example:

https://github.com/JonHylands/moz-amp/blob/gui-moz-amp/serialNumber.py

Comment 6

[Bob Clary [:bc] (inactive)]

I totally missed Jon's comment about the ammeter serial numbers but rediscovered it the hard way. :-(

If we keep the device's adb serial number and its associated ammeter serial number in the k/v store for the network adapter in the device's inventory entry, then we can quickly discover the ammeter serial number to /dev/ttyACM* port mapping by iterating over /dev/ttyACM* and calling GET_SERIAL.

Comment 7

[Bob Clary [:bc] (inactive)]

Dustin and I have agreed on this approach. Jacques, are the proposed changes to inventory ok?

+ inventory.mozilla.org
  + b2g/android devices should use the inventory.mozilla.org db
    systems.serial to store the adb serial number of the device.
  + The inventory table server_models will list each b2g/android
    device/ram combination as a distinct model.
  + Currently the key/value system.relay.0 is used to encode
    information about the relay used to control power to a device. We
    will generalize this so it can also be used to encode information
    about the ammeter used to control a b2g device. To reduce
    confusion about its use, we will rename it to
    system.power_controller.0.
  + To clearly identify the type of power controller for a device, we
    will change the pattern to include an identifier for the power
    controller type.  For panda devices, system.power_controller.0
    will be of the form relay:<relayhost>:<bank\d>:<relay\d> while for
    b2g devices, system.power_controller.0 will be of the form
    ammeter:<ammeter-type>:<ammeter-serialno>. For example, a mozilla
    type ammeter with serial number 101 would be encoded as
    "ammeter:mozilla:101". This implies a fixed pairing of device and
    ammeter. They should be labelled so as to make it clear which
    ammeter belongs to each device in the lab.  If a device (such as
    an android phone) does not support power cycling via an ammeter,
    its system.power_controller.0 value will be set to none.
  + Each ammeter should be tracked in inventory. This will also allow
    the firmware revision for each ammeter to be tracked.
+ mozpool.
  + The column devices.relay_info will be renamed to
    devices.power_controller and the system.power_controller.0 data
    will be stored in the mozpool db in devices.power_controller as is
    done currently.
  + Detect the ammeter device port to b2g device mapping and store it
    in ram dynamically but don't put it in the mozpool devices table.
  + The system.imaging_server.0 data will be stored into the
    imaging_servers table as currently done.
  + The inventory systems.serial should be stored in the mozpool db in
    a new column devices.adb_serial.
+ database changes
  + add mozpool db devices.adb_serial column.
  + Rename mozpool's devices.relay_info to devices.power_controller to
    avoid confusion about the dual use of the column for pandas and
    b2g/android devices.
  + Update the current values in the inventory key_value table to
    include the power controller type prefix. This can be done via:
    update key_value set value = concat('relay:',value) where
    key_value.key = 'system.relay.0';
  + Change the inventory's keyword/value system.relay.0 to
    system.power_controller.0.  This can be done in inventory with a
    simple update of the key_value table: update key_value set
    key_value.key = 'system.power_controller.0' where key_value.key =
    'system.relay.0';
+ configuration for panda, b2g or android devices.
  + a mozpool 'instance' (set of imaging servers) will be restricted
    to managing a single class of devices. There will be a separate
    mozpool instance for panda/relay devices, a separate mozpool
    instance for b2g/ammeter devices and possibly a separate mozpool
    instance for android devices.
  + inventorysync will be modified to handle the different possible
    system.power_controller.0 values. The processing of the
    system.power_controller.0 values will be governed by an
    [inventory] power_control_processing configuration option.
  + config.ini will be used to configure the mozpool instance for
    either panda, b2g or android devices by:
    + [inventory] filter is used to select the set of devices to be
      managed by hostname.
    + [inventory] power_control_processing would be one of:
      + relay - process as relay:<relayhost>:<bank\d>:<relay\d>
      + ammeter - process as ammeter:<ammeter-type>:<ammeter-serialno>
      + none - ignore the value since there is no power control for
        the device.
    + [mozpool] devicestatemachine will be used to choose the
      appropriate device statemachine.

Comment 8

[Jacques Uber [:uberj]]

bob,

When you say system.power_controller.0 will be set to none do you mean the string 'none'?

Comment 9

[Bob Clary [:bc] (inactive)]

(In reply to Jacques Uber [:uberj] from comment #8)
> bob,
> 
> When you say system.power_controller.0 will be set to none do you mean the
> string 'none'?

Yes, or it could be Null if that is preferable.

Comment 10

[Jacques Uber [:uberj]]

The string 'none' is actually preferable. Otherwise I think, Inventory wise, this looks like a valid plan.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I believe this project has been abandoned in favour of using a partner to manage a farm of devices.

Comment 12

[James Lal [:lightsofapollo]]

Correct (at least for our flame devices)

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Closing as WONTFIX based on comment 12.