The failure summary for tpn is empty in https://treeherder.mozilla.org/#/jobs?filter-searchStr=autophone&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&repo=mozilla-inbound&revision=867a9bdc2066&selectedJob=29439060 The autophone log contained: 2016-06-03 11:02:33,418|1597|MainThread|phonetest|INFO|nexus-6p-2|20160603103108|remote-tp4m|Running test (1/1) 2016-06-03 11:06:19,212|1597|MainThread|phonetest|WARNING|nexus-6p-2|20160603103108|remote-tp4m|Unable to find pageload metric 2016-06-03 11:06:19,213|1597|MainThread|phonetest|INFO|nexus-6p-2|20160603103108|remote-tp4m|Failed to get measurements for test remote-tp4m
Note the corresponding tsvg test also failed to get measurements for this run but did generate failure messages in the summary: TEST_UNEXPECTED_FAIL | -tp file:///sdcard/tests/autophone/TalosTest/files/ep1/talos/tsvg/manifest/svgm.manifest.remote -tpnoisy -rss -tpcycles 6 -tppagecycles 1 -tpdelay 1000 | Failed to get measurement. TEST_UNEXPECTED_FAIL | autophone-talos | No measurements detected.
is this device specific, or is this potentially build specific? I don't see anything in the logs on treeherder to indicate we have even launched fennec and tried to load a page, not sure if that is the fix we are looking to do here.
The main issue I was looking to fix here is that tpn should generate some failure messages so that we can try to look up suggested bugs that the sheriffs can use when this failure occurs after we move to tier 2. The failure to launch is a problem but not the focus here.
ok, good point, I think this is our main error: TEST_UNEXPECTED_FAIL | autophone-talos | No measurements detected. that will happen rare enough that it would be something random vs many different things. While the root cause might be different in each case, it would be a lot of work to get all of the root causes figured out and print out a useful message. I would prefer to take an approach that if we get >10 of these in a given week that we investigate further.