1277921 - Autophone - busted Talos tpn test did not generate failure summary

Status: RESOLVED DUPLICATE of bug 1381511

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

The failure summary for tpn is empty in 
https://treeherder.mozilla.org/#/jobs?filter-searchStr=autophone&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&repo=mozilla-inbound&revision=867a9bdc2066&selectedJob=29439060

The autophone log contained:

2016-06-03 11:02:33,418|1597|MainThread|phonetest|INFO|nexus-6p-2|20160603103108|remote-tp4m|Running test (1/1)
2016-06-03 11:06:19,212|1597|MainThread|phonetest|WARNING|nexus-6p-2|20160603103108|remote-tp4m|Unable to find pageload metric
2016-06-03 11:06:19,213|1597|MainThread|phonetest|INFO|nexus-6p-2|20160603103108|remote-tp4m|Failed to get measurements for test remote-tp4m

Comment 1

[Bob Clary [:bc] (inactive)]

Note the corresponding tsvg test also failed to get measurements for this run but did generate failure messages in the summary:

TEST_UNEXPECTED_FAIL | -tp file:///sdcard/tests/autophone/TalosTest/files/ep1/talos/tsvg/manifest/svgm.manifest.remote -tpnoisy -rss -tpcycles 6 -tppagecycles 1 -tpdelay 1000 | Failed to get measurement.
TEST_UNEXPECTED_FAIL | autophone-talos | No measurements detected.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

is this device specific, or is this potentially build specific?

I don't see anything in the logs on treeherder to indicate we have even launched fennec and tried to load a page, not sure if that is the fix we are looking to do here.

Comment 3

[Bob Clary [:bc] (inactive)]

The main issue I was looking to fix here is that tpn should generate some failure messages so that we can try to look up suggested bugs that the sheriffs can use when this failure occurs after we move to tier 2.

The failure to launch is a problem but not the focus here.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

ok, good point, I think this is our main error:
TEST_UNEXPECTED_FAIL | autophone-talos | No measurements detected.

that will happen rare enough that it would be something random vs many different things.  While the root cause might be different in each case, it would be a lot of work to get all of the root causes figured out and print out a useful message.

I would prefer to take an approach that if we get >10 of these in a given week that we investigate further.