1026646 - FxOS Datazilla Results Missing for startup_>_moz-app-visually-complete test after 2014.06.15

Status: RESOLVED FIXED

(Datazilla Graveyard :: Metrics, defect, P1)

Description

[Mike Lee [:mlee]]

Attachment: Datazilla Screenshot 2014-06-17 at 11.18.58 AM.png

Results for FxOS'  startup_>_moz-app-visually-complete  test stopped reporting to Datazilla after 2014.06.15 for all apps except Camera.

These results are generated by make test-perf tests. The rationale and work to support these tests is covered by bug 996038.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I would suspect bug 1020557 and bug 1025079.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

So it looks like increasing the number of runs may have caused this:

5 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/24/console
30 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/25/console

Unfortunately these failures do not cause the make target to exit with a non-zero code, so Jenkins assumes the build was successful. We need to have more visibility of such regressions. The ingestion alerts will be part of that, but these need to be configured so we need to know all the apps/tests that we expect results for.

Comment 3

[Hubert Figuiere [:hub]]

(In reply to Dave Hunt (:davehunt) from comment #2)
> So it looks like increasing the number of runs may have caused this:
> 
> 5 runs:
> http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/
> 24/console
> 30 runs:
> http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/
> 25/console
> 
> Unfortunately these failures do not cause the make target to exit with a
> non-zero code, so Jenkins assumes the build was successful. We need to have
> more visibility of such regressions. The ingestion alerts will be part of
> that, but these need to be configured so we need to know all the apps/tests
> that we expect results for.

Every test that has errors has an output. It is in the JSON.

We deliberately didn't make one test failure cause the whole test suite to fail because partial perf is better than none. Only fatal error that will prevent from running the test will cause that.

Comment 4

[Hubert Figuiere [:hub]]

Also apparently I can't access these new servers from VPN. Do I need yet another set of permission for that?

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Hubert Figuiere [:hub] from comment #3)
> We deliberately didn't make one test failure cause the whole test suite to
> fail because partial perf is better than none. Only fatal error that will
> prevent from running the test will cause that.

In b2gperf we caught errors and continued with the next app, and set the exit code accordingly so we'd still gather (and submit) the results, but also failed the build in Jenkins.

(In reply to Hubert Figuiere [:hub] from comment #4)
> Also apparently I can't access these new servers from VPN. Do I need yet
> another set of permission for that?

You should be able to access so long as you're on the Mozilla-VPN. If you can't, please raise a bug.

Comment 6

[:Eli Perelman]

Hub, I've tried finding the source of this in the logs but nothing stands out to me. Do you have any insight?

Comment 7

[Hubert Figuiere [:hub]]

I don't have access to said logs yet. See https://bugzilla.mozilla.org/show_bug.cgi?id=1036017

Comment 8

[Dave Huseby [:huseby]]

From Jeads:

Take a look at the logs and see if there is an error in the HTTP POST.  It would be in b2gperf as an error if the HTTP POST fails.  If the HTTP POST is succeeding then send jeads a link to datazilla where we would expect to see the data he said he'll try to track it down.

Comment 9

[Jonathan Griffin (:jgriffin)]

I do see the data in Datazilla:

https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=browser,calendar,camera,clock,contacts,email%20FTU,fm_radio,gallery,marketplace,messages,music,phone,settings,template,usage,video&app=phone&gaia_rev=740faa5d0060fb21&gecko_rev=8bfe3372f848&plot=avg

This data is being reported for 6 apps; do we expect more?

Comment 10

[:Eli Perelman]

Yes, we are currently missing Video, Marketplace, and Clock.

Comment 11

[Jonathan Griffin (:jgriffin)]

The harness is reporting errors for those, see:

http://people.mozilla.org/~jgriffin/b2g-inbound.perf.json.txt

Comment 12

[Jonathan Griffin (:jgriffin)]

I just filed bug 1036145 to grant Eli access to the Jenkins server, so he can look at logs himself.

Comment 13

[Hubert Figuiere [:hub]]

Clock does scriptTimeout. Video works. No Marketplace.

According to this:

http://jenkins1.qa.scl3.mozilla.com:8080/view/Performance/job/flame.b2g-inbound.perf.gaia/324/artifact/perf.json

Comment 14

[Hubert Figuiere [:hub]]

Also locally, marketplace won't work.

'Error',\n  stack: 'Error: could not find an app with the origin: \"app://marketplace.firefox.com.gaiamobile.org\"\\n  

This is somewhat know but no bug filed AFAIK.

Comment 15

[:Eli Perelman]

Bug 1036448 filed to fix external URL string problems, like the one Hub found for Marketplace.

Bug 1036436 filed to add Video application to test whitelist.

Still need to track down the timeout issue for the Clock app.

Comment 16

[:Eli Perelman]

The timeout issue of the Clock is unrelated to this data showing up. This has been resolved.