Closed Bug 1026646 Opened 7 years ago Closed 7 years ago
OS Datazilla Results Missing for startup _>_moz-app-visually-complete test after 2014 .06 .15
375.49 KB, image/png
Results for FxOS' startup_>_moz-app-visually-complete test stopped reporting to Datazilla after 2014.06.15 for all apps except Camera. These results are generated by make test-perf tests. The rationale and work to support these tests is covered by bug 996038.
So it looks like increasing the number of runs may have caused this: 5 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/24/console 30 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/25/console Unfortunately these failures do not cause the make target to exit with a non-zero code, so Jenkins assumes the build was successful. We need to have more visibility of such regressions. The ingestion alerts will be part of that, but these need to be configured so we need to know all the apps/tests that we expect results for.
(In reply to Dave Hunt (:davehunt) from comment #2) > So it looks like increasing the number of runs may have caused this: > > 5 runs: > http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/ > 24/console > 30 runs: > http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/ > 25/console > > Unfortunately these failures do not cause the make target to exit with a > non-zero code, so Jenkins assumes the build was successful. We need to have > more visibility of such regressions. The ingestion alerts will be part of > that, but these need to be configured so we need to know all the apps/tests > that we expect results for. Every test that has errors has an output. It is in the JSON. We deliberately didn't make one test failure cause the whole test suite to fail because partial perf is better than none. Only fatal error that will prevent from running the test will cause that.
Also apparently I can't access these new servers from VPN. Do I need yet another set of permission for that?
(In reply to Hubert Figuiere [:hub] from comment #3) > We deliberately didn't make one test failure cause the whole test suite to > fail because partial perf is better than none. Only fatal error that will > prevent from running the test will cause that. In b2gperf we caught errors and continued with the next app, and set the exit code accordingly so we'd still gather (and submit) the results, but also failed the build in Jenkins. (In reply to Hubert Figuiere [:hub] from comment #4) > Also apparently I can't access these new servers from VPN. Do I need yet > another set of permission for that? You should be able to access so long as you're on the Mozilla-VPN. If you can't, please raise a bug.
Whiteboard: [c=automation p= s= u=] → [c=automation p=2 s= u=]
Hub, I've tried finding the source of this in the logs but nothing stands out to me. Do you have any insight?
I don't have access to said logs yet. See https://bugzilla.mozilla.org/show_bug.cgi?id=1036017
From Jeads: Take a look at the logs and see if there is an error in the HTTP POST. It would be in b2gperf as an error if the HTTP POST fails. If the HTTP POST is succeeding then send jeads a link to datazilla where we would expect to see the data he said he'll try to track it down.
I do see the data in Datazilla: https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=browser,calendar,camera,clock,contacts,email%20FTU,fm_radio,gallery,marketplace,messages,music,phone,settings,template,usage,video&app=phone&gaia_rev=740faa5d0060fb21&gecko_rev=8bfe3372f848&plot=avg This data is being reported for 6 apps; do we expect more?
Yes, we are currently missing Video, Marketplace, and Clock.
The harness is reporting errors for those, see: http://people.mozilla.org/~jgriffin/b2g-inbound.perf.json.txt
I just filed bug 1036145 to grant Eli access to the Jenkins server, so he can look at logs himself.
Clock does scriptTimeout. Video works. No Marketplace. According to this: http://jenkins1.qa.scl3.mozilla.com:8080/view/Performance/job/flame.b2g-inbound.perf.gaia/324/artifact/perf.json
Also locally, marketplace won't work. 'Error',\n stack: 'Error: could not find an app with the origin: \"app://marketplace.firefox.com.gaiamobile.org\"\\n This is somewhat know but no bug filed AFAIK.
Bug 1036448 filed to fix external URL string problems, like the one Hub found for Marketplace. Bug 1036436 filed to add Video application to test whitelist. Still need to track down the timeout issue for the Clock app.
The timeout issue of the Clock is unrelated to this data showing up. This has been resolved.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.