Closed Bug 1026646 Opened 7 years ago Closed 7 years ago

FxOS Datazilla Results Missing for startup_>_moz-app-visually-complete test after 2014.06.15

Categories

(Datazilla Graveyard :: Metrics, defect, P1)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mlee, Assigned: Eli)

References

()

Details

(Keywords: perf, Whiteboard: [c=automation p=3 s= u=])

Attachments

(1 file)

Results for FxOS'  startup_>_moz-app-visually-complete  test stopped reporting to Datazilla after 2014.06.15 for all apps except Camera.

These results are generated by make test-perf tests. The rationale and work to support these tests is covered by bug 996038.
So it looks like increasing the number of runs may have caused this:

5 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/24/console
30 runs: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/25/console

Unfortunately these failures do not cause the make target to exit with a non-zero code, so Jenkins assumes the build was successful. We need to have more visibility of such regressions. The ingestion alerts will be part of that, but these need to be configured so we need to know all the apps/tests that we expect results for.
(In reply to Dave Hunt (:davehunt) from comment #2)
> So it looks like increasing the number of runs may have caused this:
> 
> 5 runs:
> http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/
> 24/console
> 30 runs:
> http://jenkins1.qa.scl3.mozilla.com:8080/job/flame.mozilla-central.perf.gaia/
> 25/console
> 
> Unfortunately these failures do not cause the make target to exit with a
> non-zero code, so Jenkins assumes the build was successful. We need to have
> more visibility of such regressions. The ingestion alerts will be part of
> that, but these need to be configured so we need to know all the apps/tests
> that we expect results for.

Every test that has errors has an output. It is in the JSON.

We deliberately didn't make one test failure cause the whole test suite to fail because partial perf is better than none. Only fatal error that will prevent from running the test will cause that.
Also apparently I can't access these new servers from VPN. Do I need yet another set of permission for that?
Flags: needinfo?(dave.hunt)
Depends on: 1020557
(In reply to Hubert Figuiere [:hub] from comment #3)
> We deliberately didn't make one test failure cause the whole test suite to
> fail because partial perf is better than none. Only fatal error that will
> prevent from running the test will cause that.

In b2gperf we caught errors and continued with the next app, and set the exit code accordingly so we'd still gather (and submit) the results, but also failed the build in Jenkins.

(In reply to Hubert Figuiere [:hub] from comment #4)
> Also apparently I can't access these new servers from VPN. Do I need yet
> another set of permission for that?

You should be able to access so long as you're on the Mozilla-VPN. If you can't, please raise a bug.
Flags: needinfo?(dave.hunt)
Assignee: nobody → eperelman
Whiteboard: [c=automation p= s= u=] → [c=automation p=2 s= u=]
Hub, I've tried finding the source of this in the logs but nothing stands out to me. Do you have any insight?
Flags: needinfo?(hub)
I don't have access to said logs yet. See https://bugzilla.mozilla.org/show_bug.cgi?id=1036017
Flags: needinfo?(hub)
From Jeads:

Take a look at the logs and see if there is an error in the HTTP POST.  It would be in b2gperf as an error if the HTTP POST fails.  If the HTTP POST is succeeding then send jeads a link to datazilla where we would expect to see the data he said he'll try to track it down.
Yes, we are currently missing Video, Marketplace, and Clock.
I just filed bug 1036145 to grant Eli access to the Jenkins server, so he can look at logs himself.
Also locally, marketplace won't work.

'Error',\n  stack: 'Error: could not find an app with the origin: \"app://marketplace.firefox.com.gaiamobile.org\"\\n  

This is somewhat know but no bug filed AFAIK.
Bug 1036448 filed to fix external URL string problems, like the one Hub found for Marketplace.

Bug 1036436 filed to add Video application to test whitelist.

Still need to track down the timeout issue for the Clock app.
Status: NEW → ASSIGNED
No longer depends on: gaia-perf-events, 1020557
Whiteboard: [c=automation p=2 s= u=] → [c=automation p=3 s= u=]
No longer blocks: 1036448
Depends on: 1036448
No longer blocks: 1020557
Depends on: 1037500
The timeout issue of the Clock is unrelated to this data showing up. This has been resolved.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.