Jobs are generating an "error in perfherder data, ignoring"

RESOLVED FIXED in Firefox 50

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

unspecified
mozilla50
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox50 fixed)

Details

Attachments

(2 attachments)

Sometime in the last few days, jobs started spitting out errors like this:

```
error while validating Perfherder data; ignoring
Traceback (most recent call last):
04:38:56 ERROR -
File "/builds/slave/test/scripts/mozharness/base/python.py", line 644, in _log_resource_usage
jsonschema.validate(data, schema)
04:38:56 ERROR -
File "/builds/slave/test/build/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 478, in validate
cls(schema, *args, **kwargs).validate(instance)
04:38:56 ERROR -
File "/builds/slave/test/build/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
raise error
04:38:56 ERROR -
ValidationError: None is not of type u'number'
04:38:56 ERROR -
Failed validating u'type' in schema[u'properties'][u'suites'][u'items'][u'properties'][u'subtests'][u'items'][u'properties'][u'value']:
{u'description': u'Summary value for subtest',
u'title': u'Subtest value',
u'type': u'number'}
04:38:56 ERROR -
On instance[u'suites'][2][u'subtests'][1][u'value']:
None
04:38:56 ERROR -
```

I'm guessing this is due to some kind of change in the build system that's causing the perfherder output to fail.

In addition to fixing whatever the problem is, IMO we should just turn the jobs orange if this kind of validation fails (note that we're not yet storing this data, but we probably want to), otherwise it's just going to keep breaking.
:gps, do you know what the cause of this is? I'd like to start experimenting with storing this information, at least on treeherder stage.
Flags: needinfo?(gps)

Comment 2

2 years ago
AFAIK nothing related to this changed. Was the JSON schema updated?

If someone other than you reported the bug, I would have flagged *you* for needinfo ;)
Flags: needinfo?(gps)
Did some debugging on mochitest-1, looks like we're generating the following (based on adding some debugging info to this push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7223e726213&selectedJob=24139108)

{
	"framework": {
		"name": "job_resource_usage"
	},
	"suites": [{
		"subtests": [{
			"name": "cpu_percent",
			"value": 10.57951977401129
		}, {
			"name": "io_write_bytes",
			"value": 870832640
		}, {
			"name": "io.read_bytes",
			"value": 37296128
		}, {
			"name": "io_write_time",
			"value": 41616420
		}, {
			"name": "io_read_time",
			"value": 318910
		}],
		"extraOptions": [],
		"name": "mochitest.plain-chunked.1.overall"
	}, {
		"subtests": [{
			"name": "time",
			"value": 2.248000144958496
		}, {
			"name": "cpu_percent",
			"value": 10.943750000000001
		}],
		"name": "mochitest.plain-chunked.1.install"
	}, {
		"subtests": [{
			"name": "time",
			"value": 0.0
		}, {
			"name": "cpu_percent",
			"value": null
		}],
		"name": "mochitest.plain-chunked.1.stage-files"
	}, {
		"subtests": [{
			"name": "time",
			"value": 710.164999961853
		}, {
			"name": "cpu_percent",
			"value": 10.587216312056727
		}],
		"name": "mochitest.plain-chunked.1.run-tests"
	}]
}

Perfherder really doesn't like the fact that we're setting cpu_percent to null for mochitest.plain-chunked.1.stage-files. Need to look into why that's happening. It appears that it started with this push from ahal: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4a2a06f7dad65d5b1c4c6a7ad717b9ca8354141b, but need to do some retriggers to verify that's what's happening.

In any event I'm going to make perfherder validation of this data *mandatory* in the future so it's someone else's damn job to fix this if it breaks again. :)
Assignee: nobody → wlachance
Yeah in fact it was bug 1285553, it looks like that adds a "stage files" step which is so brief that it doesn't have a cpu percentage. Let's just do a null check against cpu_percent before adding any mozharness step to perfherder data.
Blocks: 1285553
(In reply to William Lachance (:wlach) from comment #3)
> In any event I'm going to make perfherder validation of this data
> *mandatory* in the future so it's someone else's damn job to fix this if it
> breaks again. :)

I don't know.. I kind of like the setup now ;)

Sorry, I did see the error but I assumed it was an existing issue as it seemed totally unrelated to my patch.
Created attachment 8772521 [details]
Bug 1287283 - Only add cpu_percent mozharness step to perfherder if defined

Review commit: https://reviewboard.mozilla.org/r/65316/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65316/
Attachment #8772521 - Flags: review?(gps)
Attachment #8772522 - Flags: review?(gps)
Created attachment 8772522 [details]
Bug 1287283 - Fail job if mozharness perfherder data does not match schema

Review commit: https://reviewboard.mozilla.org/r/65318/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65318/
Try run looks happy, requesting review.
Comment on attachment 8772521 [details]
Bug 1287283 - Only add cpu_percent mozharness step to perfherder if defined

https://reviewboard.mozilla.org/r/65316/#review62658
Attachment #8772521 - Flags: review?(gps) → review+
Comment on attachment 8772522 [details]
Bug 1287283 - Fail job if mozharness perfherder data does not match schema

https://reviewboard.mozilla.org/r/65318/#review62662

\o/
Attachment #8772522 - Flags: review?(gps) → review+

Comment 12

2 years ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c2f35c258091
Only add cpu_percent mozharness step to perfherder if defined r=gps
https://hg.mozilla.org/integration/autoland/rev/abe9d4705864
Fail job if mozharness perfherder data does not match schema r=gps

Comment 13

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c2f35c258091
https://hg.mozilla.org/mozilla-central/rev/abe9d4705864
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox50: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla50
You need to log in before you can comment on or make changes to this bug.