1287283 - Jobs are generating an "error in perfherder data, ignoring"

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[William Lachance (:wlach)]

Sometime in the last few days, jobs started spitting out errors like this:

```
error while validating Perfherder data; ignoring
Traceback (most recent call last):
04:38:56 ERROR -
File "/builds/slave/test/scripts/mozharness/base/python.py", line 644, in _log_resource_usage
jsonschema.validate(data, schema)
04:38:56 ERROR -
File "/builds/slave/test/build/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 478, in validate
cls(schema, *args, **kwargs).validate(instance)
04:38:56 ERROR -
File "/builds/slave/test/build/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
raise error
04:38:56 ERROR -
ValidationError: None is not of type u'number'
04:38:56 ERROR -
Failed validating u'type' in schema[u'properties'][u'suites'][u'items'][u'properties'][u'subtests'][u'items'][u'properties'][u'value']:
{u'description': u'Summary value for subtest',
u'title': u'Subtest value',
u'type': u'number'}
04:38:56 ERROR -
On instance[u'suites'][2][u'subtests'][1][u'value']:
None
04:38:56 ERROR -
```

I'm guessing this is due to some kind of change in the build system that's causing the perfherder output to fail.

In addition to fixing whatever the problem is, IMO we should just turn the jobs orange if this kind of validation fails (note that we're not yet storing this data, but we probably want to), otherwise it's just going to keep breaking.

Comment 1

[William Lachance (:wlach)]

:gps, do you know what the cause of this is? I'd like to start experimenting with storing this information, at least on treeherder stage.

Comment 2

[Gregory Szorc [:gps]]

AFAIK nothing related to this changed. Was the JSON schema updated?

If someone other than you reported the bug, I would have flagged *you* for needinfo ;)

Comment 3

[William Lachance (:wlach)]

Did some debugging on mochitest-1, looks like we're generating the following (based on adding some debugging info to this push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7223e726213&selectedJob=24139108)

{
	"framework": {
		"name": "job_resource_usage"
	},
	"suites": [{
		"subtests": [{
			"name": "cpu_percent",
			"value": 10.57951977401129
		}, {
			"name": "io_write_bytes",
			"value": 870832640
		}, {
			"name": "io.read_bytes",
			"value": 37296128
		}, {
			"name": "io_write_time",
			"value": 41616420
		}, {
			"name": "io_read_time",
			"value": 318910
		}],
		"extraOptions": [],
		"name": "mochitest.plain-chunked.1.overall"
	}, {
		"subtests": [{
			"name": "time",
			"value": 2.248000144958496
		}, {
			"name": "cpu_percent",
			"value": 10.943750000000001
		}],
		"name": "mochitest.plain-chunked.1.install"
	}, {
		"subtests": [{
			"name": "time",
			"value": 0.0
		}, {
			"name": "cpu_percent",
			"value": null
		}],
		"name": "mochitest.plain-chunked.1.stage-files"
	}, {
		"subtests": [{
			"name": "time",
			"value": 710.164999961853
		}, {
			"name": "cpu_percent",
			"value": 10.587216312056727
		}],
		"name": "mochitest.plain-chunked.1.run-tests"
	}]
}

Perfherder really doesn't like the fact that we're setting cpu_percent to null for mochitest.plain-chunked.1.stage-files. Need to look into why that's happening. It appears that it started with this push from ahal: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4a2a06f7dad65d5b1c4c6a7ad717b9ca8354141b, but need to do some retriggers to verify that's what's happening.

In any event I'm going to make perfherder validation of this data *mandatory* in the future so it's someone else's damn job to fix this if it breaks again. :)

Comment 4

[William Lachance (:wlach)]

Yeah in fact it was bug 1285553, it looks like that adds a "stage files" step which is so brief that it doesn't have a cpu percentage. Let's just do a null check against cpu_percent before adding any mozharness step to perfherder data.

Comment 5

[William Lachance (:wlach)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=44b279717fbd

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to William Lachance (:wlach) from comment #3)
> In any event I'm going to make perfherder validation of this data
> *mandatory* in the future so it's someone else's damn job to fix this if it
> breaks again. :)

I don't know.. I kind of like the setup now ;)

Sorry, I did see the error but I assumed it was an existing issue as it seemed totally unrelated to my patch.

Comment 7

[William Lachance (:wlach)]

Attachment: Bug 1287283 - Only add cpu_percent mozharness step to perfherder if defined

Review commit: https://reviewboard.mozilla.org/r/65316/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65316/

Comment 8

[William Lachance (:wlach)]

Attachment: Bug 1287283 - Fail job if mozharness perfherder data does not match schema

Review commit: https://reviewboard.mozilla.org/r/65318/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65318/

Comment 9

[William Lachance (:wlach)]

Try run looks happy, requesting review.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8772521 [details]
Bug 1287283 - Only add cpu_percent mozharness step to perfherder if defined

https://reviewboard.mozilla.org/r/65316/#review62658

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8772522 [details]
Bug 1287283 - Fail job if mozharness perfherder data does not match schema

https://reviewboard.mozilla.org/r/65318/#review62662

\o/

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c2f35c258091
Only add cpu_percent mozharness step to perfherder if defined r=gps
https://hg.mozilla.org/integration/autoland/rev/abe9d4705864
Fail job if mozharness perfherder data does not match schema r=gps

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c2f35c258091
https://hg.mozilla.org/mozilla-central/rev/abe9d4705864