1600668 - Error out if PERFHERDER_DATA is malformed

Status: NEW

(Tree Management :: Perfherder, enhancement, P3)

Description

[Tarek Ziadé (:tarek)]

right now, if PERFHERDER_DATA is not in the expected output, the system will silently ignore it and the task will stay green.

I think we should generate an error in the task and turn it red

Comment 1

[Ionuț Goldan [:igoldan]]

Hi, Tarek! Could you provide some links to check this issue? Thanks!

Comment 2

[Tarek Ziadé (:tarek)]

Hey Ionuț

if you alter any task that runs on try by printing :

print('PERFHERDER_DATA: {"not_the_data_I_hoped_for": 1}')

Some projects do validate the data before it's printed in the stdout, some projects don't
perfherder should validate it using its own schema, and we should raise an error if possible in the task that printed out that line.

Comment 3

[Ionuț Goldan [:igoldan]]

Yes, that's true. I was aware of this issue for some time, but didn't raise it, unfortunately.

Dave, we should consider putting this validation mechanism at a higher level than that of individual perf data producer.
Not sure, but I believe only Talos & Raptor (plus another producer which hasn't been officially registered) are the only fair players which perform this validation. Other perf data producers don't do it.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I believe this was one of the goals of Rob's work to refactor the Perfherder results generation so that there would be a single path to generating the output, where we could add such validation. I agree that we should fail any job that attempts to generate Perfherder data that does not conform to the schema.