Open Bug 1600668 Opened 1 year ago Updated 8 months ago

Error out if PERFHERDER_DATA is malformed

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: tarek, Unassigned)

Details

right now, if PERFHERDER_DATA is not in the expected output, the system will silently ignore it and the task will stay green.

I think we should generate an error in the task and turn it red

Hi, Tarek! Could you provide some links to check this issue? Thanks!

Flags: needinfo?(tarek)

Hey Ionuț

if you alter any task that runs on try by printing :

print('PERFHERDER_DATA: {"not_the_data_I_hoped_for": 1}')

Some projects do validate the data before it's printed in the stdout, some projects don't
perfherder should validate it using its own schema, and we should raise an error if possible in the task that printed out that line.

Flags: needinfo?(tarek)

Yes, that's true. I was aware of this issue for some time, but didn't raise it, unfortunately.

Dave, we should consider putting this validation mechanism at a higher level than that of individual perf data producer.
Not sure, but I believe only Talos & Raptor (plus another producer which hasn't been officially registered) are the only fair players which perform this validation. Other perf data producers don't do it.

Flags: needinfo?(dave.hunt)

I believe this was one of the goals of Rob's work to refactor the Perfherder results generation so that there would be a single path to generating the output, where we could add such validation. I agree that we should fail any job that attempts to generate Perfherder data that does not conform to the schema.

Flags: needinfo?(dave.hunt)
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.