785236 - Add tinderbox threshold for enablePrivilege

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Bobby Holley (:bholley)]

Now that bug 757046, enablePrivilege is officially on life support within our test suite. I think it would make sense to add something to automation to count the number of enablePrivilege calls, and turn the tree orange if it passes a certain threshold. This would prevent new uses from creeping back in, as they did in bug 760802.

Joel, is this something you know how to do? I can write the platform code to spit out the numbers on shutdown if you tell me how to format it.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I think that this could be riddled with trouble.  The main reason for saying that is that we run different tests on each platform, so this threshold would be different for each type of test we run.  This means that M1 will be different than M5 and that will be different between opt and debug and pgo builds, and finally all platforms will have different tests.  

To make matters more confusing if you have a defined threshold value for M1, that set of tests will change in a short amount of time when people add new tests.  The danger is that if we have 30 instances in M1 and some tests get bumped into M2, we might have 28 instances in M1 and 2 new ones in M2.

The easiest way to get a job to turn orange is to add:
TEST-UNEXPECTED-FAIL in the logfile somewhere.

Comment 2

[Bobby Holley (:bholley)]

Thanks for the analysis, Joel. I don't think this is worth the trouble.