513350 - Validator should indicate when saved results are stale due to test changes

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect)

Description

[Dave Garrett]

If any of the add-on validator's tests are added, removed, or changed the saved results from an old run should be reset or labeled to be out of date. Currently, there is no indication that old saved results are no longer valid.

Comment 1

[Andrew Williamson [:eviljeff]]

I can confirm this.
I think it could be leading to some security issues being missed.  
e.g.
with this (https://addons.mozilla.org/en-US/developers/versions/validate/79310)
it looked like this on the review page:
"Validation results for discountfinder-1.0.3-fx+sm.xpi
7 passes

    * General 3 Running test...
    * Security 3 Running test...
    * L10n No test results found. Running test...
    * Extension-specific security 1 "
and looked like this when I pressed validate afresh:
"  65 warnings   8 passes

    * General 3 Running test...
    * Security 64 4 Running test...
    * L10n 1 Running test...
    * Extension-specific security 1 "

0 warnings turned in 64 warnings.

Comment 2

[Wil Clouser [:clouserw]]

The original (and current, I guess) design expires test results after 1 hour.  In practice, they'd actually be expiring once per day because that's when the gc script runs.  However, there was a typo in the gc script and it hasn't been running.  Typo is fixed in r54023 and cached results will be deleted daily (close to midnight) which should be fine for this bug.

Comment 3

[Dave Garrett]

All of my old test results are still showing on production. Is there a push still needed for this or is it supposed to be fixed already?

Comment 4

[Dave Garrett]

It's fixed on production now.