619536 - turn off ts_cold tests for all platforms

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dietrich Ayala (:dietrich)]

the ts_cold tests as currently implemented are not providing useful data, and are therefore:

1. wasting resources

2. confusing the reporting of performance results of check-ins

Mac and Linux ts_cold were created in bug 510587, and bug 522807 is for windows.

i'll file a new bug for implementing these tests in a way that makes sense.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Grabbing, because aiui, this looks like it should be a RelEng bug to disable this test suite in production. If I've misunderstood, let me know.

We'll need to announce this before turning off, but great to know we can. Thanks Dietrich.

Comment 2

[Robert Sayre]

(In reply to comment #0)
> the ts_cold tests as currently implemented are not providing useful data

I believe it. What's the rationale, though?

Comment 3

[(dormant account)]

(In reply to comment #2)
> (In reply to comment #0)
> > the ts_cold tests as currently implemented are not providing useful data
> 
> I believe it. What's the rationale, though?

It's counterproductive as deployed. It measures the wrong overhead on osx(ie libraries don't get flushed), and a lot of linux overhead is in loading system libraries which are normally in the cache. These issues combined with relatively non-deterministic seek delays on harddrives makes this test noisy and confusing.

It is possible to make this test less broken, but it will be much more useful to switch from the ts_cold measurement to measuring amount of disk io, ala bug 609111.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) No objections to disabling in today's platform meeting.

2) I've send a post to dev.planning and dev.tree-management about this.

Comment 5

[Chris Cooper [:coop] (he/him)]

Attachment: Turn off ts_cold tests

Comment 6

[Armen [:armenzg]]

Comment on attachment 499368 [details] [diff] [review]
Turn off ts_cold tests

stamp

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 499368 [details] [diff] [review]
Turn off ts_cold tests

https://hg.mozilla.org/build/buildbot-configs/rev/fbede0bc8f0e