1597862 - Compare and vet Browsertime tp6-1-c regular vs. conditioned-profile tp6-c-1-condprof runs (on at least OS X)

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

(Thanks to :sparky for flagging this during review.)

This chiefly applies to tp6-1-c, for both Raptor and Browsertime:

1. So far, for Raptor, it's looking like the difference between conditioned profile-enabled and stock/default runs are minimal/acceptable (https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&selected=NaN,2152998,594748,976954143&series=autoland,2152998,1,10&series=autoland,2052696,1,10&timerange=1209600

2. However, for Browsertime (which hasn't-yet landed: bug 1589267), we need to take a closer look at the differences between conditioned-profile Btime(tp6-1-c-condprof) and Btime(tp6-1-c), which appear to be greater, statistically-speaking.

Ionut, would you mind helping take a look/ help us vet both of the above comparison(s), here? Thx! I'm quite new to these comparisons in Perfherder, and want to get them right.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=277028931&tier=1%2C2%2C3&revision=f75ee1d0ed66fc229dc1368fafa2444d1b79cd1e
[1] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=276817910&tier=1%2C2%2C3&revision=f75ee1d0ed66fc229dc1368fafa2444d1b79cd1e

Comment 1

[Ionuț Goldan [:igoldan]]

I'm having some hard time with Perfherder.
I cannot do the comparison ATM, as the web app freezes on me. Probably because of bug 1597136.

Will reply back once Perfherder is properly functioning.

Comment 2

[Ionuț Goldan [:igoldan]]

(In reply to Stephen Donner [:stephend] from comment #0)

[...]
Ionut, would you mind helping take a look/ help us vet both of the above comparison(s), here? Thx! I'm quite new to these comparisons in Perfherder, and want to get them right.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=277028931&tier=1%2C2%2C3&revision=f75ee1d0ed66fc229dc1368fafa2444d1b79cd1e
[1] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=276817910&tier=1%2C2%2C3&revision=f75ee1d0ed66fc229dc1368fafa2444d1b79cd1e

There's no distinction between the PERHERDER_DATA dumps. I did 2 separate Try pushes, based on your patch, requesting the jobs separately, to be able to compare.
This is the associated Compare view for them.

I'm keeping the needinfo? so I can come back & interpret the results.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Thx a million! I've filed bug 1598183 so Browsertime signatures/annotations are distinguishable in Perfherder.

Comment 4

[Ionuț Goldan [:igoldan]]

Attachment: tp6-1-c noise levels

Browsertime tp6-1-c regular seems more noisy (has a standard deviation of 5.6% & runs for 25 minutes)

Comment 5

[Ionuț Goldan [:igoldan]]

Attachment: tp6-1-c-condprof

than conditioned-profile tp6-c-1-condprof (which has a standard deviation of 2.37% and runs for 12 minutes).
I also noticed that this test runs considerably faster than the regular one.

Comment 6

[Robert Wood [:rwood]]

Hi Stephen is this work ongoing (is this still a P1?) Thanks!

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Robert Wood [:rwood] from comment #6)

Hi Stephen is this work ongoing (is this still a P1?) Thanks!

Sorry for leaving this open; no, in addition to Ionuț's comparison, bug 1600838 landing, making conditioned profile the default for both Raptor and Browsertime, for desktop Firefox, and :sparky's data-analysis work as well, I think we're good here.