Closed Bug 1589070 Opened 3 months ago Closed 3 months ago

Raptor desktop cold load tests are reusing the initial profile and so are not cold loads

Categories

(Testing :: Raptor, defect)

Version 3
defect
Not set

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: acreskey, Assigned: Bebe)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Attached image imgur-cold-load.png

I noticed that when running the following desktop cold page load test (tested OSX and Windows10) that the same profile is used for subsequent cold loads:
./mach raptor-test --test raptor-tp6-imgur-firefox-cold

This leads to network resources being stored in the network cache and thus subsequent page loads are not technically 'cold.'

The attachment shows the browser state before the second cold load -- note that the url is already in the history.

As a result, I'm seeing performance numbers like this which highlight the cold and the warm loads:

 "name": "loadtime", 
          "replicates": [
            2681, 
            1105, 
            1083, 
            1097, 
            1082,  

A new profile is being created each time on each cold browser cycle:

https://searchfox.org/mozilla-central/rev/171109434c6f2fe086af3b2322839b346a112a99/testing/raptor/raptor/raptor.py#876

https://searchfox.org/mozilla-central/rev/171109434c6f2fe086af3b2322839b346a112a99/testing/raptor/raptor/raptor.py#157

I see your patch :bebe, did you confirm that there is a problem with the runner not updating the profile when the new one is created and self.profile is set to that one, is that the issue?

Good find Andrew! Thanks for taking this one :bebe!

FYI when this fix lands it may cause several performance sheriffing alerts for Firefox desktop cold page-load tests on all platforms.

Flags: needinfo?(marian.raiciof)
Flags: needinfo?(igoldan)
Flags: needinfo?(alexandru.ionescu)

Thanks for the info, Robert!

Flags: needinfo?(marian.raiciof)
Pushed by fstrugariu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3536d8186c68
Raptor desktop cold load tests are reusing the initial profile and so are not cold loads r=perftest-reviewers,rwood
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Assignee: nobody → fstrugariu
Regressions: 1589325
Blocks: 1502138
Flags: needinfo?(alexandru.ionescu)
Flags: needinfo?(igoldan)
You need to log in before you can comment on or make changes to this bug.