Closed Bug 1561939 (browsertime-mvp) Opened 5 years ago Closed 5 years ago

[meta] MVP for browsertime on Windows 10 in CI

Categories

(Testing :: Performance, task)

Product:
Testing
Component:
Performance
Version:
Version 3
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rwood, Unassigned)

References

(Blocks 3 open bugs)

Details

(Keywords: meta)

Attachments

(2 obsolete files)

Build a single page-load test suite (i.e. tp6-1) that uses browsertime with mitmproxy, and the existing mitmproxy recordings; running against Firefox desktop on Win10; in our existing taskcluster CI on the same hardware that raptor currently uses; reporting to perfherder.

Goal is to have data to compare browsertime vs raptor and see if it is viable to switch to browsertime.

Type: defect → task

One technical approach to this might be to lift the mozproxy bits out of testing/raptor/raptor.py for re-use, since configuring the proxy really isn't specific to the test runner; and then adding the equivalent of testing/browsertime/browsertime.py. I imagine that the proxy handling is part of the test runner right now because we need the test details to determine the playback archive to download with tooltool. Can that lift up to the mozharness raptor.py level, so that it controls the proxy fetching, etc, and just invokes the test runner with the correct environment?

Digging in further, I see that this is swimming hard against the current architecture. So is the right technical approach to generalize raptor/raptor.py into a more generic runner, with Browsertime{Desktop,Android} that handle the invocation of browsertime (in the same way that Raptor{Desktop,Android} handle starting the control server, etc)?

This probably depends on Bug 1563074.

Depends on: 1563074

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

This probably depends on Bug 1563074.

I don't think this is a hard-blocker, but agree that this would be ideal.

This is the equivalent of .pth in virtualenv declarations. It sets up
a .egg file with a sys.path addition, as far as I can tell.

Depends on D37128

This mechanical transform lays the foundation for generalizing the
underlying test runner.

Depends on D37131

Just tossing out some starting quality of life improvements and mechanical refactorings. Rob, see if they look sensible to you?

Next steps here would be to lift the proxy handling up to Perftest (or better yet, out to a little manager) and (if we're feeling gung-ho) get rid of the use of config for all these variables through to mozproxy.

After that we can add Browsertime* classes and implement run_test as appropriate. I'm still not 100% on the best way to install browsertime; ./mach browsertime --setup really leans into mach in a way that doesn't make sense in a mozharness environment. Maybe we lift the functionality out to a mozbase accessible module and do the install in testing/raptor/raptor.py in automation, and use the in-tree install in tools/browsertime locally?

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

Just tossing out some starting quality of life improvements and mechanical refactorings. Rob, see if they look sensible to you?

Next steps here would be to lift the proxy handling up to Perftest (or better yet, out to a little manager) and (if we're feeling gung-ho) get rid of the use of config for all these variables through to mozproxy.

Yep sounds good to me!

After that we can add Browsertime* classes and implement run_test as appropriate. I'm still not 100% on the best way to install browsertime; ./mach browsertime --setup really leans into mach in a way that doesn't make sense in a mozharness environment. Maybe we lift the functionality out to a mozbase accessible module and do the install in testing/raptor/raptor.py in automation, and use the in-tree install in tools/browsertime locally?

Also sounds like a good plan!

Keywords: leave-open

(In reply to Nick Alexander :nalexander [he/him] from comment #5)

Created attachment 9076310 [details]
Bug 1561939 - Part 1: Extract Perftest abstract base class. r?rwood

This mechanical transform lays the foundation for generalizing the
underlying test runner.

Depends on D37131

Can you post a try push for this please Nick? Including Raptor on Firefox/Chromium/Android, thanks!

Flags: needinfo?(nalexander)
Flags: needinfo?(nalexander)
Summary: [meta] MVP for browsertime → [meta] MVP for browsertime on Windows 10 in CI
Depends on: 1565316
Depends on: 1566166
Depends on: 1566171
Depends on: 1566174
Attachment #9076310 - Attachment description: Bug 1561939 - Part 1: Extract `Perftest` abstract base class. r?rwood → Bug 1561939 - Extract `Perftest` abstract base class. r?rwood
Alias: browsertime-mvp
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3125e12eee29
Pre: Make it easier to work with mozbase in Raptor locally. r=ahal
https://hg.mozilla.org/integration/autoland/rev/2833ba67a5b9
Extract `Perftest` abstract base class. r=rwood,perftest-reviewers

(In reply to Pulsebot from comment #9)

Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3125e12eee29
Pre: Make it easier to work with mozbase in Raptor locally. r=ahal
https://hg.mozilla.org/integration/autoland/rev/2833ba67a5b9
Extract Perftest abstract base class. r=rwood,perftest-reviewers

Ah, no, I pushed under the wrong bug number! I will try to fix.

Backout by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c2a690d8461c
Backed out 2 changesets as per nalexander`s request.

(In reply to Pulsebot from comment #11)

Backout by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c2a690d8461c
Backed out 2 changesets as per nalexander`s request.

And I'll re-land under Bug 1564282 as soon as I can figure out how. Sorry, folks!

Attachment #9076309 - Attachment description: Bug 1561939 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal → Bug 1564282 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal

Comment on attachment 9076309 [details]
Bug 1561939 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal

Revision D37131 was moved to bug 1564282. Setting attachment 9076309 [details] to obsolete.

Attachment #9076309 - Attachment is obsolete: true

Comment on attachment 9076310 [details]
Bug 1561939 - Extract Perftest abstract base class. r?rwood

Revision D37132 was moved to bug 1564282. Setting attachment 9076310 [details] to obsolete.

Attachment #9076310 - Attachment is obsolete: true
Attachment #9076309 - Attachment description: Bug 1564282 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal → Bug 1561939 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal
Attachment #9076309 - Attachment is obsolete: false
Attachment #9076310 - Attachment is obsolete: false

Comment on attachment 9076309 [details]
Bug 1561939 - Pre: Make it easier to work with mozbase in Raptor locally. r?ahal

Revision D37131 was moved to bug 1564282. Setting attachment 9076309 [details] to obsolete.

Attachment #9076309 - Attachment is obsolete: true

Comment on attachment 9076310 [details]
Bug 1561939 - Extract Perftest abstract base class. r?rwood

Revision D37132 was moved to bug 1564282. Setting attachment 9076310 [details] to obsolete.

Attachment #9076310 - Attachment is obsolete: true

Thank you for the investigations.

Flags: needinfo?(nalexander)
Flags: needinfo?(aryx.bugmail)
Flags: needinfo?(mh+mozilla)
Depends on: 1572804
Depends on: 1568048
Blocks: 1578749
Blocks: 1585010
Status: NEW → RESOLVED
Closed: 5 years ago
Keywords: leave-open
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: