1528192 - Add an option to use local page recordings instead of tooltool

Status: NEW

(Testing :: Raptor, enhancement, P3)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Currently we attempt to download page recordings from tooltool, and fall back to a local copy if it exists. Occaisionally it's useful to test a new recording, so it would be useful to have a way to skip the tooltool download.

Another (possibly better) approach is to add support for specifying a source for the recordings. This would default to tooltool, but could be replaced with a path. This might be better because an option to skip tooltool would only be useful if we remember to use it on the first run after the new recording is created.

Comment 1

[Aravind Bairi[:ABReddy]]

Hi Dave,
This is my first time working on contributing to Open Source. Can I get some details on how I can start working on this bug.

Comment 2

[Agboola Mukhtar <agbo-ola>]

Hi, I'd like to give this bug a try

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Great to see there's interest in working on this bug! I think it would be a good idea to get :rwood's thoughts on this change, as he's the owner of the Raptor test harness. Let's see how he thinks we might best solve it.

Comment 4

[Robert Wood [:rwood]]

With the current behaviour, when testing a new recording you can just copy the .mp recording file into your local repo obj../testing/raptor folder, and Raptor will automatically find it there. Then you can just use an existing test INI file that has the 'playback_pageset_manifest' option pointing to an existing manifest for an already existing test suite just as a temporary workaround - then once the new recording is verified and uploaded to tooltool just change that test INI setting to the new manifest. So I don't think adding this is really necessary, but can see the appeal.

:davehunt, I'll leave it up to you - if you do wish to go ahead, I would suggest adding a 'playback_pageset_source' test INI setting, that could be "= tooltool' or '= local'. If set to 'local' then just skip the tooltool pageset download completely - and leave the rest as/is and Raptor will just look in the obj.../testing/raptor folder for the .mp file. Makes it a bit simpler than specifying a path in the test INI.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Robert Wood [:rwood] from comment #4)

With the current behaviour, when testing a new recording you can just copy the .mp recording file into your local repo obj../testing/raptor folder, and Raptor will automatically find it there. Then you can just use an existing test INI file that has the 'playback_pageset_manifest' option pointing to an existing manifest for an already existing test suite just as a temporary workaround - then once the new recording is verified and uploaded to tooltool just change that test INI setting to the new manifest. So I don't think adding this is really necessary, but can see the appeal.

When I was testing the new recordings for the mobile sites I put them into obj../testing/raptor but they got replaced when the tests ran with the copies from tooltool. Did I do something wrong? Do I need to modify the playback_pageset_manifest to something that doesn't exist?

:davehunt, I'll leave it up to you - if you do wish to go ahead, I would suggest adding a 'playback_pageset_source' test INI setting, that could be "= tooltool' or '= local'. If set to 'local' then just skip the tooltool pageset download completely - and leave the rest as/is and Raptor will just look in the obj.../testing/raptor folder for the .mp file. Makes it a bit simpler than specifying a path in the test INI.

I think I like the idea of a playback_pageset_source option.

Comment 6

[Robert Wood [:rwood]]

(In reply to Agboola Mukhtar <agbo-ola> from comment #2)

Hi, I'd like to give this bug a try

Hi Agboola, another contributor expressed interest in this same bug a couple of days before you, so this bug is currently taken. However thanks for your interest in contributing to Raptor - and if you see another good-first-bug that you'd like to try out instead, your contribution would be very welcome!

Comment 7

[Robert Wood [:rwood]]

Ok thanks :davehunt.

(In reply to Aravind Bairi[:ABReddy] from comment #1)

Hi Dave,
This is my first time working on contributing to Open Source. Can I get some details on how I can start working on this bug.

Hi Aravind, thanks for your interest in contributing to Mozilla and Raptor!

Do you have a local Firefox development environment setup already? If not, that would be your first step :) Please see 'Step 1' at [1].

Once you have your local dev environment up and running and have done a successful local Firefox build (via './mach build'), the next step would be to get familiar with the Raptor performance framework [2]. Then try running Raptor from your local repo using './mach raptor-test' as noted in the wiki. Once you have Raptor running locally please let me know and I can help point you in the general direction for creating a patch for this issue. Thanks!

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction

[2] https://wiki.mozilla.org/Performance_sheriffing/Raptor

Comment 8

[Aravind Bairi[:ABReddy]]

(In reply to Robert Wood [:rwood] from comment #7)

Ok thanks :davehunt.

(In reply to Aravind Bairi[:ABReddy] from comment #1)

Hi Dave,
This is my first time working on contributing to Open Source. Can I get some details on how I can start working on this bug.

Hi Aravind, thanks for your interest in contributing to Mozilla and Raptor!

Do you have a local Firefox development environment setup already? If not, that would be your first step :) Please see 'Step 1' at [1].

Once you have your local dev environment up and running and have done a successful local Firefox build (via './mach build'), the next step would be to get familiar with the Raptor performance framework [2]. Then try running Raptor from your local repo using './mach raptor-test' as noted in the wiki. Once you have Raptor running locally please let me know and I can help point you in the general direction for creating a patch for this issue. Thanks!

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction

[2] https://wiki.mozilla.org/Performance_sheriffing/Raptor

Thanks :rwood. I already have the local Firedox dev environment. As you have suggested I am going through the Raptor Framework. I would further dive in if you could help me with the next steps.

Comment 9

[Robert Wood [:rwood]]

(In reply to Aravind Bairi[:ABReddy] from comment #8)

Thanks :rwood. I already have the local Firedox dev environment. As you have suggested I am going through the Raptor Framework. I would further dive in if you could help me with the next steps.

Excellent, once you are able to run Raptor locally then you are all set to begin work on this patch. Currently Raptor uses mitmproxy recordings that are stored on a mozilla service called 'tooltool'. You'll see in a Raptor test INI file [0] the name of the tooltool manifest file is specified, which allows the recording to be downloaded from tooltool.

This issue is about also adding an option to use mitmproxy recordings that are already available on the developer's local machine instead of using the 'tooltool' download.

So to support that, let's add a new Raptor test INI option called 'playback_pageset_source'. In the test INI files this will either have a value of 'tooltool' indicating to use the current behaviour, or a value of 'local'. If the test INI file sets the value to 'local', then Raptor should skip downloading the mitmproxy recording from tooltool which is done here [1].

Hint, the Raptor test INI manifest files are read in /raptor/manifest.py, starting at this line for the list of test INI settings required for pageload / playback tests [2].

Hope that helps!

[0] https://searchfox.org/mozilla-central/rev/b36e97fc776635655e84f2048ff59f38fa8a4626/testing/raptor/raptor/tests/raptor-tp6-1.ini#11

[1] https://searchfox.org/mozilla-central/rev/b36e97fc776635655e84f2048ff59f38fa8a4626/testing/raptor/raptor/playback/mitmproxy.py#119

[2] https://searchfox.org/mozilla-central/rev/b36e97fc776635655e84f2048ff59f38fa8a4626/testing/raptor/raptor/manifest.py#21

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Aravind: are you still interested in working on this?

Comment 11

[Nirmay Dhruv [:nirmay]]

Hi [:sparky], I have had a look over the files which require to be changed. I have thought somethings regarding this bug, but am stuck and would require some help.
I don't know how can I access or use variables from a config file into a python file? I haven't done it before, I did search for any config parser used in the code to access or use those variables but am having some difficulty.
Please help me out with this.

Comment 12

[Kimberly Sereduck :kimberlythegeek]

:bebe showed me his process for testing local recordings. While it is not programmatic, it will work in the interim for anyone want to test a local recording file.

Modify these lines to the mitmproxy version used for recording (found under "proxy_version" in metadata.json in the recording zip file), and replace playback_files with the path to zip file (must remain an array)

example:

"playback_version": "6.0.2",
"playback_files": ["/Users/ksereduck/Workspace/mozilla-central/artifacts/mitm6-darwin-firefox-amazon.zip"],