Closed Bug 1370607 Opened 2 years ago Closed 2 years ago

Add additional tests to the talos quantum-pageload-e10s suite

Categories

(Testing :: Talos, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox55 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

Details

(Whiteboard: [PI:June])

Attachments

(1 file)

Currently the talos quantum-pageload-e10s test suite consist of a pageload test for google (windows only). Add additional tests to the talos quantum-pageload-e10s suite, expanding to more urls/sites (still for windows only).

Some of the sites (i.e. Facebook) will require login support. This is the current unknown that will require the most talos integration work.

This bug will also include renaming the talos 'Quantum_1' test to 'quantum_pageload_google'. The new tests added will use the same naming convention.

The mitmproxy playback archives created in Bug 1362554 will be used for these new tests.
Comment on attachment 8875428 [details]
Bug 1370607 - Add additional talos tests to the quantum-pageload-e10s suite;

https://reviewboard.mozilla.org/r/146874/#review150938

a few small things, r- for simplifying the test definitions.

::: testing/talos/talos.json:118
(Diff revision 1)
>                  "\"c:/Program Files/Microsoft Windows Performance Toolkit/xperf.exe\""
>              ]
>          },
>          "quantum-pageload-e10s": {
> -            "tests": ["Quantum_1"],
> -            "mitmproxy_recording_set": "mitmproxy-recording-set.zip",
> +            "tests": ["quantum_pageload_google", "quantum_pageload_youtube", "quantum_pageload_amazon", "quantum_pageload_facebook"],
> +            "mitmproxy_recording_set": "mitmproxy-recording-set-win10.zip",

we should file a bug to have a way for different OS flavors to access unique recording sets.  Right now this is OK, but a future bug to handle this will be needed.

::: testing/talos/talos/mitmproxy/mitmproxy-playback-set.manifest:4
(Diff revision 1)
>  [
>      {
> -        "filename": "mitmproxy-recording-set.zip",
> -        "size": 679582,
> +        "filename": "mitmproxy-recording-set-win10.zip",
> +        "size": TODO,

my r+ can carry over when the exact values come in after the tooltool update.

::: testing/talos/talos/test.py:798
(Diff revision 1)
>      lower_is_better = True
>      alert_threshold = 5.0
>  
>  
>  @register_test()
> -class Quantum_1(PageloaderTest):
> +class quantum_pageload_google(PageloaderTest):

I would do this:
class quantum_pageload(PageloaderTest):
  <all the common fields>

class quantum_pageload_google(quantum_pageload):
  tpmanifest = '...'
  
class quantum_pageload_amazon(quantum_pageload):
  tpmanifest = '...'
  
and so forth :)
Attachment #8875428 - Flags: review?(jmaher) → review-
Comment on attachment 8875428 [details]
Bug 1370607 - Add additional talos tests to the quantum-pageload-e10s suite;

https://reviewboard.mozilla.org/r/146874/#review150938

> we should file a bug to have a way for different OS flavors to access unique recording sets.  Right now this is OK, but a future bug to handle this will be needed.

Good idea, filed:

Bug 1371021 - For talos quantum-pageload-e10s suite allow OS flavours to have their own mitmproxy playback sets

> I would do this:
> class quantum_pageload(PageloaderTest):
>   <all the common fields>
> 
> class quantum_pageload_google(quantum_pageload):
>   tpmanifest = '...'
>   
> class quantum_pageload_amazon(quantum_pageload):
>   tpmanifest = '...'
>   
> and so forth :)

Ah yes makes sense, thanks!
Comment on attachment 8875428 [details]
Bug 1370607 - Add additional talos tests to the quantum-pageload-e10s suite;

https://reviewboard.mozilla.org/r/146874/#review150956

looks great
Attachment #8875428 - Flags: review?(jmaher) → review+
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/69a0d95d19bb
Add additional talos tests to the quantum-pageload-e10s suite; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/69a0d95d19bb
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.