1532365 - Add a Raptor manifest unit test for use_live_sites

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P3)

Description

[Robert Wood [:rwood]]

Bug 1531169 lands the ability to run Raptor page-load tests using live sites. Add a new unit test in test-manifest.py [0] to test get_browser_test_list where a test INI has 'use_live_sites = true' and verify the resulting test manifest settings has 'playback' set to None, etc.

May be a bit of an overhaul to test_manifest.py to allow it to use a produced/mock test INI file (that has 'use_live_sites = true') instead of using a real test INI file for the unit tests.

[0] https://searchfox.org/mozilla-central/source/testing/raptor/test/test_manifest.py

Comment 1

[Myeongjun Go]

Hello :)
I'm reading code and contents to solve this issue. I have some questions.
When use_live_sites is true, 'playback' be set to None already. [0]
Do It need verification of this?

Also, raptor-youtube-playback.ini[1] having use_live_sites flag doesn't have measure values. So an error occurs. [2]
Is this ini file intended?
Please let me know if there's anything I'm misunderstanding.

[0] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#448
[1] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/tests/benchmarks/raptor-youtube-playback.ini
[2] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#143

Comment 2

[Greg Mierzwinski [:sparky]]

(In reply to Myeongjun Go from comment #1)

Hello :)
I'm reading code and contents to solve this issue. I have some questions.
When use_live_sites is true, 'playback' be set to None already. [0]
Do It need verification of this?

Hello Myeongjun! Yes, exactly, you need to verify that playback is None when use_live_sites is True.

Also, raptor-youtube-playback.ini[1] having use_live_sites flag doesn't have measure values. So an error occurs. [2]
Is this ini file intended?
Please let me know if there's anything I'm misunderstanding.

I'm not sure how you are calling this code or what you are calling it with, but if there is anything in the measure field, it won't get replaced here and it would fail: https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#127-129

You're going to want to run the test using this function (which sets the playback to None when use_live_sites is True: https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#287

Comment 3

[Greg Mierzwinski [:sparky]]

Hello Myeongjun! How's everything going for you?

If this bug is too complex or missing information, we can find another bug to hack on. :)

Comment 4

[Myeongjun Go]

Sorry for the late reply.
I'm preparing to submit a patch now :)

Comment 5

[Myeongjun Go]

Attachment: Bug 1532365 - Add a Raptor manifest unit test for use_live_sites r=sparky

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e98c8caadb97
Add a Raptor manifest unit test for use_live_sites r=sparky,perftest-reviewers

Comment 7

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/e98c8caadb97