Closed Bug 1532365 Opened 1 year ago Closed 3 months ago

Add a Raptor manifest unit test for use_live_sites

Categories

(Testing :: Raptor, enhancement, P3)

Version 3
enhancement

Tracking

(firefox77 fixed)

RESOLVED FIXED
mozilla77
Tracking Status
firefox77 --- fixed

People

(Reporter: rwood, Assigned: myeongjun.ko, Mentored)

References

Details

(Whiteboard: [lang=py])

User Story

See here for information on how to contribute: https://wiki.mozilla.org/TestEngineering/Performance/NewContributors

Attachments

(1 file)

Bug 1531169 lands the ability to run Raptor page-load tests using live sites. Add a new unit test in test-manifest.py [0] to test get_browser_test_list where a test INI has 'use_live_sites = true' and verify the resulting test manifest settings has 'playback' set to None, etc.

May be a bit of an overhaul to test_manifest.py to allow it to use a produced/mock test INI file (that has 'use_live_sites = true') instead of using a real test INI file for the unit tests.

[0] https://searchfox.org/mozilla-central/source/testing/raptor/test/test_manifest.py

Mentor: gmierz2
User Story: (updated)

Hello :)
I'm reading code and contents to solve this issue. I have some questions.
When use_live_sites is true, 'playback' be set to None already. [0]
Do It need verification of this?

Also, raptor-youtube-playback.ini[1] having use_live_sites flag doesn't have measure values. So an error occurs. [2]
Is this ini file intended?
Please let me know if there's anything I'm misunderstanding.

[0] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#448
[1] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/tests/benchmarks/raptor-youtube-playback.ini
[2] https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#143

(In reply to Myeongjun Go from comment #1)

Hello :)
I'm reading code and contents to solve this issue. I have some questions.
When use_live_sites is true, 'playback' be set to None already. [0]
Do It need verification of this?

Hello Myeongjun! Yes, exactly, you need to verify that playback is None when use_live_sites is True.

Also, raptor-youtube-playback.ini[1] having use_live_sites flag doesn't have measure values. So an error occurs. [2]
Is this ini file intended?
Please let me know if there's anything I'm misunderstanding.

I'm not sure how you are calling this code or what you are calling it with, but if there is anything in the measure field, it won't get replaced here and it would fail: https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#127-129

You're going to want to run the test using this function (which sets the playback to None when use_live_sites is True: https://dxr.mozilla.org/mozilla-central/rev/a6a5a4f31ea26906e584e162b8d9485e6cacca32/testing/raptor/raptor/manifest.py#287

Hello Myeongjun! How's everything going for you?

If this bug is too complex or missing information, we can find another bug to hack on. :)

Flags: needinfo?(myeongjun.ko)

Sorry for the late reply.
I'm preparing to submit a patch now :)

Flags: needinfo?(myeongjun.ko)
Assignee: nobody → myeongjun.ko
Status: NEW → ASSIGNED
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/e98c8caadb97
Add a Raptor manifest unit test for use_live_sites r=sparky,perftest-reviewers
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla77
You need to log in before you can comment on or make changes to this bug.