1585046 - [meta] Expand the raptor-browsertime unit test coverage

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P2)

Description

[Robert Wood [:rwood]]

Add more raptor framework python unit tests. Note: Focus on the browsertime-raptor specific code path, since we may be moving to browsertime soon and eventually removing the raptor webext, control server etc.

Comment 1

[Florin Strugariu [:Bebe]]

Octavian can you take this one please

Comment 2

[Octavian Negru]

Yes, I will work on it after finishing work on Bug 1585966.

Comment 3

[Octavian Negru]

Unit testing can be placed in testing/raptor/test/test_raptor.py file,
since both Raptor and Browsertime components reside in testing/raptor/raptor/raptor.py file.

The unit testing of the Browsertime component can be done with the following steps:
1. Adapt Browsertime to already-written test-cases (add Browsertime to @pytest.mark.parametrize):
-test_build_profile
-test_start_browser
2. Write new test case for run_test method.

Proposal: write tests only for public Browsertime methods. The methods in which Browsertime have its inner logic, like
-remove_mozprofile_delimiters_from_profile
-check_for_crashes
-clean_up
-run_test_setup
-run_test_teardown
must not be tested, because they will make the test cases more brittle.

Note: step 1 is already done and will be submitted along with step 2 in one single commit.

Edit: this comment is outdated. The discussion related to raptor.py tested components will continue here

Comment 4

[Ionuț Goldan [:igoldan]]

I see that Browsertime's integration is mostly spread across 3 main modules:

• raptor.py
• output.py
• results.py

We should split this ticket in 3 subtasks, one for each module.

Comment 5

[Florin Strugariu [:Bebe]]

note:
keep in mind that output.py and results.py will be refactored in the future so maybe we should not spend to much time in these modules.

Comment 6

[Octavian Negru]

OK, I will try not to refactor, but to better shape the code for easy testing.
No breaking changes will be added.

Comment 7

[Robert Wood [:rwood]]

Hey Octavian, are you finished with this bug or are you adding more unit tests? If you're finished please close, thanks!

Comment 8

[Octavian Negru]

Finished, resolved. Thanks for reminding :)