1449184 - Add support for raptor test timeouts

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Robert Wood [:rwood]]

Need timeout logic to handle when test pages don't load (i.e. 404) or when the page loads but not all measurements are received.

Have a default timeout value but allow it to be over-ridden on a per-test basis.

It would be ideal to have a unit test that loads a test page but then intentionally doesn't receive the expected events/measurements.

Comment 1

[Robert Wood [:rwood]]

Attachment: https://github.com/rwood-moz/raptor/pull/19

Basic pageload timeout support for raptor firefox and chrome

Comment 2

[Robert Wood [:rwood]]

Comment on attachment 8962868 [details] [review]
https://github.com/rwood-moz/raptor/pull/19

I'll file a separate bug for adding a test for this (as I'm not even sure if end-to-end pytest/unit tests are possible).

Comment 3

[Andrew Halberstadt [:ahal]]

Yes, end-to-end tests are definitely possible, it would involve running the 'raptor' command in a subprocess. It will be a bit of a pain to set up, the tricky bits will be getting 'firefox' and 'chrome' binaries to test. There's some prior art in mozilla-central for this:
https://searchfox.org/mozilla-central/source/testing/mochitest/tests/python/test_basic_mochitest_plain.py#21

In order to find a firefox binary, that uses this fixture (which is conveniently already vendored into this repo):
https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/selftest/fixtures.py#97

Though it currently only works in mozilla-central.

Comment 4

[Andrew Halberstadt [:ahal]]

Comment on attachment 8962868 [details] [review]
https://github.com/rwood-moz/raptor/pull/19

Thanks, lgtm.

Comment 5

[Robert Wood [:rwood]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Yes, end-to-end tests are definitely possible, it would involve running the
> 'raptor' command in a subprocess. It will be a bit of a pain to set up, the
> tricky bits will be getting 'firefox' and 'chrome' binaries to test. There's
> some prior art in mozilla-central for this:
> https://searchfox.org/mozilla-central/source/testing/mochitest/tests/python/
> test_basic_mochitest_plain.py#21
> 
> In order to find a firefox binary, that uses this fixture (which is
> conveniently already vendored into this repo):
> https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/
> selftest/fixtures.py#97
> 
> Though it currently only works in mozilla-central.

Cool thanks, I filed Bug 1449572 and I'll follow-up on this there.

Comment 6

[Robert Wood [:rwood]]

Merged: https://github.com/rwood-moz/raptor/commit/3d7367d2197ab47f983b61e8659639099fd694d1