1546611 - Fix "None" checks when validating test manifests

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

For streaming tests we cannot always run them through mitmproxy and a recorded playback pageset manifest. As such using the option use_live_sites in the manifest shouldn't require the playback_pageset_manifest and playback_recordings entries:

09:49:37 INFO - raptor-manifest /Users/henrik/code/gecko/testing/raptor/raptor/raptor.ini
09:49:37 INFO - raptor-manifest configuring settings for test raptor-stream-youtube-firefox
09:49:37 INFO - raptor-manifest using live sites so turning playback off!
09:49:37 INFO - raptor-manifest using live sites so appending '-live' to the test name
09:49:37 INFO - raptor-manifest ERROR: setting 'playback_pageset_manifest' is required but not found in /Users/henrik/code/gecko/testing/raptor/raptor/tests/raptor-stream-youtube.ini
09:49:37 INFO - raptor-manifest ERROR: setting 'playback_recordings' is required but not found in /Users/henrik/code/gecko/testing/raptor/raptor/tests/raptor-stream-youtube.ini
09:49:37 INFO - raptor-manifest test raptor-stream-youtube-firefox-live is not valid due to missing settings
09:49:37 INFO - raptor-main abort: no tests found

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Problem is that playback in test_details is None, and that is not covered by the condition above:

https://searchfox.org/mozilla-central/rev/ec489aa170b6486891cf3625717d6fa12bcd11c1/testing/raptor/raptor/manifest.py#76

Other checks suffer from the same problem. So I'm going to fix all of them.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f22775d230c0998fd8233933abaf5a767d7576d

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1546611 - [raptor] Fix None checks when validating test manifests. r=rwood,davehunt

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d4742516f3b5
[raptor] Fix None checks when validating test manifests. r=davehunt

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1546611 - [raptor] Fix checks for invalid manifest entries

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/d4742516f3b5