1100444 - Update web-platform-test manifest expected results when Aurora 35 merges to Beta

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

We know from bug 1066621 that some w-p-t will fail when they hit Beta due to feature disablings. Currently, there's no way to selectively annotate a test or to pref on a feature at runtime within that harness.

As a result, we'll need to include a manifest update at the time of the merge to avoid the perma-fail. I'm attaching a patch here that should take care of it.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Attachment: Test data updates

Comment 2

[Rail Aliiev [:rail]]

Can we disable the test based on environment variable MOZ_DEV_EDITION set to 1?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #2)
> Can we disable the test based on environment variable MOZ_DEV_EDITION set to
> 1?

Comment 4

[James Graham [:jgraham]]

Currently no.

Disabling the tests seems strictly worse than running the tests with the correct branch-specific expectations in any case.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to James Graham [:jgraham] from comment #4)
> Currently no.
> 
> Disabling the tests seems strictly worse than running the tests with the
> correct branch-specific expectations in any case.

Whatever the right thing to do is, it needs to be do-able via mozconfigs. We're landing the patch from this bug on this cycle as an exception, but going forward we need to be able to enable/disable things with our automation. Landing random patches is not how to deal with changes at uplift.

Comment 6

[James Graham [:jgraham]]

So I don't really know how that's possible.

We take a bunch of tests from upstream that may depend on any platform feature. In general we don't know up-front which of these tests will be affected by differences between nightly and aurora or aurora and beta. Therefore, in general, we will have to either land a patch to change the expectation for any tests that do turn out to be affected, or land a patch to disable the tests that will be affected.