1168540 - [meta] WPT referrer policy tests

Status: RESOLVED DUPLICATE of bug 1409600

(Core :: DOM: Security, defect, P3)

Description

[Franziskus Kiefer [:franziskus]]

I ran the referrer policy tests from w3c [1] and it looks mixed. On nightly I get 143 pass, 102 fail, and 236 timeout. I'm not sure what the reason for the timouts is, but the fails look legit, e.g.

assert_equals: Reported Referrer URL is 'stripped-referrer'. expected (string) "http://web-platform.test:8000/referrer-policy/no-referrer-when-downgrade/http-csp/same-origin/http-http/worker-request/insecure-protocol.http.html" but got (undefined) undefined


[1] https://github.com/w3c/web-platform-tests/tree/master/referrer-policy

Comment 1

[Franziskus Kiefer [:franziskus]]

Bug filed for document.origin

https://github.com/w3c/web-platform-tests/issues/1886

Comment 2

[Franziskus Kiefer [:franziskus]]

* https://github.com/w3c/web-platform-tests/issues/1960
* https://github.com/w3c/web-platform-tests/issues/1961

Comment 3

[Thomas Nguyen (:tnguyen)]

I tried to the referrer policy tests and here's my result: 589 pass, 33 fail, and 680 timeout.
At the first look into 33 failed cases:
- Bug 1260664: 1 case fails which refers to referrerpolicy attribute. I think it should be referrerPolicy (with capital P) in the test
- Bug 1223838: currently perElementReferrer is disabled by default, therefore setting referrerpolicy attribute in element takes no effect.
- Lbnl, in these web platform tests, we change referrer policy by setting referrerpolicy content attribute.

      var elementAttributesForDeliveryMethod = {
        "attr-referrer":  {referrerpolicy: t._scenario.referrer_policy},
        "rel-noreferrer": {rel: "noreferrer"}
      };

 Looking at this for a while and I found it totally has no effect. I have to change it to referrerPolicy (capital P).
Is there a capitalization issue with this feature, or our implementation is missing? Am I missing any point?
[1] spec and probably [2] may refer to this issue
[1] https://w3c.github.io/webappsec-referrer-policy/#referrer-policy-delivery-referrer-attribute
[2] https://w3c.github.io/webappsec-referrer-policy/#html-iframe-element-attributes
I still don't know why there's too much timeout tests :(

Comment 4

[Franziskus Kiefer [:franziskus]]

(In reply to Thomas Nguyen[:tnguyen][:thomas][:nguyen] from comment #3)
> I tried to the referrer policy tests and here's my result: 589 pass, 33
> fail, and 680 timeout.

Timeouts come from certificate issues in web-platform-tests. You have to either import the certificate or manually accept it before running the tests (also see [4]).

> At the first look into 33 failed cases:
> - Bug 1260664: 1 case fails which refers to referrerpolicy attribute. I
> think it should be referrerPolicy (with capital P) in the test

> - Lbnl, in these web platform tests, we change referrer policy by setting
> referrerpolicy content attribute.
> 
>       var elementAttributesForDeliveryMethod = {
>         "attr-referrer":  {referrerpolicy: t._scenario.referrer_policy},
>         "rel-noreferrer": {rel: "noreferrer"}
>       };
> 
>  Looking at this for a while and I found it totally has no effect. I have to
> change it to referrerPolicy (capital P).

The content attribute is referrerpolicy. Only the IDL attribute is referrerPolicy (see [1][2][3], the example in [3] looks wrong).

> - Bug 1223838: currently perElementReferrer is disabled by default,
> therefore setting referrerpolicy attribute in element takes no effect.

this is probably the most important thing to do first: enable per element referrer before running the tests. otherwise a lot will obviously fail (just flip the pref manually before starting the tests).

[1] https://github.com/w3c/webappsec-referrer-policy/issues/3
[2] https://rawgit.com/w3c/webappsec-referrer-policy/master/index.html#element-interface-extensions
[3] https://rawgit.com/w3c/webappsec-referrer-policy/master/index.html#referrer-policy-delivery-referrer-attribute
[4] https://github.com/w3c/web-platform-tests/issues/1961

Comment 5

[Thomas Nguyen (:tnguyen)]

Thanks Frankziskus. Let me try importing certificate and run again.

Comment 6

[Thomas Nguyen (:tnguyen)]

Thanks Franziskus, ran again ( must enable perElementReferrer), and here's result: total 77 failed cases and 1225 passed cases
- Bug 1260664: 1 failed case in total of 77 cases fail
- Bug 1178337: 28 failed cases in total of 77 cases fail
- Bug 1261298: 38 failed cases in total of 77 cases fail
- Bug 1261003: 38 failed cases in total of 77 cases fail
( 1 failed case may contain several different reasons). I will look at all above dependency bugs before running this again.

Comment 7

[Thomas Nguyen (:tnguyen)]

> - Bug 1261003: 38 failed cases in total of 77 cases fail
Edit, should be: Bug 1261003: 76 failed cases in total of 77 cases fail

Comment 8

[Ethan Tseng [:ethan]]

Hi Franziskus,

Just want to confirm it with you.
All sub-bugs for this bug were resolved. Can we mark this bug as resolved as well?
Should we be aware of anything else for this bug?

Comment 9

[Franziskus Kiefer [:franziskus]]

hm I thought we could close this because most things should be working. But looking at a recent m-c run [1] some tests are disabled. So I think there're two things we should look into before potentially closing this.

* img tests are not enabled yet (see discussions in bug 1265864 and related)
* there are some tests that are not enabled at the moment I think (see [1]). We should investigate why and potentially enable them.

I think we should keep this open to track progress on those things but I wouldn't consider it blocking to enabling the referrer policy attribute (we have basic coverage and know why some are disabled).

[1] https://archive.mozilla.org/pub/firefox/nightly/2016/05/2016-05-25-06-37-10-mozilla-central/mozilla-central_ubuntu32_vm_test_pgo-web-platform-tests-4-bm01-tests1-linux32-build2.txt.gz

Comment 10

[Ethan Tseng [:ethan]]

(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #9)

Franziskus, thanks a lot for your insightful information!
 
> * img tests are not enabled yet (see discussions in bug 1265864 and related)
> * there are some tests that are not enabled at the moment I think (see [1]).
> We should investigate why and potentially enable them.
Okay. Thomas and I will keep track on this bug and img tests.

> I think we should keep this open to track progress on those things but I
> wouldn't consider it blocking to enabling the referrer policy attribute (we
> have basic coverage and know why some are disabled).
Sure. Thanks!

Comment 11

[Thomas Nguyen (:tnguyen)]

Thanks Ethan and Franziskus

(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #9)
> hm I thought we could close this because most things should be working. But
> looking at a recent m-c run [1] some tests are disabled. So I think there're
> two things we should look into before potentially closing this.
> 
> * img tests are not enabled yet (see discussions in bug 1265864 and related)
> * there are some tests that are not enabled at the moment I think (see [1]).
> We should investigate why and potentially enable them.

I looked into the bug 1265864 and then Bug 1193461 and  we likely have to disable img test. The reason is python 2.7.11 is not deployed on all try test machines.

I will keep my eye on the bugs, update here once test machines are upgraded.

Just want to confirm, I tried to run all referrer policy tests locally (with python 2.7.11, certainly), and all 1302 test cases passed.  

> I think we should keep this open to track progress on those things but I
> wouldn't consider it blocking to enabling the referrer policy attribute (we
> have basic coverage and know why some are disabled).

Totally agree, thanks

Comment 12

[Thomas Nguyen (:tnguyen)]

Close it and track spec in bug 1409600