1924929 - Multiple Perma Wd failures on mozilla-beta

Status: RESOLVED FIXED

(Remote Protocol :: WebDriver BiDi, defect, P2)

Description

[Norisz Fay [:noriszfay]]

Beta push with failures

Parsed log: https://treeherder.mozilla.org/logviewer?job_id=478465474&repo=mozilla-beta&lineNumber=32601
Full log: https://firefoxci.taskcluster-artifacts.net/JJdjDyjYQr6wMFxhvSPs1A/0/public/logs/live_backing.log

[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - TEST-PASS | /webdriver/tests/bidi/storage/get_cookies/partition.py | test_default_partition 
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - TEST-PASS | /webdriver/tests/bidi/storage/get_cookies/partition.py | test_partition_context_with_different_domain 
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - TEST-PASS | /webdriver/tests/bidi/storage/get_cookies/partition.py | test_partition_context_iframe[same_origin] 
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/storage/get_cookies/partition.py | test_partition_context_iframe[cross_origin] - AssertionError
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x7fc9308e1ee0>
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - new_tab = {'context': '0e0c061a-18e0-4086-9635-71b6f7905b00'}
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - inline = <function inline.<locals>.inline at 0x7fc93084a0d0>
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - domain_value = <function domain_value.<locals>.domain_value at 0x7fc93084a160>
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - domain = 'alt'
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - set_cookie = <function set_cookie.<locals>.set_cookie at 0x7fc93084cd30>
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -     @pytest.mark.parametrize("domain", ["", "alt"], ids=["same_origin", "cross_origin"])
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -     async def test_partition_context_iframe(
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         bidi_session, new_tab, inline, domain_value, domain, set_cookie
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -     ):
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         iframe_url = inline("<div id='in-iframe'>foo</div>", domain=domain)
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         page_url = inline(f"<iframe src='{iframe_url}'></iframe>")
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         await bidi_session.browsing_context.navigate(
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -             context=new_tab["context"], url=page_url, wait="complete"
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         )
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -     
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         contexts = await bidi_session.browsing_context.get_tree(root=new_tab["context"])
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         iframe_context = contexts[0]["children"][0]
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -     
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         cookie_name = "foo"
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         cookie_value = "bar"
[task 2024-10-16T02:55:20.729Z] 02:55:20     INFO -         await set_cookie(
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             cookie=create_cookie(
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 domain=domain_value(domain),
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 name=cookie_name,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 value=NetworkStringValue(cookie_value),
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             ),
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             partition=BrowsingContextPartitionDescriptor(iframe_context["context"]),
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -         )
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -     
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -         # Check that added cookies are present on the right context
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -         cookies = await bidi_session.storage.get_cookies(
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             partition=BrowsingContextPartitionDescriptor(iframe_context["context"])
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -         )
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -     
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO - >       recursive_compare(
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             {
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 "cookies": [
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                     {
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "domain": domain_value(domain=domain),
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "httpOnly": False,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "name": cookie_name,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "path": "/",
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "sameSite": "none",
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "secure": True,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "size": 6,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                         "value": {"type": "string", "value": cookie_value},
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                     }
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 ],
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 "partitionKey": {"userContext": "default"},
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             },
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -             cookies,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -         )
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x7fc9308e1ee0>
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO - contexts   = [{'children': [{'children': [],
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 'context': '19327352833',
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 'originalOpener': None,
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 'url': 'https://not-web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv+id%3D%27in-iframe%27%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8',
[task 2024-10-16T02:55:20.730Z] 02:55:20     INFO -                 'userContext': 'default'}],
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'context': '0e0c061a-18e0-4086-9635-71b6f7905b00',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'originalOpener': None,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'parent': None,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'url': 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Ciframe+src%3D%27https%3A%2F%2Fnot-web-platform.test%3A8443%2Fwebdriver%2Ftests%2Fsupport%2Finline.py%3Fdoc%3D%253C%2521doctype%2Bhtml%253E%250A%253Cmeta%2Bcharset%253DUTF-8%253E%250A%253Cdiv%2Bid%253D%2527in-iframe%2527%253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'userContext': 'default'}]
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - cookie_name = 'foo'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - cookie_value = 'bar'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - cookies    = {'cookies': [],
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'partitionKey': {'sourceOrigin': 'https://not-web-platform.test:8443',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -                   'userContext': 'default'}}
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - domain     = 'alt'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - domain_value = <function domain_value.<locals>.domain_value at 0x7fc93084a160>
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - iframe_context = {'children': [],
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'context': '19327352833',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'originalOpener': None,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'url': 'https://not-web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv+id%3D%27in-iframe%27%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'userContext': 'default'}
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - iframe_url = 'https://not-web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Cdiv+id%3D%27in-iframe%27%3Efoo%3C%2Fdiv%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - inline     = <function inline.<locals>.inline at 0x7fc93084a0d0>
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - new_tab    = {'context': '0e0c061a-18e0-4086-9635-71b6f7905b00'}
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - page_url   = 'https://web-platform.test:8443/webdriver/tests/support/inline.py?doc=%3C%21doctype+html%3E%0A%3Cmeta+charset%3DUTF-8%3E%0A%3Ciframe+src%3D%27https%3A%2F%2Fnot-web-platform.test%3A8443%2Fwebdriver%2Ftests%2Fsupport%2Finline.py%3Fdoc%3D%253C%2521doctype%2Bhtml%253E%250A%253Cmeta%2Bcharset%253DUTF-8%253E%250A%253Cdiv%2Bid%253D%2527in-iframe%2527%253Efoo%253C%252Fdiv%253E%26mime%3Dtext%252Fhtml%26charset%3DUTF-8%27%3E%3C%2Fiframe%3E&mime=text%2Fhtml&charset=UTF-8'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - set_cookie = <function set_cookie.<locals>.set_cookie at 0x7fc93084cd30>
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - tests/web-platform/tests/webdriver/tests/bidi/storage/get_cookies/partition.py:205: 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - tests/web-platform/tests/webdriver/tests/bidi/__init__.py:25: in recursive_compare
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -     recursive_compare(expected[key], actual[key])
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -         actual     = {'cookies': [],
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'partitionKey': {'sourceOrigin': 'https://not-web-platform.test:8443',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -                   'userContext': 'default'}}
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -         expected   = {'cookies': [{'domain': 'not-web-platform.test',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'httpOnly': False,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'name': 'foo',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'path': '/',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'sameSite': 'none',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'secure': True,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'size': 6,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -               'value': {'type': 'string', 'value': 'bar'}}],
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -  'partitionKey': {'userContext': 'default'}}
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -         key        = 'cookies'
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - expected = [{'domain': 'not-web-platform.test', 'httpOnly': False, 'name': 'foo', 'path': '/', ...}]
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - actual = []
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -     def recursive_compare(expected: Any, actual: Any) -> None:
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -         if callable(expected):
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -             expected(actual)
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -             return
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -     
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -         if isinstance(actual, List) and isinstance(expected, List):
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - >           assert len(expected) == len(actual)
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - E           AssertionError
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - actual     = []
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - expected   = [{'domain': 'not-web-platform.test',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'httpOnly': False,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'name': 'foo',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'path': '/',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'sameSite': 'none',
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'secure': True,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'size': 6,
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO -   'value': {'type': 'string', 'value': 'bar'}}]
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - 
[task 2024-10-16T02:55:20.732Z] 02:55:20     INFO - tests/web-platform/tests/webdriver/tests/bidi/__init__.py:14: AssertionError
[task 2024-10-16T02:55:20.733Z] 02:55:20     INFO - ..
[task 2024-10-16T02:55:20.733Z] 02:55:20     INFO - TEST-OK | /webdriver/tests/bidi/storage/get_cookies/partition.py | took 9387ms

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume that we have to revert the changes from bug 1899978 given that CHIPS is no longer enabled on beta? Probably we should wait with using this preference until the CHIPS feature is actually released.

Comment 2

[Alexandra Borovova [:Sasha]]

Attachment: Bug 1924929 - Disable tests dependent on "network.cookie.CHIPS.enabled" pref on beta and stable.

Comment 3

[Alexandra Borovova [:Sasha]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #1)

I assume that we have to revert the changes from bug 1899978 given that CHIPS is no longer enabled on beta? Probably we should wait with using this preference until the CHIPS feature is actually released.

As I mentioned on Matrix, I don't think that we should revert the update of the pref, since network.cookie.CHIPS.enabled is the right pref for the changes we require, and we still have a fallback in place when the pref is disabled. The only problem is that we have the tests which rely on this pref to be enabled (and revert wouldn't solve it, we would just rely on a different pref which is also enabled only in nightly). In my opinion, ideally, we could enforce the pref to be enabled for these tests, and we could write alternative mozilla-specific tests with this pref being disabled to cover both scenarios. Unfortunately, I couldn't make it work so far.
The other option is to just mark the tests as expected to fail in beta and stable. I've created a patch for this for now.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1923663

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/45ed66ca8512
Disable tests dependent on "network.cookie.CHIPS.enabled" pref on beta and stable. r=webdriver-reviewers,whimboo

Comment 6

[Alexandra Borovova [:Sasha]]

Attachment: Bug 1924929 - Disable tests dependent on "network.cookie.CHIPS.enabled" pref on beta and stable.

Original Revision: https://phabricator.services.mozilla.com/D225765

Comment 7

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: The tests will keep on failing on beta
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: None
• Risk associated with taking this patch: Low
• Explanation of risk level: The patch only disables the tests for beta and stable
• String changes made/needed: None
• Is Android affected?: yes

Comment 8

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/e4eed90f403f

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/45ed66ca8512

Comment 10

[Intermittent Failures Robot]

78 failures in 880 pushes (0.089 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 78

Platform and build breakdown:

• windows11-64-2009-qr: 8
  • debug: 8
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-qr: 8
  • opt: 8
• android-em-7-0-x86_64-qr: 8
  • debug-isolated-process: 8
• macosx1015-64-qr: 4
  • debug: 4
• linux1804-64-asan-qr: 4
  • opt: 4
• linux1804-64-qr: 4
  • debug: 4
• windows11-32-2009-qr: 4
  • debug: 4
• linux1804-64-tsan-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 4
  • opt: 4
• windows11-64-2009-devedition-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• linux1804-64-devedition-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-lite-qr: 8
  • opt: 8
• macosx1015-64-shippable-qr: 4
  • opt: 4
• macosx1015-64-devedition-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1924929&startday=2024-10-16&endday=2024-10-16&tree=all

Comment 11

[Intermittent Failures Robot]

78 failures in 4579 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 78

Platform and build breakdown:

• windows11-64-2009-qr: 8
  • debug: 8
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-qr: 8
  • opt: 8
• android-em-7-0-x86_64-qr: 8
  • debug-isolated-process: 8
• macosx1015-64-qr: 4
  • debug: 4
• linux1804-64-asan-qr: 4
  • opt: 4
• linux1804-64-qr: 4
  • debug: 4
• windows11-32-2009-qr: 4
  • debug: 4
• linux1804-64-tsan-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 4
  • opt: 4
• windows11-64-2009-devedition-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• linux1804-64-devedition-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-lite-qr: 8
  • opt: 8
• macosx1015-64-shippable-qr: 4
  • opt: 4
• macosx1015-64-devedition-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1924929&startday=2024-10-14&endday=2024-10-20&tree=all