1711938 - Intermittent [taskcluster:error] Task timeout after 2700 seconds. Force killing container. [stuck cloning]

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=340184589&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Bi6rlCVsTcy5Pg4pm2rzVw/runs/0/artifacts/public/logs/live_backing.log

---

run-task started in /builds/worker
[cache 2021-05-19T15:41:12.687Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2021-05-19T15:41:12.688Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2021-05-19T15:41:12.688Z] volume /builds/worker/checkouts is a cache
[setup 2021-05-19T15:41:12.688Z] running as worker:worker
[vcs 2021-05-19T15:41:12.688Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2021-05-19T15:41:12.769Z] hgmointernal rate miss; using public hg.mozilla.org service
[vcs 2021-05-19T15:41:12.770Z] fetching hg.mozilla.org fingerprint from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgfingerprint
[vcs 2021-05-19T15:41:12.803Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--config', 'hostsecurity.hg.mozilla.org:fingerprints=sha256:FF:E7:8D:93:E9:56:3C:C0:19:FC:00:4C:18:B9:86:E5:08:E5:10:F5:E2:EA:48:E8:22:D3:A3:3A:CA:99:C3:4C,sha256:17:38:aa:92:0b:84:3e:aa:8e:52:52:e9:4c:2f:98:a9:0e:bf:6c:3e:e9:15:ff:0a:29:80:f7:06:02:5b:e8:48', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '3a4df3aa8eb27c433aec64e592289147d1e91ecf', 'https://hg.mozilla.org/integration/autoland', '/builds/worker/checkouts/gecko']
[vcs 2021-05-19T15:41:12.879Z] (using Mercurial 4.8.1)
[vcs 2021-05-19T15:41:12.879Z] ensuring https://hg.mozilla.org/integration/autoland@3a4df3aa8eb27c433aec64e592289147d1e91ecf is available at /builds/worker/checkouts/gecko
[vcs 2021-05-19T15:41:13.245Z] (shared store has an active lock; assuming it is left over from a previous process and that the store is corrupt; deleting store and destination just to be sure)
[vcs 2021-05-19T15:41:13.248Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[taskcluster:error] Task timeout after 2700 seconds. Force killing container.
[taskcluster 2021-05-19 16:26:12.502Z] === Task Finished ===
[taskcluster 2021-05-19 16:26:12.502Z] Unsuccessful task run with exit code: -1 completed in 2700.88 seconds```

Comment 1

[Intermittent Failures Robot]

1 failures in 3061 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• lint: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1711938&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 2

[Dorel Luca [:dluca]]

Hi Chris,

Can you please take a look at this bug? I think your push of Bug 1712672 made this bug be almost perma.

https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=linting%2Copt%2Csource-test-wpt-metadata-fission-regression%2Cwpt-fis&revision=9425fdc1cf74f432e6fc8aca1209315b30d57da9

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Is the conditional attribution of wpt meta data with bugs supported like in https://hg.mozilla.org/integration/autoland/rev/9425fdc1cf74f432e6fc8aca1209315b30d57da9 ? The issue is the TypeError: Object of type 'set' is not JSON serializable.

Comment 4

[Chris Peterson [:cpeterson]]

(In reply to Dorel Luca [:dluca] from comment #2)

Can you please take a look at this bug? I think your push of Bug 1712672 made this bug be almost perma.

https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=linting%2Copt%2Csource-test-wpt-metadata-fission-regression%2Cwpt-fis&revision=9425fdc1cf74f432e6fc8aca1209315b30d57da9

James, is there a way to see the particular WPT .ini file is causing the error?

In the log, I see a reference to webvtt/api/VTTRegion/regionAnchorX.html immediately before the fissionregressions.py exception. My change didn't touch webvtt/api/VTTRegion/regionAnchorX.html.ini.

Before the reference to regionAnchorX.html, I see a reference to webrtc/RTCPeerConnection-videoDetectorTest.html, which my change did touch:

https://hg.mozilla.org/integration/autoland/diff/9425fdc1cf74f432e6fc8aca1209315b30d57da9/testing/web-platform/meta/webrtc/RTCPeerConnection-videoDetectorTest.html.ini

Comment 5

[Intermittent Failures Robot]

33 failures in 2908 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #33 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 19
• mozilla-central: 13
• try: 1

Platform and build breakdown:

• lint: 30
  • opt: 30
• android-4-0-armv7-api16: 3
  • pgo: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1711938&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 6

[Chris Peterson [:cpeterson]]

I reproduced this error locally with ./mach wpt-fission-regressions --all-json test-regressions.json --untriaged test-untriaged.txt --platform win-opt but don't know how to debug the Python error yet:

./mach wpt-fission-regressions --all-json test-regressions.json --untriaged test-untriaged.txt --platform win-opt
 0:00.19 INFO Loading test manifest
 0:00.51 INFO Skipping manifest download because existing file is recent
 0:14.42 INFO Loading tests win-opt baseline
 1:26.66 INFO Loading tests win-opt fission
Traceback (most recent call last):
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/mach_commands.py", line 383, in run
    return fissionregressions.run(logger, src_root, obj_root, **kwargs)
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/fissionregressions.py", line 480, in run
    write_all(test_results, kwargs["all_json"])
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/fissionregressions.py", line 494, in write_all
    json.dump(json_data, f, indent=2)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 1 more time]
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type set is not JSON serializable
> /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py(179)default()
-> raise TypeError(f'Object of type {o.__class__.__name__} '

Comment 7

[James Graham [:jgraham]]

Attachment: Bug 1711938 - Fix fission wpt data serialization error,

Comment 8

[Intermittent Failures Robot]

34 failures in 3065 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #42 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 3
• autoland: 20
• try: 3
• mozilla-central: 8

Platform and build breakdown:

• lint: 33
  • opt: 33
• android-em-7-0-x86_64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1711938&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/acfd10981480
Fix fission wpt data serialization error, r=cpeterson

Comment 10

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/acfd10981480

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/01c6fcb5dcc0

Comment 12

[Intermittent Failures Robot]

4 failures in 3502 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 1

Platform and build breakdown:

• lint: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1711938&startday=2021-06-07&endday=2021-06-13&tree=all