Closed Bug 1711938 Opened 3 years ago Closed 3 years ago

Intermittent [taskcluster:error] Task timeout after 2700 seconds. Force killing container. [stuck cloning]

Categories

(Firefox Build System :: General, defect, P5)

defect

Tracking

(firefox-esr78 unaffected, firefox89 unaffected, firefox90 fixed, firefox91 fixed)

RESOLVED FIXED
91 Branch
Tracking Status
firefox-esr78 --- unaffected
firefox89 --- unaffected
firefox90 --- fixed
firefox91 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jgraham)

References

(Regression)

Details

(Keywords: intermittent-failure, regression)

Attachments

(1 file)

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=340184589&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Bi6rlCVsTcy5Pg4pm2rzVw/runs/0/artifacts/public/logs/live_backing.log


run-task started in /builds/worker
[cache 2021-05-19T15:41:12.687Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2021-05-19T15:41:12.688Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2021-05-19T15:41:12.688Z] volume /builds/worker/checkouts is a cache
[setup 2021-05-19T15:41:12.688Z] running as worker:worker
[vcs 2021-05-19T15:41:12.688Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2021-05-19T15:41:12.769Z] hgmointernal rate miss; using public hg.mozilla.org service
[vcs 2021-05-19T15:41:12.770Z] fetching hg.mozilla.org fingerprint from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgfingerprint
[vcs 2021-05-19T15:41:12.803Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--config', 'hostsecurity.hg.mozilla.org:fingerprints=sha256:FF:E7:8D:93:E9:56:3C:C0:19:FC:00:4C:18:B9:86:E5:08:E5:10:F5:E2:EA:48:E8:22:D3:A3:3A:CA:99:C3:4C,sha256:17:38:aa:92:0b:84:3e:aa:8e:52:52:e9:4c:2f:98:a9:0e:bf:6c:3e:e9:15:ff:0a:29:80:f7:06:02:5b:e8:48', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '3a4df3aa8eb27c433aec64e592289147d1e91ecf', 'https://hg.mozilla.org/integration/autoland', '/builds/worker/checkouts/gecko']
[vcs 2021-05-19T15:41:12.879Z] (using Mercurial 4.8.1)
[vcs 2021-05-19T15:41:12.879Z] ensuring https://hg.mozilla.org/integration/autoland@3a4df3aa8eb27c433aec64e592289147d1e91ecf is available at /builds/worker/checkouts/gecko
[vcs 2021-05-19T15:41:13.245Z] (shared store has an active lock; assuming it is left over from a previous process and that the store is corrupt; deleting store and destination just to be sure)
[vcs 2021-05-19T15:41:13.248Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[taskcluster:error] Task timeout after 2700 seconds. Force killing container.
[taskcluster 2021-05-19 16:26:12.502Z] === Task Finished ===
[taskcluster 2021-05-19 16:26:12.502Z] Unsuccessful task run with exit code: -1 completed in 2700.88 seconds```
Summary: Intermittent [taskcluster:error] Task timeout after 2700 seconds. Force killing container. → Intermittent [taskcluster:error] Task timeout after 2700 seconds. Force killing container. [stuck cloning]
Flags: needinfo?(cpeterson)
Regressed by: 1712672
Has Regression Range: --- → yes

Is the conditional attribution of wpt meta data with bugs supported like in https://hg.mozilla.org/integration/autoland/rev/9425fdc1cf74f432e6fc8aca1209315b30d57da9 ? The issue is the TypeError: Object of type 'set' is not JSON serializable.

Flags: needinfo?(james)

(In reply to Dorel Luca [:dluca] from comment #2)

Can you please take a look at this bug? I think your push of Bug 1712672 made this bug be almost perma.

https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=linting%2Copt%2Csource-test-wpt-metadata-fission-regression%2Cwpt-fis&revision=9425fdc1cf74f432e6fc8aca1209315b30d57da9

James, is there a way to see the particular WPT .ini file is causing the error?

In the log, I see a reference to webvtt/api/VTTRegion/regionAnchorX.html immediately before the fissionregressions.py exception. My change didn't touch webvtt/api/VTTRegion/regionAnchorX.html.ini.

Before the reference to regionAnchorX.html, I see a reference to webrtc/RTCPeerConnection-videoDetectorTest.html, which my change did touch:

https://hg.mozilla.org/integration/autoland/diff/9425fdc1cf74f432e6fc8aca1209315b30d57da9/testing/web-platform/meta/webrtc/RTCPeerConnection-videoDetectorTest.html.ini

I reproduced this error locally with ./mach wpt-fission-regressions --all-json test-regressions.json --untriaged test-untriaged.txt --platform win-opt but don't know how to debug the Python error yet:

./mach wpt-fission-regressions --all-json test-regressions.json --untriaged test-untriaged.txt --platform win-opt
 0:00.19 INFO Loading test manifest
 0:00.51 INFO Skipping manifest download because existing file is recent
 0:14.42 INFO Loading tests win-opt baseline
 1:26.66 INFO Loading tests win-opt fission
Traceback (most recent call last):
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/mach_commands.py", line 383, in run
    return fissionregressions.run(logger, src_root, obj_root, **kwargs)
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/fissionregressions.py", line 480, in run
    write_all(test_results, kwargs["all_json"])
  File "/Users/chris/Code/mozilla/firefox/testing/web-platform/fissionregressions.py", line 494, in write_all
    json.dump(json_data, f, indent=2)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 1 more time]
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type set is not JSON serializable
> /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py(179)default()
-> raise TypeError(f'Object of type {o.__class__.__name__} '
Flags: needinfo?(cpeterson)
Assignee: nobody → james
Status: NEW → ASSIGNED
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/acfd10981480
Fix fission wpt data serialization error, r=cpeterson
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch
Flags: needinfo?(james)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: