Closed Bug 1762128 Opened 2 years ago Closed 2 years ago

Intermittent bouncer-check abort: repository requires features unknown to this Mercurial: revlog-compression-zstd!

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
defect

Tracking

(firefox-esr9199+ fixed, firefox98 wontfix, firefox99 unaffected, firefox100blocking fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr91 99+ fixed
firefox98 --- wontfix
firefox99 --- unaffected
firefox100 blocking fixed

People

(Reporter: intermittent-bug-filer, Assigned: jcristau)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Bug 1762128 - bump the cache name for decision images to stop interfering with other trees. r?#releng
48 bytes, text/x-phabricator-request
Details | Review

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=372784930&repo=mozilla-esr91
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QGV_invzTaOeSFH0jOznBQ/runs/0/artifacts/public/logs/live_backing.log

[taskcluster 2022-03-30 07:00:57.131Z] Task ID: QGV_invzTaOeSFH0jOznBQ
[taskcluster 2022-03-30 07:00:57.131Z] Worker ID: i-04e63840f5e404157
[taskcluster 2022-03-30 07:00:57.131Z] Worker Group: us-west-1
[taskcluster 2022-03-30 07:00:57.131Z] Worker Node Type: m5d.xlarge
[taskcluster 2022-03-30 07:00:57.131Z] Worker Type: decision
[taskcluster 2022-03-30 07:00:57.131Z] Public IP: 54.176.13.217
[taskcluster 2022-03-30 07:00:57.131Z] Hostname: ip-10-143-40-204
[taskcluster 2022-03-30 07:00:57.131Z] using cache "gecko-level-3-checkouts-sparse-v2" -> /builds/worker/checkouts
[taskcluster 2022-03-30 07:00:57.653Z] === Task Starting ===
[setup 2022-03-30T07:00:58.023Z] run-task started in /
[cache 2022-03-30T07:00:58.024Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[setup 2022-03-30T07:00:58.024Z] running as worker:worker
[vcs 2022-03-30T07:00:58.024Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2022-03-30T07:00:58.129Z] hgmointernal rate miss; using public hg.mozilla.org service
[vcs 2022-03-30T07:00:58.129Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--sparseprofile', 'build/sparse-profiles/taskgraph', '--revision', 'd8503523b4cc610ec3bade0787f200afdef738e6', 'https://hg.mozilla.org/releases/mozilla-esr91', '/builds/worker/checkouts/gecko']
[vcs 2022-03-30T07:00:58.182Z] (using Mercurial 4.8.1)
[vcs 2022-03-30T07:00:58.182Z] ensuring https://hg.mozilla.org/releases/mozilla-esr91@d8503523b4cc610ec3bade0787f200afdef738e6 is available at /builds/worker/checkouts/gecko
[vcs 2022-03-30T07:00:58.183Z] (existing repository shared store: /builds/worker/checkouts/hg-store/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29/.hg)
[vcs 2022-03-30T07:00:58.368Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["m5d.xlarge"], "hgVersion": "4.8.1", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.18381404876708984}, {"extraOptions": ["m5d.xlarge"], "hgVersion": "4.8.1", "lowerIsBetter": true, "name": "overall_nopull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.18381404876708984}, {"extraOptions": ["m5d.xlarge"], "hgVersion": "4.8.1", "lowerIsBetter": true, "name": "overall_nopull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.18381404876708984}, {"extraOptions": ["m5d.xlarge"], "hgVersion": "4.8.1", "lowerIsBetter": true, "name": "overall_nopull_populatedwdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.18381404876708984}]}
[vcs 2022-03-30T07:00:58.368Z] abort: repository requires features unknown to this Mercurial: revlog-compression-zstd!
[vcs 2022-03-30T07:00:58.368Z] (see https://mercurial-scm.org/wiki/MissingRequirement for more information)
[taskcluster 2022-03-30 07:00:58.623Z] === Task Finished ===
[taskcluster 2022-03-30 07:00:58.691Z] Artifact "public/docker-contexts" not found at "/builds/worker/checkouts/gecko/docker-contexts"
[taskcluster 2022-03-30 07:00:58.770Z] Unsuccessful task run with exit code: 255 completed in 1.639 seconds

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Task Configuration' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: General → Task Configuration
Component: Task Configuration → Release Automation: Other
Product: Firefox Build System → Release Engineering
QA Contact: gbrown
Summary: Intermittent abort: repository requires features unknown to this Mercurial: revlog-compression-zstd! → Intermittent bouncer-check abort: repository requires features unknown to this Mercurial: revlog-compression-zstd!

Could this be from switching the decision image (and hg version?) on central, causing the shared hg store to be incompatible with the decision image for esr91? Do we have a way of separating the caches?

Flags: needinfo?(ahal)

It's happening on mozilla-release too, on searchfox job

The searchfox failures are for non-mozilla-central repositories, e.g. release and esr91. Are those failures from https://hg.mozilla.org/mozilla-central/rev/a16419d72687 ?

Flags: needinfo?(gabriel)

Oh, yes that seems likely. I think it would be easier to try uplifting the image bump patch everywhere. Also these types of issues aside I think we'd want to keep sharing caches.

hg 4.8 (e.g. from the v3.0.1 decision image) seems to choke on cached repos
created with 5.8 (from v4.0.0).

Assignee: nobody → jcristau
Status: NEW → ASSIGNED
Severity: S4 → S1
tracking-firefox100: --- → ?
tracking-firefox99: --- → ?
Priority: P5 → P1

I think we should either land the patch from comment 6 on central or back out the regressing change. Either way we'll have to quarantine workers with the bad cache afterwards. And once we're out of RC week we can consider whether it's worth uplifting.

Pushed by jcristau@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/13fd764e0b74
bump the cache name for decision images to stop interfering with other trees. r=releng-reviewers,ahal a=release

https://hg.mozilla.org/mozilla-central/rev/13fd764e0b74

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox100: affected → fixed
Resolution: --- → FIXED
Flags: needinfo?(gabriel)
Flags: needinfo?(ahal)

Issue has been observed again for mozilla-esr91. Is there anything left to be done here?

Flags: needinfo?(gabriel)
Flags: needinfo?(gabriel)

I have seen this twice today on try server.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

This is happening because we updated the Decision image recently. This included a Mercurial version bump. This caused tasks to cache gecko clones cloned with the newer version of Mercurial in the V2 cache. I was looking at some of the revisions on try where we are running into the issue.

This revision, for instance, is using version 3.0.1 of the decision image and V2 of the cache

Pretty sure these were getting a cached checkout cloned with the newer Mercurial from the V2 cache. We bumped the cache, but revisions using the older image might still get an incompatible checkout. Pushes based on the latest revision (set to use v4 of the decision image) should work.

All that's left to do is terminate any workers with the broken cache.

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
Resolution: --- → FIXED

For posterity's sake, https://hg.mozilla.org/mozilla-central/rev/a16419d72687 was the regressing commit (since it landed "no bug" and we there can't effectively link to anything in bugzilla). I'm going to look at uplifting that to ESR now as it's still hitting randomly at least on Try pushes from what I can see.

See Also: → 1764372
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: