1959683 - periodic updates task hanging due to new search-config-icon files

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=503257567&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W4ih-gYkQae4roSg6Gv-ZA/runs/0/artifacts/public/logs/live_backing.log

---

+ cp -f /builds/worker/firefox/nsSTSPreloadList.inc mozilla-beta/security/manager/ssl/
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS'
+ '[' true == true ']'
+ stage_hpkp_files
+ cd /builds/worker
+ cp -f /builds/worker/data/StaticHPKPins.h.out mozilla-beta/security/manager/ssl/StaticHPKPins.h
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP'
+ '[' true == true ']'
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings'
+ '[' true == true ']'
+ stage_tld_suffix_files
+ cd /builds/worker
+ cp -a public_suffix_list.dat mozilla-beta//netwerk/dns/effective_tld_names.dat
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes'
+ '[' true == true ']'
+ stage_mobile_experiments_files
+ cd /builds/worker
+ cp fenix-experiments-new.json mozilla-beta/mobile/android/fenix/app/src/main/res/raw/initial_experiments.json
+ cp focus-experiments-new.json mozilla-beta/mobile/android/focus-android/app/src/main/res/raw/initial_experiments.json
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments'
+ '[' true == true ']'
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments ct-logs'
+ '[' false == true ']'
+ '[' false == true ']'
+ '[' true == true ']'
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments ct-logs - a=repo-update'
+ '[' false == true ']'
+ /usr/bin/hg -R mozilla-beta commit -u 'ffxbld <[e-mail]>' -m 'No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments ct-logs - a=repo-update'
+ push_repo
+ '[' false == true ']'
+ cd mozilla-beta
+ '[' '!' -r /builds/worker/.arcrc ']'
++ command -v arc
+ ARC=
++ command -v arcanist
+ ARC=/usr/bin/arcanist
+ '[' -z '#release-managers' ']'
++ /usr/bin/arcanist list
++ grep 'Needs Review'
++ grep -E 'mozilla-beta repo-update'
++ awk 'match($0, /D[0-9]+[^: ]/) { print substr($0, RSTART, RLENGTH)  }'
+ /usr/bin/arcanist diff --verbatim --reviewers '#release-managers'
You have untracked files in this working copy.
  Working copy: /builds/worker/mozilla-beta/
  Untracked changes in working copy:
  (To ignore these 54 change(s), add them to ".hgignore".)
    services/settings/dumps/main/search-config-icons/0a57b0cf-34f0-4d09-96e4-dbd6e3355410
    services/settings/dumps/main/search-config-icons/0a57b0cf-34f0-4d09-96e4-dbd6e3355410.meta.json
    services/settings/dumps/main/search-config-icons/2ecca3f8-c1ef-43cc-b053-886d1ae46c36
    services/settings/dumps/main/search-config-icons/2ecca3f8-c1ef-43cc-b053-886d1ae46c36.meta.json
    services/settings/dumps/main/search-config-icons/39d0b17d-c020-4890-932f-83c0f6ed130b
    services/settings/dumps/main/search-config-icons/39d0b17d-c020-4890-932f-83c0f6ed130b.meta.json
    services/settings/dumps/main/search-config-icons/41135a88-093d-4077-873b-9de1ae133427
    services/settings/dumps/main/search-config-icons/41135a88-093d-4077-873b-9de1ae133427.meta.json
    services/settings/dumps/main/search-config-icons/41f0d805-3775-4988-8d8c-5ad8ccd86d1c
    services/settings/dumps/main/search-config-icons/41f0d805-3775-4988-8d8c-5ad8ccd86d1c.meta.json
    services/settings/dumps/main/search-config-icons/48c72361-cd67-412e-bd7f-f81a43c10791
    services/settings/dumps/main/search-config-icons/48c72361-cd67-412e-bd7f-f81a43c10791.meta.json
    services/settings/dumps/main/search-config-icons/5203dd03-2c55-4b53-9c60-58258d587be1
    services/settings/dumps/main/search-config-icons/5203dd03-2c55-4b53-9c60-58258d587be1.meta.json
    services/settings/dumps/main/search-config-icons/5914932e-66ba-4126-8be5-d37beadd9532
    services/settings/dumps/main/search-config-icons/5914932e-66ba-4126-8be5-d37beadd9532.meta.json
    services/settings/dumps/main/search-config-icons/6644f26f-28ea-4222-929d-5d43a02dae05
    services/settings/dumps/main/search-config-icons/6644f26f-28ea-4222-929d-5d43a02dae05.meta.json
    services/settings/dumps/main/search-config-icons/6e36a151-e4f4-4117-9067-1ca82c47d01a
    services/settings/dumps/main/search-config-icons/6e36a151-e4f4-4117-9067-1ca82c47d01a.meta.json
    services/settings/dumps/main/search-config-icons/7072564d-a573-4750-bf33-f0a07631c9eb
    services/settings/dumps/main/search-config-icons/7072564d-a573-4750-bf33-f0a07631c9eb.meta.json
    services/settings/dumps/main/search-config-icons/71f41a0c-5b70-4116-b30f-e62089083522
    services/settings/dumps/main/search-config-icons/71f41a0c-5b70-4116-b30f-e62089083522.meta.json
    services/settings/dumps/main/search-config-icons/74f94dc2-caf6-4b90-b3d2-f3e2f7714d88
    services/settings/dumps/main/search-config-icons/74f94dc2-caf6-4b90-b3d2-f3e2f7714d88.meta.json
    services/settings/dumps/main/search-config-icons/764e3b14-fe16-4feb-8384-124c516a5afa
    services/settings/dumps/main/search-config-icons/764e3b14-fe16-4feb-8384-124c516a5afa.meta.json
    services/settings/dumps/main/search-config-icons/7c81cf98-7c11-4afd-8279-db89118a6dfb
    services/settings/dumps/main/search-config-icons/7c81cf98-7c11-4afd-8279-db89118a6dfb.meta.json
    services/settings/dumps/main/search-config-icons/7cb4d88a-d4df-45b2-87e4-f896eaf1bbdb
    services/settings/dumps/main/search-config-icons/7cb4d88a-d4df-45b2-87e4-f896eaf1bbdb.meta.json
    services/settings/dumps/main/search-config-icons/7edaf4fe-a8a0-432b-86d2-bf75ebe80851
    services/settings/dumps/main/search-config-icons/7edaf4fe-a8a0-432b-86d2-bf75ebe80851.meta.json
    services/settings/dumps/main/search-config-icons/8abb10a7-212f-46b5-a7b4-244f414e3810
    services/settings/dumps/main/search-config-icons/8abb10a7-212f-46b5-a7b4-244f414e3810.meta.json
    services/settings/dumps/main/search-config-icons/94a84724-c30f-4767-ba42-01cc37fc31a4
    services/settings/dumps/main/search-config-icons/94a84724-c30f-4767-ba42-01cc37fc31a4.meta.json
    services/settings/dumps/main/search-config-icons/95ed201d-4ab8-4cb8-831d-454f53cab0f8
    services/settings/dumps/main/search-config-icons/95ed201d-4ab8-4cb8-831d-454f53cab0f8.meta.json
    services/settings/dumps/main/search-config-icons/9802e63d-05ec-48ba-93f9-746e0981ad98
    services/settings/dumps/main/search-config-icons/9802e63d-05ec-48ba-93f9-746e0981ad98.meta.json
    services/settings/dumps/main/search-config-icons/a83f24e4-602c-47bd-930c-ad0947ee1adf
    services/settings/dumps/main/search-config-icons/a83f24e4-602c-47bd-930c-ad0947ee1adf.meta.json
    services/settings/dumps/main/search-config-icons/b882b24d-1776-4ef9-9016-0bdbd935eda3
    services/settings/dumps/main/search-config-icons/b882b24d-1776-4ef9-9016-0bdbd935eda3.meta.json
    services/settings/dumps/main/search-config-icons/b9424309-f601-4a69-98ca-ca68e65633e6
    services/settings/dumps/main/search-config-icons/b9424309-f601-4a69-98ca-ca68e65633e6.meta.json
    services/settings/dumps/main/search-config-icons/db0e1627-ae89-4c25-8944-a9481d8512d9
    services/settings/dumps/main/search-config-icons/db0e1627-ae89-4c25-8944-a9481d8512d9.meta.json
    services/settings/dumps/main/search-config-icons/eb62e768-151b-45d1-9fe5-9e1d2a5991c5
    services/settings/dumps/main/search-config-icons/eb62e768-151b-45d1-9fe5-9e1d2a5991c5.meta.json
    services/settings/dumps/main/search-config-icons/fe75ce3f-1545-400c-b28c-ad771054e69f
    services/settings/dumps/main/search-config-icons/fe75ce3f-1545-400c-b28c-ad771054e69f.meta.json
    Ignore these 54 untracked file(s) and continue? [y/N] 
[taskcluster:error] Task timeout after 14400 seconds. Force killing container.
[taskcluster 2025-04-10 12:08:07.308Z] === Task Finished ===
[taskcluster 2025-04-10 12:08:07.308Z] Unsuccessful task run with exit code: -1 completed in 14418.385 seconds

Comment 1

[[e-mail] (:bhearsum)]

Mark, it looks like the untracked files that are causing the job to hang are from search-config-icons, which you might know something about according to file history. Do you know whether or not we should be ignoring these or adding them?

Comment 2

[Mark Banner (:standard8)]

(In reply to [e-mail] (:bhearsum) from comment #1)

Mark, it looks like the untracked files that are causing the job to hang are from search-config-icons, which you might know something about according to file history. Do you know whether or not we should be ignoring these or adding them?

I didn't actually realise that the task would fail, I just assumed it would ignore them.

For search-config-icons, we should ignore extras. We do have bug 1959163 ready to land, but we'll still need to ignore some. We could probably filter them out in the periodic-file-scripts, but that requires matching JEXL filter expressions.

Can we get it to not prompt for now, and we can think about how we tidy this up for future in a follow-up?

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Bug 1959683: purge search-config-icons after running remote settings update r?standard8!,jcristau!

This will avoid arc diff hanging if untracked files are around. I've taken this approach rather than using arc diff --allow-untracked to avoid accidentally missing newly added files from other periodic updates.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1959683: use /dev/null as stdin for arc diff r?jcristau!

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/efe56d7cba09
use /dev/null as stdin for arc diff r=jcristau

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/453d2ffee9df
purge search-config-icons after running remote settings update r=Standard8,jcristau

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 9478286 [details]
Bug 1959683: purge search-config-icons after running remote settings update r?standard8!,jcristau!

Beta/Release Uplift Approval Request

• User impact if declined/Reason for urgency: periodically updated files (such as certificate pinning lists) may not be updated
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The alternative is to risk not updating important files.
• String changes made/needed:
• Is Android affected?: No

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: it ensures some necessary automation will continue to work
• User impact if declined: periodically updated files (such as certificate pinning lists) may not be updated
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The alternative is to risk not updating important files.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: Bug 1959683: purge search-config-icons after running remote settings update

Comment 9

[Phabricator Automation]

esr115 Uplift Approval Request

• User impact if declined: periodically updated files (such as certificate pinning lists) may not be updated
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: The alternative is to risk not updating important files.
• String changes made/needed: n/a
• Is Android affected?: no

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/c2035c54786e

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/ab567cf1e801
use /dev/null as stdin for arc diff r=jcristau
https://hg.mozilla.org/mozilla-central/rev/d309138776e6
purge search-config-icons after running remote settings update r=Standard8,jcristau

Comment 12

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/ab567cf1e801
https://hg.mozilla.org/mozilla-central/rev/d309138776e6

Comment 13

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/a3b9ea3ec058

Comment 14

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/3659e2b568b5

Comment 15

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/efe56d7cba09
https://hg.mozilla.org/mozilla-central/rev/453d2ffee9df

Comment 16

[Intermittent Failures Robot]

6 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 1
• mozilla-release: 2
• mozilla-esr128: 1

Table

	no_variant
linux64--/opt	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1959683&startday=2025-04-07&endday=2025-04-13&tree=all