1905265 - Perma beta Linux x64 opt pfu [taskcluster:error] Task timeout after 14400 seconds. Force killing container.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=464350338&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/N_J9lfByQteSsvx71K8-7A/runs/7/artifacts/public/logs/live_backing.log

---

+ echo 'INFO: diffing in-tree suffix list against the suffix list from AMO...'
INFO: diffing in-tree suffix list against the suffix list from AMO...
+ /usr/bin/diff -u public_suffix_list.dat effective_tld_names.dat
+ tee /home/worker/artifacts/effective_tld_names.diff
--- public_suffix_list.dat	2024-06-28 02:54:21.056390653 +0000
+++ effective_tld_names.dat	2024-06-28 02:54:23.340561890 +0000
@@ -13824,7 +13824,7 @@
 // Submitted by Bohdan Dub <[e-mail]>
 ie.ua
 
-// HostyHosting (https://hostyhosting.com)
+// HostyHosting (hostyhosting.com)
 hostyhosting.io
 
 // Hypernode B.V. : https://www.hypernode.com/
@@ -14853,15 +14853,14 @@
 myradweb.net
 servername.us
 
-// Raidboxes GmbH : https://raidboxes.de
-// Submitted by Auke Tembrink <[e-mail]>
-myrdbx.io
-site.rb-hosting.io
-
 // Redgate Software: https://red-gate.com
 // Submitted by Andrew Farries <[e-mail]>
 instances.spawn.cc
 
+// Redstar Consultants : https://www.redstarconsultants.com/
+// Submitted by Jons Slemmer <[e-mail]>
+instantcloud.cn
+
 // Russian Academy of Sciences
 // Submitted by Tech Support <[e-mail]>
 ras.ru
@@ -15438,7 +15437,7 @@
 ts.net
 *.c.ts.net
 
-// TASK geographical domains (https://www.task.gda.pl/uslugi/dns)
+// TASK geographical domains (www.task.gda.pl/uslugi/dns)
 gda.pl
 gdansk.pl
 gdynia.pl
+ '[' -s /home/worker/artifacts/effective_tld_names.diff ']'
+ return 0
+ SUFFIX_LIST_UPDATED=true
+ '[' true == true ']'
+ compare_mobile_experiments
+ echo 'INFO wget -nv https://experimenter.services.mozilla.com/api/v6/experiments-first-run/'
INFO wget -nv https://experimenter.services.mozilla.com/api/v6/experiments-first-run/
+ wget -nv -O experiments.json https://experimenter.services.mozilla.com/api/v6/experiments-first-run/
2024-06-28 02:54:23 URL:https://experimenter.services.mozilla.com/api/v6/experiments-first-run/ [6079/6079] -> "experiments.json" [1]
+ wget -nv -O fenix-experiments-old.json https://hg.mozilla.org/releases/mozilla-beta/raw-file/default/mobile/android/fenix/app/src/main/res/raw/initial_experiments.json
2024-06-28 02:54:26 URL:https://hg.mozilla.org/releases/mozilla-beta/raw-file/default/mobile/android/fenix/app/src/main/res/raw/initial_experiments.json [4951/4951] -> "fenix-experiments-old.json" [1]
+ wget -nv -O focus-experiments-old.json https://hg.mozilla.org/releases/mozilla-beta/raw-file/default/mobile/android/focus-android/app/src/main/res/raw/initial_experiments.json
2024-06-28 02:54:29 URL:https://hg.mozilla.org/releases/mozilla-beta/raw-file/default/mobile/android/focus-android/app/src/main/res/raw/initial_experiments.json [17/17] -> "focus-experiments-old.json" [1]
+ /usr/bin/jq --arg APP_NAME fenix '{"data":map(select(.appName == $APP_NAME))}'
+ /usr/bin/jq --arg APP_NAME focus_android '{"data":map(select(.appName == $APP_NAME))}'
+ /usr/bin/diff -u fenix-experiments-old.json fenix-experiments-new.json
+ /usr/bin/diff -u focus-experiments-old.json focus-experiments-new.json
+ '[' -s /home/worker/artifacts/initial_experiments.diff ']'
+ return 0
+ MOBILE_EXPERIMENTS_UPDATED=true
+ '[' true == false ']'
+ '[' '' == true ']'
+ clone_repo
+ cd /builds/worker
+ '[' '!' -d mozilla-beta ']'
+ /usr/bin/hg robustcheckout --sharebase /tmp/hg-store -b default https://hg.mozilla.org/releases/mozilla-beta mozilla-beta
(using Mercurial 6.4.3)
ensuring https://hg.mozilla.org/releases/mozilla-beta@default is available at mozilla-beta
(sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
applying clone bundle from https://storage.googleapis.com/moz-hg-bundles-gcp-us-west1/releases/mozilla-beta/509d718631bbf5f7dfc599beab4bf9f51732fcfc.stream-v2.hg
838253 files to transfer, 5.11 GB of data
transferred 5.11 GB in 85.1 seconds (61.6 MB/sec)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 22 changesets with 109 changes to 102 files
new changesets a4a21962875c:0a432f12b22b
searching for changes
no changes found
(pulling to obtain default)
(remote resolved default to 0a432f12b22b3ee976f97cf92f2384eae7d7d2b2; result is not deterministic)
(revision already present locally; not pulling)
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
373611 files updated, 0 files merged, 0 files removed, 0 files unresolved
updated to 0a432f12b22b3ee976f97cf92f2384eae7d7d2b2
PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 91.55355501174927}, {"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "update", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 28.293495655059814}, {"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 122.38387131690979}, {"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 122.38387131690979}, {"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 122.38387131690979}, {"extraOptions": ["projects/970387039909/machineTypes/n2-custom-64-131072"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 122.38387131690979}]}
+ /usr/bin/hg -R mozilla-beta pull
pulling from https://hg.mozilla.org/releases/mozilla-beta
searching for changes
no changes found
+ /usr/bin/hg -R mozilla-beta update -C default
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update'
+ '[' true == true ']'
+ stage_hsts_files
+ cd /builds/worker
+ cp -f /builds/worker/firefox/nsSTSPreloadList.inc mozilla-beta/security/manager/ssl/
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS'
+ '[' true == true ']'
+ stage_hpkp_files
+ cd /builds/worker
+ cp -f /builds/worker/data/StaticHPKPins.h.out mozilla-beta/security/manager/ssl/StaticHPKPins.h
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP'
+ '[' true == true ']'
+ stage_remote_settings_files
+ cd /builds/worker
+ cp -a /builds/worker/data/remote-settings.out/blocklists /builds/worker/data/remote-settings.out/main /builds/worker/data/remote-settings.out/security-state mozilla-beta/services/settings/dumps
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings'
+ '[' true == true ']'
+ stage_tld_suffix_files
+ cd /builds/worker
+ cp -a public_suffix_list.dat mozilla-beta//netwerk/dns/effective_tld_names.dat
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes'
+ '[' true == true ']'
+ stage_mobile_experiments_files
+ cd /builds/worker
+ cp fenix-experiments-new.json mozilla-beta/mobile/android/fenix/app/src/main/res/raw/initial_experiments.json
+ cp focus-experiments-new.json mozilla-beta/mobile/android/focus-android/app/src/main/res/raw/initial_experiments.json
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments'
+ '[' false == true ']'
+ '[' false == true ']'
+ '[' true == true ']'
+ COMMIT_MESSAGE='No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments - a=repo-update'
+ /usr/bin/hg -R mozilla-beta commit -u 'ffxbld <[e-mail]>' -m 'No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes mobile-experiments - a=repo-update'
+ push_repo
+ cd mozilla-beta
+ '[' '!' -r /builds/worker/.arcrc ']'
++ command -v arc
+ ARC=
++ command -v arcanist
+ ARC=/usr/bin/arcanist
+ '[' -z '#release-managers' ']'
++ /usr/bin/arcanist list
++ grep 'Needs Review'
++ grep -E 'mozilla-beta repo-update'
++ awk 'match($0, /D[0-9]+[^: ]/) { print substr($0, RSTART, RLENGTH)  }'
+ /usr/bin/arcanist diff --verbatim --reviewers '#release-managers'
You have untracked files in this working copy.

  Working copy: /builds/worker/mozilla-beta/

  Untracked changes in working copy:
  (To ignore these 4 change(s), add them to ".hgignore".)
    services/settings/dumps/main/search-config-icons/39979158-80cf-4f0c-8bd6-9220eb09e751
    services/settings/dumps/main/search-config-icons/39979158-80cf-4f0c-8bd6-9220eb09e751.meta.json
    services/settings/dumps/main/search-config-icons/7bf4ca37-e2b8-4d31-a1c3-979bc0e85131
    services/settings/dumps/main/search-config-icons/7bf4ca37-e2b8-4d31-a1c3-979bc0e85131.meta.json

    Ignore these 4 untracked file(s) and continue? [y/N] 
[taskcluster:error] Task timeout after 14400 seconds. Force killing container.
[taskcluster 2024-06-28 03:53:22.614Z] === Task Finished ===
[taskcluster 2024-06-28 03:53:22.615Z] Unsuccessful task run with exit code: -1 completed in 14415.155 seconds

Comment 1

[Julien Cristau [:jcristau]]

This might be the first time the remote-settings dump update results in new files. Mark, can you confirm we should be hg add-ing them as part of the periodic update?

Comment 2

[Mark Banner (:standard8)]

(In reply to Julien Cristau [:jcristau] from comment #1)

This might be the first time the remote-settings dump update results in new files.

Most likely, and this is the first time through the process so we're discovering as we go. I also didn't expect this pfu run to be hit - I thought it'd be next week. Anyway...

Mark, can you confirm we should be hg add-ing them as part of the periodic update?

Unfortunately that's not enough, we also need to add them to the moz.build files as well, as wildcards don't work correctly - bug 1873408. Given bug 1873408 comment 2, I'm not sure we'd be able to get wildcards fixed anyway.

So it might be that bug 1873448 will need to somehow find a way to inject the relevant information into the moz.build files as well.

In the meantime, I wonder if we might be able to auto-ignore this changes in the pfu script?

The short term issue, we're going to fix by manually adding the files as part of bug 1905126.

Comment 3

[Julien Cristau [:jcristau]]

[Tracking Requested - why for this release]: broken periodic updates

Comment 4

[Julien Cristau [:jcristau]]

Depending on how long it takes to get this working again I guess we can skip the search-config-icons collection in the periodic-updates script, to still get the rest of the updates?

Comment 5

[Mark Banner (:standard8)]

(In reply to Julien Cristau [:jcristau] from comment #4)

Depending on how long it takes to get this working again I guess we can skip the search-config-icons collection in the periodic-updates script, to still get the rest of the updates?

We'll have patches up on bug 1905126 today.

Comment 6

[Mark Banner (:standard8)]

With the landing of bug 1905126 on beta, this should be fixed now.

Comment 7

[Intermittent Failures Robot]

1 failures in 4116 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905265&startday=2024-06-24&endday=2024-06-30&tree=all