1603988 - Perma [tier 2] bustage on symbols opt "error": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YNhHOTooRCO6mHjSgdRWRw/artifacts/public/build/target.crashreporter-symbols.zip can't be found (404)

Status: RESOLVED FIXED

(Socorro :: Symbols, defect, P1)

Description

[Bogdan Tara[:bogdan_tara | bogdant]]

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=281219990&repo=mozilla-central&lineNumber=158

https://firefoxci.taskcluster-artifacts.net/EeeCYPkVQ029JV67hYePBw/1/public/logs/live_backing.log

[task 2019-12-15T01:05:57.591Z] executing ['bash', '-cx', './mach python toolkit/crashreporter/tools/upload_symbols.py https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YNhHOTooRCO6mHjSgdRWRw/artifacts/public/build/target.crashreporter-symbols.zip']in /builds/worker/checkouts/gecko
[task 2019-12-15T01:05:57.592Z] + ./mach python toolkit/crashreporter/tools/upload_symbols.py https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YNhHOTooRCO6mHjSgdRWRw/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-12-15T01:05:58.367Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2019-12-15T01:05:58.367Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2019-12-15T01:05:58.367Z] Installing setuptools, pip, wheel...
[task 2019-12-15T01:05:59.590Z] done.
[task 2019-12-15T01:06:00.302Z] running build_ext
[task 2019-12-15T01:06:00.303Z] building 'psutil._psutil_linux' extension
[task 2019-12-15T01:06:00.303Z] creating build
[task 2019-12-15T01:06:00.303Z] creating build/temp.linux-x86_64-2.7
[task 2019-12-15T01:06:00.303Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-12-15T01:06:00.303Z] creating build/lib.linux-x86_64-2.7
[task 2019-12-15T01:06:00.303Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-12-15T01:06:00.303Z] building 'psutil._psutil_posix' extension
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-12-15T01:06:00.303Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-12-15T01:06:00.303Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-12-15T01:06:00.303Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-12-15T01:06:00.303Z]
[task 2019-12-15T01:06:00.303Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-12-15T01:06:00.635Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2019-12-15T01:06:00.731Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YNhHOTooRCO6mHjSgdRWRw/artifacts/public/build/target.crashreporter-symbols.zip" to "https://symbols.mozilla.org/upload/"
[task 2019-12-15T01:06:00.731Z] INFO:upload-symbols:Attempt 1 of 5...
[task 2019-12-15T01:06:01.294Z] ERROR:upload-symbols:Error: got HTTP response 400: Bad Request
[task 2019-12-15T01:06:01.380Z] ERROR:upload-symbols:Response body:
[task 2019-12-15T01:06:01.380Z] ====================
[task 2019-12-15T01:06:01.380Z] {"error": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YNhHOTooRCO6mHjSgdRWRw/artifacts/public/build/target.crashreporter-symbols.zip can't be found (404)"}
[task 2019-12-15T01:06:01.380Z] ====================
[task 2019-12-15T01:06:01.380Z]
[taskcluster 2019-12-15 01:06:01.750Z] === Task Finished ===
[taskcluster 2019-12-15 01:06:01.751Z] Unsuccessful task run with exit code: 1 completed in 112.998 seconds

Comment 1

[Bogdan Tara[:bogdan_tara | bogdant]]

Tom, could you, please, take a look?

Comment 2

[Intermittent Failures Robot]

4 failures in 4867 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform breakdown:

• symbols: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 3

[Tom Prince [:tomprince]]

The failure was happening because the system-symbols task failed (due to disk full) silently, thus there being no zip file to upload. I suspect this may also be an issue if there are no new symboles to upload.

Comment 4

[Marco Castelluccio [:marco]]

This was caused by the migration of the OS symbols task from bug 1596775. The job was previously not failing when there were no symbols to upload (https://github.com/marco-c/breakpad-mac-update-symbols/blob/21221733edfbcac49d40e50fc219fab7d17437a0/upload.sh#L10-L12).

Comment 5

[Marco Castelluccio [:marco]]

So we need to do two things:

1. Increase the disk space available for the symbol fetching tasks (and make them fail loudly when the disk space is exhausted);
2. Don't fail in the upload task if there are no symbols to upload.

Comment 6

[Intermittent Failures Robot]

10 failures in 4582 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform breakdown:

• symbols: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 7

[Intermittent Failures Robot]

10 failures in 1365 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform breakdown:

• symbols: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 8

[Andreea Pavel [:apavel]]

There are 11 total failures in the last 7 days on symbols opt.
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-13&endday=2019-12-20&tree=all&bug=1603988

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283433801&repo=mozilla-central&lineNumber=169

Comment 9

[Intermittent Failures Robot]

10 failures in 1862 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform breakdown:

• symbols: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 10

[Intermittent Failures Robot]

8 failures in 4472 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform breakdown:

• symbols: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 11

[Andreea Pavel [:apavel]]

Recent occurences: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=284788404&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&searchStr=symbols%2Copt%2Csystem-symbols-mac-upload-symbols%2C%28symsystem-symbols-mac%29&revision=0690f68a8d9984d2653792e3294bf876eecef651

Comment 12

[Marco Castelluccio [:marco]]

The "No space left on device" error is a bit strange. The worker types we were using in the previous hook, which was not failing, had VolumeSize: 100, the new ones have VolumeSize: 120.

Comment 13

[Calixte Denizet (:calixte)]

Attachment: Bug 1603988 - Part 1: Avoid to fail system-symbols-mac-upload-symbols when no symbols to upload

Currently, when task system-symbols-mac doesn't produce artifacts because of no data or failure, then the task system-symbols-mac-upload-symbols is failing too.
So this patch aims to not fail the task in such a case.

Comment 14

[Marco Castelluccio [:marco]]

Regarding the disk space issue, we'll need to create a new worker type for these tasks with larger disks. We can also use smaller instance types, 4xlarge are larger than what we need.

Comment 15

[Marco Castelluccio [:marco]]

We should also bring back the cache of processed artifacts that was dropped as part of the migration, to make the Mac task finish earlier.

Comment 16

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #15)

We should also bring back the cache of processed artifacts that was dropped as part of the migration, to make the Mac task finish earlier.

I've filed bug 1609130 for this.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2c381bb89f53
Avoid to fail system-symbols-mac-upload-symbols when no symbols to upload r=marco

Comment 18

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/2c381bb89f53

Comment 19

[Marco Castelluccio [:marco]]

Only half of it has been fixed.

Comment 20

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/96793d65cdd5
Backed out changeset 2c381bb89f53 for causing symbols build bustages. a=backout

Comment 21

[Cosmin Sabou [:CosminS]]

Backed out on central: https://hg.mozilla.org/mozilla-central/rev/96793d65cdd538f5f58b66ae2cbf9357f6232a98

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Symbol upload failures which caused the backout: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel%2Csuperseded%2Cretry&revision=2ee6c4c250523ed74cc95c155abc0fd802113d2e&searchStr=symbol

Comment 23

[Intermittent Failures Robot]

4 failures in 3773 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform breakdown:

• symbols: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 24

[Calixte Denizet (:calixte)]

I made a try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b8534fa775a8a99520775309ea5bf7d4b715895b
and it seems to be good.

Comment 25

[Calixte Denizet (:calixte)]

Attachment: Bug 1603988 - Part 2: Add a TC index for generated artifacts for mac symbols and reuse them in the next run

To avoid to process again and again the same packages, we can use a cache.
So this patch add a TC index for the artifacts generated in a run and then we can reuse it in the next run.

Comment 26

[Calixte Denizet (:calixte)]

I made a try to use the cache generated in a previous run:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=286152719&revision=8bdb489826ff185b1710f9dfc08d7e090cb3fab9
Everything is green and the task took only few minutes instead of 3 hours !

Comment 27

[Intermittent Failures Robot]

10 failures in 4004 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform breakdown:

• symbols: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 28

[Marco Castelluccio [:marco]]

(In reply to Calixte Denizet (:calixte) from comment #25)

Created attachment 9122657 [details]
Bug 1603988 - Part 2: Add a TC index for generated artifacts for mac symbols and reuse them in the next run

To avoid to process again and again the same packages, we can use a cache.
So this patch add a TC index for the artifacts generated in a run and then we can reuse it in the next run.

This patch is going to fix bug 1609130.

Comment 29

[Intermittent Failures Robot]

9 failures in 2190 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9

Platform breakdown:

• symbols: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/84ed6a978242
Part 1: Avoid to fail system-symbols-mac-upload-symbols when no symbols to upload r=marco,gsvelto

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1091ce1d58ec
Part 2: Add a TC index for generated artifacts for mac symbols and reuse them in the next run r=tomprince

Comment 32

[Marco Castelluccio [:marco]]

The bug won't be fully fixed until we switch to a new worker type with a larger disk.

Comment 33

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/84ed6a978242
https://hg.mozilla.org/mozilla-central/rev/1091ce1d58ec

Comment 34

[Intermittent Failures Robot]

1 failures in 3364 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• symbols: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603988&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 35

[Will Kahn-Greene [:willkg] ET needinfo? me]

Is this still an issue?

Comment 36

[Andreea Pavel [:apavel]]

No recent failures: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2020-10-15&endday=2021-02-12&tree=trunk&bug=1603988

Comment 37

[Calixte Denizet (:calixte)]

:apavel, thank you.
If the issue appears again we could then change the worker.