1487141 - fetch-win32-symbols and fetch-mac-update-symbols not uploading symbols

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[(not currently active) Ted Mielczarek]

We didn't get symbols for the latest Microsoft Patch Tuesday updates, which was filed as bug 1486825. Looking into it I found that the task to fetch the symbols was succeeding, but the task to upload them was failing:
https://tools.taskcluster.net/groups/VfdD4pJLT327vS6fONCQyw

Turns out both the fetch-win32-symbols and fetch-mac-update-symbols jobs are using the outdated private Docker image to upload symbols:
https://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/file/a4190e71d18f/upload-task.json
https://github.com/luser/breakpad-mac-update-symbols/blob/75a4cd731f81212943b084c07010b64beef839d6/upload-task.json

...and it looks like someone finally deleted that Docker image. 

The right way to fix this would be to make those tasks use a token stored as a taskcluster secret, which is what we do for Firefox symbol uploads now:
https://dxr.mozilla.org/mozilla-central/rev/3205543f957cd2a6905486d73aa897535fdd9825/taskcluster/ci/upload-symbols/kind.yml#20

It shouldn't be very hard, I just haven't had time to do it.

Comment 1

[Marcia Knous [:marcia]]

Hello Ted - We are getting closer to end game for 63 - any progress on this?

Comment 2

[(not currently active) Ted Mielczarek]

I did another manual upload but I'm having a hard time finding time to fix the underlying issue here. I'd be happy to provide info to guide someone else to fix it.

Comment 3

[(not currently active) Ted Mielczarek]

We met at the All Hands last week to discuss handing over responsibility for these cron jobs to RelMan, so hopefully this will get fixed soon and future breakage won't be blocked on me fixing it personally.

Comment 4

[Stephen A Pohl [:spohl] (OOO until 2/27)]

What is the expected timeline for this work? We need symbols for 10.14.2 18C54 (see bug 1517012). Should I go ahead and scrape those symbols manually, or will we have this bug fixed soon so that I don't need to worry about it?

Comment 5

[Marco Castelluccio [:marco]]

I hope I will get to fix the upload by the end of the week. You can upload them manually if you want them asap.

Comment 6

[Marco Castelluccio [:marco]]

The Mac tasks should be fixed, spohl could you verify?

Comment 7

[Stephen A Pohl [:spohl] (OOO until 2/27)]

(In reply to Marco Castelluccio [:marco] from comment #6)

The Mac tasks should be fixed, spohl could you verify?

Redirecting to Marcia. I'm guessing reprocessing a crash report from bug 1517012 would be sufficient?

Comment 8

[Marco Castelluccio [:marco]]

The Windows tasks should be fixed too.

The code now lives at https://github.com/marco-c/breakpad-mac-update-symbols and https://github.com/marco-c/breakpad-win-update-symbols. It's still hacky, maybe in the future we will move it to https://github.com/mozilla/release-services.

Comment 9

[Marcia Knous [:marcia]]

(In reply to Stephen A Pohl [:spohl] from comment #7)

(In reply to Marco Castelluccio [:marco] from comment #6)

The Mac tasks should be fixed, spohl could you verify?

Redirecting to Marcia. I'm guessing reprocessing a crash report from bug 1517012 would be sufficient?

I was able to successfully reprocess one crash from that bug.