1502995 - Uploads to tooltool don't work

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Ryan was trying to upload something to tooltool today, and kept hitting problems trying to access the file afterwards.

Looking at our S3 buckets, it doesn't look like we've had any successful uploads since Sept 7th.

Also, the logging is a bit confusing. There are lines in papertrail that look like:
Oct 29 15:04:11 54.91.40.77 mozilla/release-services/production/tooltool/api: 2018-10-29 19:04:11.242065 tooltool_api.api: Sending digest `7dfd3e8fb12dac7ab4650c33ce8c7fca1430734ce4fd7fe258bfe5163d38682e07f3e1ca93c592eaa12d4a234cee9f1d10c794c6584fb25318d1899633c05778` to queue `exchange/releng-services-production/tooltool/api` exfor route `check_file_pending_uploads`. (tooltool_operation='download_file' tooltool_sha512='4c200d59db17afcaa4deec9f083a75362b8baf0974483cb573b37587d605315513590c076f263f0a5e9165695751a076a839579e169a24507daf4cbc2b30d85e') 

I don't think tooltool_operation is correct here. Perhaps the logging binding isn't working properly?

Comment 1

[Jordan Lund (:jlund)]

firebot> https://bugzil.la/1502995 — NEW, [e-mail] — Uploads to tooltool don't work
11:33:04 
<RyanVM> note that this is blocking win aarch64 work
11:34:16 
<•dlabici|ciduty> RyanVM: I wonder if its not because of the permissions, when we changed platforms even CiDuty had issues, I could try to do an upload, just to test an make sure its working
11:34:36 I'm also wondering if he updated the uploading script, so it doesn't use the old fqdn
11:35:03 bz → bz_away
11:35:07 
<RyanVM> I'm using https://github.com/mozilla/build-tooltool/blob/master/tooltool.py
11:35:16 
<•dlabici|ciduty> Tho I see a few extra issues pointed by Catlee
11:35:17 
<RyanVM> everything claims to work in the console
11:38:05 
<•dlabici|ciduty> Lemme do a quick test
11:41:54 stephend|afk → stephend
11:42:41 
<catlee> RyanVM: yeah, except when I look in the S3 buckets , there's nothing new in there
11:42:47 maybe I'm not looking in the right place
11:43:32 
<RyanVM> that seem consistent with what I'm seeing at least (no download icon for my recent uploads and failing attempts to use them in CI)
11:45:40 
<•dlabici|ciduty> Yeah, I see the same outcome.
11:46:03 Upload happened, but no download link provided in the console and if I manually search for it, there is no download icon

Comment 2

[Rok Garbas [:garbas]]

Attachment: GitHub Pull Request

Sorry I wasn't aware this is a blocker.

I already released the patch to production and tested it by uploading a file. I can confirm that the download icon was visible.

:RyanVM Can you try uploading again?

:jlund thank you for pasting the logs from the IRC. i was already AFK at that time.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

We're good, thanks!