1565259 - push gecko-dev mapping to an s3 bucket

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[:dhouse]

:catlee asked if vcs-sync could post the gecko-dev mappings to an s3 bucket (compressed?) instead of the mapper service

for poc/mvp: we have the global mapping file on disk, and so we could post that to s3 without needing to change the vcs-sync conversion process

Comment 1

[:dhouse]

I'll make this a priority to complete in Q4. I did not prioritize doing it this quarter since the mapper has been working, and so I never started the work.

Comment 2

[:dhouse]

This was completed this week because the mapper service was down after the TC migration last weekend.
vcs-sync was already pushing log archives to s3, so I added a push of the mappings (using s3api calls instead of s3put, and storing the md5 of the file to check before upload so we do not re-upload the same mapping)

Comment 3

[:dhouse]

Attachment: mappings_upload_to_s3.sh

Comment 4

[:dhouse]

With adding this, I reviewed the files in s3 for vcs-sync and added lifecycle policies to keep logs for 95days max (in glacier). The mapping is as standard storage type because we're replacing it daily (when new commits are added).

Comment 5

[:dhouse]

Attachment: mappings_upload_to_s3.sh