Closed Bug 1365422 Opened 6 years ago Closed 5 years ago

write a script to populate a fake previous release

Categories

(Release Engineering :: Release Automation: Other, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: jlorenzo)

References

Details

Attachments

(2 files)

We need these for staging releases, specifically devedition today.
14:58 <aki> hey rail! wondering if you need any help with dawn stuff
15:02 <rail> thanks for asking! one of the known things is that we need to create a fake previous release, otherwise we would need to change a lot in our automation. maybe something like a script that would download a CI build (probably just complete MARs) and upload it as a fake release
15:03 <rail> no need to generate l10n, we can reuse en-US MARs 
15:03 <rail> also we may need to generate the cheksums files
15:03 <rail> SHA512sUMS
15:04 <aki> this is basically populating candidates/ ?
15:04 <rail> yeah
15:04 <rail> or releases, I forgot which one we use
15:04 <rail> we can copy
15:04 <rail> otherwise I'm just waiting for a CI build to start a release to uncover more issues :)
15:05 <rail> there is no bug for the fake release scrip, feel free to file one

Guessing this would be boto3-based.
If the bits already live in the proper s3 bucket, we could also

s3cmd -c CFG_PATH cp "s3://BUCKET/KEY1" "s3://BUCKET/KEY2"

However, if we're based off the nightly-signing devedition tasks in https://treeherder.mozilla.org/#/jobs?repo=jamun&selectedJob=99521191 , we need to find the nightly signing tasks, find the target.complete.mar, and then upload to s3.
Blocks: 1353825
I manage to find the complete MARs attached as artifacts to the corresponding TC task (which is used by release promotion):

https://tools.taskcluster.net/index/#gecko.v2.jamun.latest.firefox.macosx64-opt/gecko.v2.jamun.latest.firefox.macosx64-opt pointed me to https://tools.taskcluster.net/task-inspector/#SiTO4UE6TuaiPqLLN0QcuA/0 which has those.
I forked aki's script (comment 4) and started to adapt it for new locations.
Assignee: nobody → jlorenzo
download() is implemented
Attachment #8868619 - Flags: review?(aki)
Attachment #8868619 - Flags: review?(aki) → review+
Attached file Implement upload() PR
(New thumb-wrestle round!)

This PR uploads the same mar for every locale (with one mar per architecture). At the end, it also uploads SHA512SUMS.

It has been tested on this staging bucket[1]. We'll need to release beetmoverscript and mozapkpublisher on Pypi (the latter because of [2]).

[1] https://console.aws.amazon.com/s3/buckets/mozilla-releng-beet-mover-dev/pub/devedition/candidates/54.0b1-candidates/build1
[2] https://github.com/mozilla-releng/mozapkpublisher/pull/29
Attachment #8869094 - Flags: review?(mtabara)
Attachment #8869094 - Flags: review?(jlund)
Comment on attachment 8869094 [details] [review]
Implement upload() PR

not really a hard r- but I wanted to point out I have some open questions in Github. Please r? me again when those have been replied to or addressed.
Attachment #8869094 - Flags: review?(mtabara)
Attachment #8869094 - Flags: review?(jlund)
Attachment #8869094 - Flags: review-
Comment on attachment 8869094 [details] [review]
Implement upload() PR

Sorry about the confusion. I tested for one platform, and it turned out uploading 5 platforms at the same time was an issue.

I investigated it and tweaked the network config to start uploading batch of 93 files (aka 1 platform at once). I detailed the context in the code. 

So here's the corrected version. I tested it twice, by looking at the following:
* Number of files within each platform folder => 93 files per platform
* Size of the devedition folder (27.7GB, which corresponds to 93*(sum of size of the 5 mars)
* Number of lines in SHA512SUMS => 465 which equals to 93*5
* Format of the SHA512SUMS
* Network usage which is about 27GB

For reference, it took about 23 minutes to run the scriptm with a 200 Mbps upload connection. I'd recommend to run it either in a datacenter or behind an optic fiber connection.
Attachment #8869094 - Flags: review- → review?(jlund)
Priority: -- → P1
Comment on attachment 8869094 [details] [review]
Implement upload() PR

forgot to clear this after reviewing in github last week
Attachment #8869094 - Flags: review?(jlund) → review+
Let's mark it as done. We ended up not using this unfortunately. :/ I just created a fake bouncer entry pointing to one of the betas to avoid any confusion because of the files published.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.