Closed Bug 1424482 Opened 2 years ago Closed 2 years ago

Get rid of balrog_props in beetmoverscript and beetmover jobs

Categories

(Release Engineering :: Release Automation: Other, enhancement, P2)

enhancement

Tracking

(firefox60 fixed)

RESOLVED FIXED
Tracking Status
firefox60 --- fixed

People

(Reporter: mtabara, Assigned: jlorenzo)

References

Details

Attachments

(4 files)

https://trello.com/c/5ck1TKn5/177-replace-balrogpropsjson-with-values-at-graph-generation-time-rather-than-downloading-a-build-artifact

This worked fine while we had nightlies for Linux. Then it gradually evolved to something nasty for all-nightly-platforms but it got extremely hackish for when we started adding releases.

This has to stop now. We need all that information in tree and have beetmover drop this dependency. The more products we need in beetmover, the more hackish this becomes.
Assignee: nobody → jlorenzo
Attached file beetmover PR
Attachment #8951196 - Flags: review?(mtabara)
Comment on attachment 8951196 [details] [review]
beetmover PR

Comments in the PR. Feel free to reset me again when you want me to glance again
Attachment #8951196 - Flags: review?(mtabara) → review+
Comment on attachment 8951308 [details]
Bug 1424482 - beetmover: Get rid of balrog_props in favor of task payload

https://reviewboard.mozilla.org/r/220560/#review226906

Nice to see tangiable progress here in such a short time, awesome work!
Attachment #8951308 - Flags: review?(mtabara) → review+
Not sure if those lint/f8 errors are solved already or not but might be worth running the checks before you consider landing this to central.
I triggered Fennec/Devedition staging releases to test my checksums changes and noticed two issues:

1. the release-generate-checksums-fennec job has "Firefox" within its payload release props. I'm reusing the functions that were added in this bug so I'm thinking there could be some error here https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/transforms/beetmover.py#l459 or we need to add "fennec" as part of that check too.  Because of appName = Firefox, beetmoverscript thinks it's a Firefox job, hence submits the files under the wrong bucket/path.

2. The "beetmover-source-linux64-fennec-source/opt" is also failing, most likely for the same reason as the release-props are the same as above. 


Offtopic: This reminds, we really need to resurrect the conversation with CloudOps to address the passwords / buckets issue.
Dropping a NI to Johan, to discuss this together on Monday, before I forget :}
Flags: needinfo?(jlorenzo)
Thank you for finding this out, Mihai!

The diff generated by taskcluster-diff.py is quite big to be humanely parsed. I used a few tricks to make sure I didn't create new platform values for instance, but I had missed this case. I relied on these commands:

> cd build/braindump/taskcluster/taskgraph-diff
> grep 'platform":' json/1424482/*.diff | cut -d':' -f'2-' | sort | uniq 
This checks no new platform was added by printing out all possible type of platforms.

> grep 'appName":' json/1424482/*.diff | cut -d':' -f'2-' | sort | uniq
Same thing with app names

> grep -i 'firefox"' json/1424482/*fennec*.jso
New check: Checks no fennec graph show any Firefox app name.
Flags: needinfo?(jlorenzo)
All devedition beetmover-repackage jobs are failing since we landed this patch. Turns out the hacks in beetmoverscript are quite ugly and make this difficult to debug.

In the old way of using balrog props, the platform would be set to `linux` whereas now, taken from in-tree is `linux-devedition`, which is correct one, but breaks the other automation.

When it gets in the `update_props()` function, it appends another "devedition" to it so some of the artifacts can't be found in the templating.

I pushed an inoffensive fix for this, https://github.com/mozilla-releng/beetmoverscript/pull/101/commits/f98a0d905398a791f66cb39fd899ff901b3c8e47 will roll-out another dev beetmoverscript.
Blocks: 1432219
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d69d6fe4a1b8
beetmover: Get rid of balrog_props in favor of task payload r=mtabara
https://hg.mozilla.org/mozilla-central/rev/d69d6fe4a1b8
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
See Also: → 1423182
Attachment #8955112 - Attachment is patch: true
Attachment #8955112 - Flags: review?(mtabara) → review+
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/e33efdb3e151
part 2: Remove upstream artifact when no path is defined r=mtabara a=bustage
Comment on attachment 8955112 [details] [diff] [review]
Bug 1424482 - part 2: Remove upstream artifact when no path is defined r=mtabara a=bustage

See comment 22.
Attachment #8955112 - Flags: checked-in+
Duplicate of this bug: 1423182
See Also: → 1442684
Blocks: 1449150
See Also: → 1443873
You need to log in before you can comment on or make changes to this bug.