1083319 - Trigger nightly builds after merge day uplift automatically

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

In https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Kick_off_m-c_and_m-a_nightlies we trigger nightly builds when we bump the version on mozilla-central and merge mozilla-central to aurora. We should script this.

Comment 1

[[e-mail] (:bhearsum)]

Going to look at this soon. Might end up fixing this and bug 1083324 in the same patch.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: trigger nightlies and bundles in gecko migration script

I haven't done a whole lot of testing on this, but I'm also not quite sure how to, given that staging buildapi only accepts requests and doesn't do anything with them. The testing I have done is to print out what request would've been made, and to that end things look okay. Honestly, the best way to verify this is probably as part of the next migrations, given that I'm on the hook for them.

This ended up being relatively simple, so I'm worried that I'm missing something. Here's the summary though:
* Create a SelfServeMixin with a method for triggering arbitrary builders.
* Add a "trigger_builders" step in the gecko migration script that looks over the builders specified in the config and triggers them. I'm not particularly pleased about putting all those builder names in the config. We could use the nightly build triggering to avoid nightly builder names being there, but we'd still need to trigger the bundles as we are now, so I thought it was simplest to just do everything the same way.

I wasn't sure which nightly builders we trigger, so I kept them all in the list for now. I kindof think that we probably don't need to trigger debug/asan/localizer - that sort of thing. Please let me know what the SOP here is and I'll adjust it.

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8523058 [details] [diff] [review]
trigger nightlies and bundles in gecko migration script

Review of attachment 8523058 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/merge_day/central_to_aurora.py
@@ +65,5 @@
> +        "comm-central hg bundle",
> +        # Nightly builds
> +        "Android 2.3 mozilla-aurora nightly",
> +        "Android 4.2 x86 mozilla-aurora nightly",
> +        "Linux mozilla-aurora nightly",

Hmmm.... I wonder how to keep the list up to date or dry run this action in advance to avoid exceptions or silent errors.

This doesn't block this patch, but would be great to have a followup.

::: scripts/merge_day/gecko_migration.py
@@ +676,5 @@
>                      message = error_message
>                  self.fatal(message)
>  
> +    def trigger_builders(self):
> +        branch = self.config["to_repo_url"].split("/")[-1]

Can you rstrip("/") before splitting to avoid situations like bug 1082136?
Something like

 branch = self.config["to_repo_url"].rstrip("/").split("/")[-1]

BTW, would be great to start adding unit tests for this script.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: new and improved triggering

I think this is much better:
* Use nightly build endpoint of self serve instead of specifying builders - this should eliminate any maintenance issues.
* Remove comm builders/branches, because we don't deal with merging them
* Move self serve servers to separate configs, to make testing in staging easier.
* Fix SelfServeMixin to be able to import requests
* Fix urls not to have extra "/" in them.
* Add some logging and comments

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8524115 [details] [diff] [review]
new and improved triggering

Review of attachment 8524115 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/merge_day/gecko_migration.py
@@ +695,5 @@
> +           builders to trigger that aren't part of the nightly builder set.
> +           For example: hg bundle generation builders.
> +        """
> +        dirs = self.query_abs_dirs()
> +        branch = self.config["to_repo_url"].split("/")[-1]

conditional r+ on replacing the line above to

branch = self.config["to_repo_url"].rstrip("/").split("/")[-1]

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 8524115 [details] [diff] [review]
new and improved triggering

Landed with the rstrip fix.

Comment 8

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/mozharness/rev/c4060dfc1b65

Comment 9

[[e-mail] (:bhearsum)]

Turns out this doesn't work because of bug 1104252. Most likely need to change it to use the public self serve interface, and somehow deal with the fact that it needs authentication.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: use public buildapi interface

I'm not terribly happy with this, but it works. I chose to put the username/password prompts in the script rather than the library, because I don't want a library to require user interaction - other scripts could potentially feed auth through via private files on machines (eg, oauth.txt).

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8527907 [details] [diff] [review]
use public buildapi interface

Ouch... Let's file a bug to make it work non-interactive so we can run it from buildbot in the future.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8527907 [details] [diff] [review]
use public buildapi interface

Review of attachment 8527907 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Rail Aliiev [:rail] from comment #11)
> Comment on attachment 8527907 [details] [diff] [review]
> use public buildapi interface
> 
> Ouch... Let's file a bug to make it work non-interactive so we can run it
> from buildbot in the future.

bug 1104283.

Comment 13

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/3d598caf7661