944005 - Rewrite Clobberer Service

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Hopkins (:jhopkins)]

Based on the objectives in:

 https://etherpad.mozilla.org/newclobberer

Code will live here initially:

 http://hg.mozilla.org/users/jhopkins_mozilla.com/newclobberer/

Comment 1

[John Hopkins (:jhopkins)]

Need to put this on the back burner while I work on bug 886640.  Happy to review any contributed code, however!

Comment 3

[Morgan Phillips [:mrrrgn]]

Attachment: GitHub PR

This minimal implementation has been tested with the client and the current clobberer db implementation for compatibility.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8504147 [details] [review]
GitHub PR

(reviewed on github)

Comment 5

[Morgan Phillips [:mrrrgn]]

Attachment: relengapi-clobberer endpoint which replicates always_clobber.php behavior

Please let me know if I'm driving you all nutty and should request reviews from more folks.

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8505939 [details] [review]
relengapi-clobberer endpoint which replicates always_clobber.php behavior

reviewed on github, r+

Comment 7

[Morgan Phillips [:mrrrgn]]

Attachment: swapping out clobberer api in staging configs

Comment 8

[Morgan Phillips [:mrrrgn]]

Client tests and tests using a private buildmaster are working with these changes: http://dev-master1.srv.releng.scl3.mozilla.com:8046/builders/WINNT%205.2%20try%20build/builds/21/steps/checking_clobber_times/logs/stdio

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8506349 [details] [diff] [review]
swapping out clobberer api in staging configs

The patch is trivial, let's save some time for bhearsum/catlee

Comment 10

[Morgan Phillips [:mrrrgn]]

Attachment: swapping out clobberer api in staging configs (mozharness)

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8506364 [details] [diff] [review]
swapping out clobberer api in staging configs (mozharness)

ship it!

Comment 12

[Morgan Phillips [:mrrrgn]]

Attachment: Adds Angular UI

Just keeping things in sync between BZ and GH

Comment 13

[Morgan Phillips [:mrrrgn]]

Attachment: Fixes Specific Slave Clobbering Logic

This has been tested with old clobberer for backwards compatibility with good success.

Comment 14

[Morgan Phillips [:mrrrgn]]

Attachment: Remove users' ability to see release builds

Ben, I got good feedback about the code; but would feel good if you gave this a final thumbs up before I merge since we discussed it on IRC.

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8506349 [details] [diff] [review]
swapping out clobberer api in staging configs

https://hg.mozilla.org/build/buildbot-configs/rev/9ed7b194f428

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8506364 [details] [diff] [review]
swapping out clobberer api in staging configs (mozharness)

https://hg.mozilla.org/build/mozharness/rev/1ad50eb7f82a

Comment 17

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #15)
> Comment on attachment 8506349 [details] [diff] [review]
> swapping out clobberer api in staging configs
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/9ed7b194f428

FTR, I had to skip the esr24 related hunks. esr24 is dead! :)

Comment 18

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9ed7b194f428

Comment 19

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/1ad50eb7f82a

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 8509643 [details] [review]
Remove users' ability to see release builds

I think this is no longer needed because the PR is closed?

Comment 21

[Morgan Phillips [:mrrrgn]]

Attachment: Remove users' ability to see/clobber release builds

Ben, this took your suggestion to the previous PR into account. I caused a nasty merge situation in the other pull request and decided to abandon it (Travis wasn't working because of the merge errors).

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 8510396 [details]
Remove users' ability to see/clobber release builds

I left one comment on the PR, but nothing that blokcs.

Comment 23

[Morgan Phillips [:mrrrgn]]

Attachment: A Nice Search Feature for the UI

Comment 24

[Morgan Phillips [:mrrrgn]]

Everything's done on clobberer that needs doing until the old clobberer is deprecated, you can see the status here: https://etherpad.mozilla.org/newclobberer

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 8507904 [details] [review]
Adds Angular UI

addressed in github

Comment 26

[Morgan Phillips [:mrrrgn]]

Attachment: Add deprecation message to current clobberer

Just a message in an h1 tag

Comment 27

[Morgan Phillips [:mrrrgn]]

Attachment: Buildbot Configs Change to New Clobberer (Prod)

Comment 28

[Morgan Phillips [:mrrrgn]]

Attachment: Mozharness to new prod clobberer

Comment 29

[Morgan Phillips [:mrrrgn]]

Attachment: Ash mozharness to new prod clobberer

Comment 30

[Morgan Phillips [:mrrrgn]]

Attachment: Puppet manifests to new prod clobberer

Comment 31

[Morgan Phillips [:mrrrgn]]

Attachment: Puppet to new prod clobberer

Comment 32

[Morgan Phillips [:mrrrgn]]

Attachment: relabs-puppet-clobberer-prod.diff

Comment 33

[Morgan Phillips [:mrrrgn]]

Attachment: tools-scripts-clobberer-prod.diff

I waited a bit on these because I wasn't familiar with them. Looked them over, and this seems right. Mostly these were just fallback values hard coded into the script.

Comment 34

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/7c491b4779c1

Comment 35

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/mozharness/rev/dd409298ca1a

Comment 36

[Morgan Phillips [:mrrrgn]]

Closing this task, since it's now a part of the production configs. I'll make smaller bugs to cover any small cleanup tasks.

Comment 38

[Pete Moore [:pmoore][:pete]]

Do we have bugs on file already to:

1) bring down old clobberer (https://secure.pub.build.mozilla.org/clobberer/)
2) remove old clobberer service from source control (under http://hg.mozilla.org/build/tools/file/e61d4d45346d/clobberer)
3) remove references to old clobber from the wiki (https://wiki.mozilla.org/ReleaseEngineering/Applications/Clobberer)
4) update mana pages:
       https://mana.mozilla.org/wiki/display/IT/Clobberer
       https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=24805981
5) Remove CNAME records from inventory (https://inventory.mozilla.org/en-US/core/search/#q=clobberer)
6) decommission old clobberer servers
7) clean up old nagios checks for the old clobberer environments

Other possible work:
1) I see a seamonkey-clobberer.mozilla.org CNAME in inventory, but I don't think we are hosting any seamonkey builders, can we decomm that too? (question for Callek)
2) I couldn't find references to the old clobberer service in our puppet repo - but is that managed by an IT puppet repo instead? If so, maybe we need to pull it too (or ask IT to pull it).
3) Are there any secrets stored anywhere which should be cleaned up relating to the old service?

Apologies if some or all of this is covered elsewhere already.

Thanks,
Pete

Comment 39

[Pete Moore [:pmoore][:pete]]

(In reply to Morgan Phillips [:mrrrgn] from comment #36)
> Closing this task, since it's now a part of the production configs. I'll
> make smaller bugs to cover any small cleanup tasks.

Sorry, only just seen this! :D

Hopefully the list is useful anyway.

Comment 40

[Justin Wood (:Callek)]

(In reply to Pete Moore [:pete][:pmoore] from comment #38)
> Other possible work:
> 1) I see a seamonkey-clobberer.mozilla.org CNAME in inventory, but I don't
> think we are hosting any seamonkey builders, can we decomm that too?
> (question for Callek)

Just to answer, we can't kill off that CNAME. Mozilla does host seamonkeys machines, but 'releng' does not, they are completely managed by community (which includes me though).

That CNAME is necessary last I checked, and doesn't point to any of the managed-by-releng systems.

Comment 41

[Pete Moore [:pmoore][:pete]]

(In reply to Justin Wood (:Callek) from comment #40)
> (In reply to Pete Moore [:pete][:pmoore] from comment #38)
> > Other possible work:
> > 1) I see a seamonkey-clobberer.mozilla.org CNAME in inventory, but I don't
> > think we are hosting any seamonkey builders, can we decomm that too?
> > (question for Callek)
> 
> Just to answer, we can't kill off that CNAME. Mozilla does host seamonkeys
> machines, but 'releng' does not, they are completely managed by community
> (which includes me though).
> 
> That CNAME is necessary last I checked, and doesn't point to any of the
> managed-by-releng systems.

Will it cause seamonkey any problems if we deprecate the old clobberer? For example, do they rely on our tools repo, or do they have their own fork?

Comment 42

[Justin Wood (:Callek)]

It's forked.  The rest of this plan looked good to me

Comment 43

[Ed Morley [:emorley]]

For anyone else who was struggling to find the new RoR (the repo in comment 0 is out of date, and the various Mana/wiki pages don't link to the source afaict), it's here:
https://github.com/mozilla/build-relengapi-clobberer

I've also filed an issue for adding links to the source in the footer of the clobberer:
https://github.com/mozilla/build-relengapi-clobberer/issues/31

Comment 44

[Justin Wood (:Callek)]

Just as an fyi, all our repos are now listed at https://wiki.mozilla.org/ReleaseEngineering/Repositories as well