1351326 - patches to disable bb nightlies on macosx

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Kim Moir [:kmoir] ET]

keep the talos tests running through bbb

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Sorry again for a gather-context kind of question, just to make sure I'm on the right track:

1. So far we've been working with BB jobs run through BBB and scheduled via TC. The machines behind BB were actually Mac phisical machines 
2. While we had the regression issue and didn't know how that would end, we considered moving towards TC workers that were running on Mac phisically machines
3. Ever since the X compiled builds are fashioanable yet again, we're looking at switching off BB entirely and moving towards running cross-compiled builds on actual TC Linux workers. So we don't need Mac machines at all in the near future.

@kmoir: Is this true?

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1351326 - patches to disable bb nightlies on macosx.

Review commit: https://reviewboard.mozilla.org/r/150818/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/150818/

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Assuming I'm not wrong, the following obs:

1. Currently we're still experimenting with X-compiled builds but we expect soon to make the switch to Tier1, hence merge the code from `date` to central. Whatever patches we end up writing for this bug, they need to land on central only *after* we've declared victory over the Tier-1 TC macos nightlies, right?

2. Is there a way I can check what my patch is affecting? A staging buildbot-master for this or something alike. 
@aselage: I remember you mentioned something like this during our morning meeting. Is there any docs on how I can do this?

Comment 4

[Alin Selagea [:aselagea]]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #3)

> 2. Is there a way I can check what my patch is affecting? A staging
> buildbot-master for this or something alike. 
> @aselage: I remember you mentioned something like this during our morning
> meeting. Is there any docs on how I can do this?

https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master#Create_a_tests_master has some useful data, but it's not really up-to-date.

Basically, what you need to do is:
    - login to dev-master2.bb.releng.use1
    - create your own folder under /builds/buildbot/
    - wget 'https://hg.mozilla.org/build/braindump/file/tip/buildbot-related/create-staging-master.pl'
    - clone 'braindump' repo
    - create your own build/test master
e.g. 'perl create-staging-master.pl -u mtabara --master-kind=build --master-dir=build-master --http-port=8013'
    - cd to master folder && activate virtual env
    - compute the initial list of builds/tests:
e.g. "../braindump/buildbot-related/builder_list.py master/master.cfg > old"
    - do the bb-configs changes you need
    - compute the list of builds/tests following your changes
e.g. "../braindump/buildbot-related/builder_list.py master/master.cfg > new"
    - compute the diff between them

Notes:
[1] you can adjust the branches/platforms you need for your master in <master_folder>/master/master_config.json
[2] to make sure your changes don't break anything on other masters when landed in production, you can run 'test-masters.sh' script located in <master_folder>/buildbot-configs/

If you need help, please feel free to ping me or the other buildduty folks on IRC. ^

Comment 5

[Kim Moir [:kmoir] ET]

Mihai your assessment is correct.  We would like to disable the nightly, and opt Macosx builds to ride the trains on buildbot.   I think the debug builds are already in this state. Further to Alin's links, this is also useful - for comparing jobs before and after your buildbot changes. 

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8879533 [details]
Bug 1351326 - patches to disable bb nightlies on macosx.

https://reviewboard.mozilla.org/r/150818/#review155624

So I think you're going to retest this patch on your master now that you have those instructions.  But one thing to fix is that that this change will need a new loops since now trunk is at 56 and the patches below are adding to loops that specify at least gecko 53

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1351326 - patches to disable bb nightlies on macosx.

Review commit: https://reviewboard.mozilla.org/r/151280/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/151280/

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Kim Moir [:kmoir] from comment #6)
> Comment on attachment 8879533 [details]
> Bug 1351326 - patches to disable bb nightlies on macosx.
> 
> https://reviewboard.mozilla.org/r/150818/#review155624
> 
> So I think you're going to retest this patch on your master now that you
> have those instructions.  But one thing to fix is that that this change will
> need a new loops since now trunk is at 56 and the patches below are adding
> to loops that specify at least gecko 53

Yeah, makes more sense now, thanks for the comments!
Repushed the patch to review with refactored code.

@aselagea: thanks a lot for super detailed instructions!
However, I'm traveling today and will have limited internet connection. Is there any chance you could test this patch for me should you already have that setup from https://bugzilla.mozilla.org/show_bug.cgi?id=1351326#c4 ?

Thank you!

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug1351326.patch

new patches to disable nightlies on macosx on trunk

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1351326builderdiff.patch

builder diff

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 8880000 [details]
bug1351326builderdiff.patch

Robert, just as an FYI this bug will end up disabling OSX nightlies from "Oak", they will be re-enabled once you get the cron.yml change (which we'll be doing in another bug) merged in, to support nightlies there. [and then adding "oak" to its projects].

Feel free to ping us in #releng and we can assist if the code is up to date. (If there is a problem here, we can probably special case oak to keep BB running in the short term)

Comment 12

[Robert Strong (they/them - no direct email)]

Thanks for the heads up... it should be ok and if we need oak for testing lzma and sha384 before they are enabled again I'll ping you.

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Bah, the reviewboard is so confusing when you follow-up a patch to review, it doesn't request the review request.

Just to make sure I understand this right for future changes like this:
@kmoir: in my followed-up patch from https://reviewboard.mozilla.org/r/150816/diff/2#index_header I also disabled opt unit tests for the newly disabled macosx nightlies but I see this missing from the patch you followed-up with. Is this something that we're not supposed to do too whenever we disable nightlies?

Comment 14

[Alin Selagea [:aselagea]]

In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)

> @aselagea: thanks a lot for super detailed instructions!
> However, I'm traveling today and will have limited internet connection. Is
> there any chance you could test this patch for me should you already have
> that setup from https://bugzilla.mozilla.org/show_bug.cgi?id=1351326#c4 ?
> 
> Thank you!

Sorry, I've been dealing with other bugs yesterday. It seems Kim created a new patch to address this though. Thanks, Kim!

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #14)
> In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)
> 
> > @aselagea: thanks a lot for super detailed instructions!
> > However, I'm traveling today and will have limited internet connection. Is
> > there any chance you could test this patch for me should you already have
> > that setup from https://bugzilla.mozilla.org/show_bug.cgi?id=1351326#c4 ?
> > 
> > Thank you!
> 
> Sorry, I've been dealing with other bugs yesterday. It seems Kim created a
> new patch to address this though. Thanks, Kim!

@aselagea: absolutely no worries!

Comment 16

[Kim Moir [:kmoir] ET]

Mihai tests on macosx are scheduled on tc on generic workers since last week.  This is why we didn't need to change the macosx test configs when we disabled the nightlies. 

I think we are done here! yay!

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Kim Moir [:kmoir] from comment #16)
> Mihai tests on macosx are scheduled on tc on generic workers since last
> week.  This is why we didn't need to change the macosx test configs when we
> disabled the nightlies. 
> 
> I think we are done here! yay!

Ah, right, I remember :wcosta's emails now. Sorry for the confusion, makes sense now.
Thanks!