1372241 - kill Linux and OS X buildbot-based addon devel builds everywhere

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P1)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

RyanVM> mtabara|mergeduty: another fun one - Linux addon builds are busted with "07:34:46     INFO -  /usr/bin/env: /tools/buildbot/bin/python: No such file or directory"
15:38:37 <RyanVM> not sure we even care about those builds...
15:39:47 <mtabara|mergeduty> RyanVM: thanks for the heads-up! I'll file a bug and poke around for some help. I don't have too much context on that end.
15:40:24 <RyanVM> mtabara|mergeduty: please CC :sheriffs when you do
15:41:07 <mtabara|mergeduty> sure, thanks
15:42:49 <mtabara|mergeduty> RyanVM: https://hg.mozilla.org/releases/mozilla-beta/rev/93f21ca4b60b should fixed the earlier scope issues. I suppose we should/can cancel the jobs that were already scheduled before that change to spare some cpu time?
15:42:59 <RyanVM> meh
15:43:08 <RyanVM> lots of busted builds anyway :P
15:43:17 <RyanVM> i'd rather see what I can see from those pushes
15:45:48 <RyanVM> mtabara|mergeduty: is there a bug filed for updating bugzilla yet?
15:47:53 <RyanVM> mtabara|mergeduty: we have a Dawn meeting in 13min, I can ask there if we even need the addon-devel builds anymore

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Not sure if this is gonna be discussed at the Dawn meeting today. If it does, please ignore this. If not, can you please, Ben, tell me if we need these builds or not? I don't have too much context on this.

Thanks!

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1)
> Not sure if this is gonna be discussed at the Dawn meeting today. If it
> does, please ignore this. If not, can you please, Ben, tell me if we need
> these builds or not? I don't have too much context on this.
> 
> Thanks!

We talked about this briefly. We can turn these builds off for mozilla-beta (DevEdition takes their place), but they still need to work on mozilla-release (because we have no other unsigned addon builds there).

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Wait a minute, we have TC-based versions of these builds too.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=linux%20addon

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Stealing some context from #releng for later use:

16:43:37 <RyanVM> bhearsum: I missed earlier that we run TC-based linux addon builds too
16:43:46 ⇐ FlorinMezei quit ([e-mail]) Quit: KVIrc 4.2.0 Equilibrium http://www.kvirc.net/
16:43:51 — RyanVM votes killing the bb ones with fire unless there's another reason for keeping them I can't think of
16:43:55 <~bhearsum> ok
16:44:05 <~bhearsum> i think we should kill them all on mozilla-beta

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)
> Stealing some context from #releng for later use:
> 
> 16:43:37 <RyanVM> bhearsum: I missed earlier that we run TC-based linux
> addon builds too
> 16:43:46 ⇐ FlorinMezei quit ([e-mail]) Quit: KVIrc
> 4.2.0 Equilibrium http://www.kvirc.net/
> 16:43:51 — RyanVM votes killing the bb ones with fire unless there's another
> reason for keeping them I can't think of
> 16:43:55 <~bhearsum> ok
> 16:44:05 <~bhearsum> i think we should kill them all on mozilla-beta

Just to spell it out a bit more, we should kill the buildbot-based addon builds on both mozilla-beta (which are currently failing anyways) and mozilla-release, because we run TC ones as well. We should also keep the TC ones turned for mozilla-beta to ensure that they continue to work when we uplift to mozilla-release.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

More context:

RyanVM> bhearsum: I missed earlier that we run TC-based linux addon builds too
16:43:51 — RyanVM votes killing the bb ones with fire unless there's another reason for keeping them I can't think of
16:43:55 <~bhearsum> ok
16:44:05 <~bhearsum> i think we should kill them all on mozilla-beta
16:44:42 <~bhearsum> the only argument i could see for keeping them is to make sure we catch bustage there instead of mozilla-release
16:44:47 <~bhearsum> which is...not a terrible reason
16:45:16 <~bhearsum> i don't even understand the failure
16:45:36 <~bhearsum> it times out?
16:45:46 <RyanVM> yeah, I agree that it'd be preferable to not find out for the first time that they're busted when they hit m-r
16:45:54 <RyanVM> can't find python or something?
16:46:16 <~bhearsum> oh, i see
16:46:26 <~bhearsum> "with output_timeout 2400" doesn't mean it timed out...just that 2400s is the timeout
16:46:54 <~bhearsum> i suspect that the Dawn work isn't the cause of this, tbh
16:47:09 <~bhearsum> probably a lot has changed on mozilla-central since the last nightly
16:47:13 <RyanVM> agreed
16:47:28 <RyanVM> I was just tying it to Dawn because DE kinda of removed the need for those builds on Beta
16:47:31 <~bhearsum> what a useless error message.....
16:47:34 <~bhearsum> oooh
16:47:57 <~bhearsum> these logs are so hard to read...
16:48:58 <~bhearsum> i wonder if the docker image changed or something
16:49:55 <~bhearsum> it looks like they used to run:
16:49:56 <~bhearsum> python2.7 mach --log-no-times build -v'] with output_timeout 2400
16:50:04 <~bhearsum> and now they run:
16:50:04 <~bhearsum> /tools/buildbot/bin/python mach --log-no-times build -v'] with output_timeout 2400
16:52:23 <RyanVM> mtabara|mergeduty: OK with you if I file the BMO migration bug?
16:54:36 <~bhearsum> RyanVM: oh, i think i understand what you're saying now
16:54:46 <~bhearsum> it just clicked with me that we have buildbot+tc builds
16:54:57 <~bhearsum> so yes, let's just kill the buildbot ones since the tc ones work :)
16:55:43 <~bhearsum> probably something changed in mozharness in 55 that broke the buildbot ones
16:55:49 <~bhearsum> since they still work fine on mozilla-release
16:59:15 <RyanVM> sounds perfect, thanks
16:59:27 <RyanVM> the only concern I have is that I'm not sure where these builds get exposed for download
16:59:36 <RyanVM> and are the TC builds discoverable?
16:59:51 <~bhearsum> no clue
16:59:59 <~bhearsum> i assume there's an index for them
17:00:02 <@catlee> https://wiki.mozilla.org/Add-ons/Extension_Signing#Latest_Builds
17:00:11 <@catlee> that's the only reference I know to them
17:00:48 <mtabara|mergeduty> bhearsum: thanks for updating that
17:00:49 <~bhearsum> ok, so we'll have to update that to point at tc...
17:00:58 <~bhearsum> mtabara|mergeduty: np, are you going to take care of that bug?
17:01:50 <mtabara|mergeduty> sure, why not. I have a similar one for tcmigration so I can joint them together and dig to learn how to do that, should be fairly easy and straighforward AFAIK
17:01:52 <~bhearsum> we have indexes like https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-beta.latest.firefox/gecko.v2.mozilla-beta.latest.firefox.macosx64-add-on-devel
17:01:55 <~bhearsum> ok!

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Note that we don't have Windows TC addon-devel builds at the moment, so the buildbot jobs for those will need to remain unless someone wants to get those stood up on TC.

Comment 8

[Jordan Lund (:jlund)]

fwiw - beta repacks are failing in a similar manner to:

> RyanVM> mtabara|mergeduty: another fun one - Linux addon builds are busted with "07:34:46     INFO -  /usr/bin/env: /tools/buildbot/bin/python: No such file or directory"

suspect this is: https://hg.mozilla.org/mozilla-central/rev/481da97f0f10#l27.35

probably have to file a new ticket and address this issue since we can't turn beta repacks off ... unfortunately :)

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Any updates on this, Mihai?

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)
> Any updates on this, Mihai?

Started working on this and bug 1351326 yesterday. I am traveling today for the All-hands, will redirect to :kmoir or buildduty folks if I don't nail it by the time my flight is scheduled.

Leaving the NI to visit this later.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

@kmoir: I'm gonna be traveling today and won't be able to continue on this until late in the evening when I arrive. Seems like :sherrifs want this as soon as possible. Is this something that buildduty can look into later on today? Thanks for the help.

Comment 12

[Kim Moir [:kmoir] ET]

I didn't have time to look at it since we were migrating macosx builds today.  If you still need someone to look at it tomorrow, let me know and I'll ask the buildduty folks

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Kim Moir [:kmoir] from comment #12)
> I didn't have time to look at it since we were migrating macosx builds
> today.  If you still need someone to look at it tomorrow, let me know and
> I'll ask the buildduty folks

I'm having limited internet connection and I expect I won't make much progress until Friday/Saturday PST. Is this something #buildduty folks can look into? Thank you.

Comment 14

[Alin Selagea [:aselagea]]

I'll take a look at this.

Comment 15

[Alin Selagea [:aselagea]]

Attachment: bug_1372241.patch

Based on https://bugzilla.mozilla.org/show_bug.cgi?id=1372241#c7, I only disabled Linux and OS X builders since we don't have TC builders for Windows.

Comment 16

[Alin Selagea [:aselagea]]

Builder diff:

--- old 2017-06-22 08:04:02.281047920 -0700
+++ new 2017-06-22 08:19:42.710521022 -0700
-Linux x86-64 add-on-devel mozilla-beta build ScriptFactory
-OS X 10.7  add-on-devel mozilla-beta build ScriptFactory
-Linux x86-64 add-on-devel mozilla-release build ScriptFactory
-OS X 10.7  add-on-devel mozilla-release build ScriptFactory

Comment 17

[Kim Moir [:kmoir] ET]

If we are getting rid of them entirely, we could should clean up the code to remove the definitions for 
'linux64-add-on-devel', 'macosx64-add-on-devel' from buildbot-configs

Comment 18

[Ryan VanderMeulen [:RyanVM]]

In production.

Comment 19

[Alin Selagea [:aselagea]]

Attachment: remove_addon_definitions.patch

Remove definitions for Linux and OS X add-on builders.

Comment 20

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8880753 [details] [diff] [review]
remove_addon_definitions.patch

Looks good,tested and everything went well.

Comment 21

[Alin Selagea [:aselagea]]

Comment on attachment 8880753 [details] [diff] [review]
remove_addon_definitions.patch

https://hg.mozilla.org/build/buildbot-configs/rev/411b8e007202
https://hg.mozilla.org/build/buildbot-configs/rev/b2107cecdccd

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #21)
> Comment on attachment 8880753 [details] [diff] [review]
> remove_addon_definitions.patch
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/411b8e007202
> https://hg.mozilla.org/build/buildbot-configs/rev/b2107cecdccd

Thanks a lot for taking care of this!