staging-master doesn't start up masters properly on boot

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
9 years ago
5 years ago

People

(Reporter: nthomas, Assigned: lsblakk)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [automation][staging])

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

9 years ago
moz2-master and moz2-master need to use their Makefiles, rather be started from what's in /etc/default/buildbot. Should check the other entries to see if they are still current too.

Updated

9 years ago
Assignee: nobody → armenzg
Status: NEW → ASSIGNED

Comment 1

9 years ago
Created attachment 386741 [details] [diff] [review]
startup master through Makefiles
Attachment #386741 - Flags: review?(nthomas)
(Reporter)

Comment 2

9 years ago
Comment on attachment 386741 [details] [diff] [review]
startup master through Makefiles

This patch is against /etc/init.d/buildbot ? I think this will break unittest-master because we don't use a Makefile there. Are you proposing that we start using a Makefile there too ?

Need much more information than a bare patch.
Attachment #386741 - Flags: review?(nthomas) → review-

Comment 3

9 years ago
(In reply to comment #2)
> (From update of attachment 386741 [details] [diff] [review])
> This patch is against /etc/init.d/buildbot ? 
Yes it is
> I think this will break
> unittest-master because we don't use a Makefile there. Are you proposing that
> we start using a Makefile there too ?
>
I didn't think of it. I don't see benefits to use the Makefiles aside of making the changes to /etc/init.d/buildbot
> 
> Need much more information than a bare patch.
Do you have any suggestions on how to proceed? or shall I make the script work for both scenarios?
If we can avoid modifying the init script that would be great. It's used on all of our Linux machines - most of which don't use a Makefile to start Buildbot.

Comment 5

9 years ago
I am thinking out loud, tell me if it makes sense.
We don't make any changes to /etc/init.d/buildbot
We remove moz2-master and moz2-master2 entries from /etc/default/buildbot
We add a script on /etc/init.d to start with the Makefiles the mentioned 2 masters

Sounds good?
(In reply to comment #5)
> I am thinking out loud, tell me if it makes sense.
> We don't make any changes to /etc/init.d/buildbot
> We remove moz2-master and moz2-master2 entries from /etc/default/buildbot
> We add a script on /etc/init.d to start with the Makefiles the mentioned 2
> masters
> 
> Sounds good?

I think that's pretty reasonable.

Updated

9 years ago
Priority: -- → P2

Comment 7

9 years ago
Created attachment 387248 [details] [diff] [review]
stop starting moz2-master and moz2-master2 the typical way
Attachment #386741 - Attachment is obsolete: true
Attachment #387248 - Flags: review?(nthomas)

Comment 8

9 years ago
Created attachment 387253 [details] [diff] [review]
/etc/init.d/buildbot_staging

This script would be placed under /etc/init.d

BTW I have just copied and pasted the init scripts comments from the buildbot one without really knowing if it requires any changes

If it all looks good I will test all these in the morning.

How does it all look?
Attachment #387253 - Flags: review?(nthomas)
(Reporter)

Comment 9

9 years ago
Comment on attachment 387248 [details] [diff] [review]
stop starting moz2-master and moz2-master2 the typical way

We don't have any slaves for "Mozilla1.8.1 Automation Staging" anymore, so that whole section can go. ie we only need the 1.9.0 Unit test one. 

r+ with that change.
Attachment #387248 - Flags: review?(nthomas) → review+
(Reporter)

Comment 10

9 years ago
Comment on attachment 387253 [details] [diff] [review]
/etc/init.d/buildbot_staging

I haven't tested this either (!) but it seems highly likely to fail.

>### BEGIN INIT INFO
># Provides:          cvsd

Nit: s/cvsd/moz2-masters/

>case "$1" in
>  start)
>        su -s /bin/bash -c "make -C /builds/buildbot/moz2-master start"
>        su -s /bin/bash -c "make -C /builds/buildbot/moz2-master2 start"
>        exit $?
>        ;;

All of these su commands will need a " - ${USER}" added to the end of each line (without the quotes), and a definition of 
  USER=cltbld
earlier in the script.
Attachment #387253 - Flags: review?(nthomas) → review-

Comment 11

9 years ago
Created attachment 387462 [details] [diff] [review]
 /etc/init.d/buildbot_staging v1
Attachment #387253 - Attachment is obsolete: true
Attachment #387462 - Flags: review?(nthomas)

Comment 12

9 years ago
(In reply to comment #9)
> (From update of attachment 387248 [details] [diff] [review])
> We don't have any slaves for "Mozilla1.8.1 Automation Staging" anymore, so that
> whole section can go. ie we only need the 1.9.0 Unit test one. 
> 
> r+ with that change.
OK

BTW do we want buildbot_staging to be registered as a boot up process, right?

Comment 13

9 years ago
Tested on staging and it works.

The only scenario that seems to don't work is when I call the script without any arguments but not sure why:
[cltbld@staging-master buildbot]$ /etc/init.d/buildbot_staging
/etc/init.d/buildbot_staging: line 38: log_warning_msg: command not found

Updated

9 years ago
Priority: P2 → P3
(Reporter)

Updated

9 years ago
Attachment #387462 - Flags: review?(nthomas) → review+
(Reporter)

Comment 14

9 years ago
Comment on attachment 387462 [details] [diff] [review]
 /etc/init.d/buildbot_staging v1

>### END INIT INFO

Add '. /lib/lsb/init-functions' here to avoid the log_warning_msg problem. Looks fine otherwise, and sorry this review got lost.

IIRC there is an issue with the nagios check, which doesn't count processes started via the makefile but does the ones we do manually. It's defined at check_buildbot defined in /etc/nagios/nrpe.cfg. Do you know anything about that Ben ?

Updated

9 years ago
Assignee: armenzg → nobody
Priority: P3 → --

Updated

9 years ago
Component: Release Engineering → Release Engineering: Future
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Status: ASSIGNED → NEW
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3
Whiteboard: [automation][staging]
(Assignee)

Updated

8 years ago
Assignee: nobody → lsblakk
(Reporter)

Comment 16

8 years ago
staging-master died today after getting migrated from one ESX host to another so I took a look at what the current situation is. 

We were starting the old 1.9.0 unittest master via /etc/default/buildbot and chkconfig - I've disabled that (using chkconfig) since it's obsolete. We weren't starting the scheduler_master, so I added a @reboot rule in the crontab for cltbld. That prevents production jobs hanging on sendchange's to staging-master:9009.

Given we often leave the builder_master* not running these days it may not be worth starting them on boot. What do people think ?

Comment 17

8 years ago
FTR I leave running the builder_master pointing back to the clean repositories.

In my opinion I wouldn't bother restarting the builder_master as long as the scheduler_master is started and picking up the sendchanges.
I agree with not starting them on boot.  When I'm using staging master I like to start my borrowed builder_master, with my configs, from scratch.

Is there anything left to do here?
Since no one is piping in with remaining tasks for this I'm closing it.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.