1215294 - end_to_end_reconfig.sh should update master/master_config.json

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kim Moir [:kmoir] ET]

This is is needed when we add/remove platforms from tools/buildfarm/maintenance/production_masters.json

Right now it is a manual step, see 

https://bugzilla.mozilla.org/show_bug.cgi?id=1205785#c39

Comment 1

[Kim Moir [:kmoir] ET]

From 
https://bugzilla.mozilla.org/show_bug.cgi?id=1205785#c39

Looking here
http://hg.mozilla.org/build/tools/file/tip/buildfarm/maintenance/end_to_end_reconfig.sh#l481

production_masters_url='http://hg.mozilla.org/build/tools/raw-file/tip/buildfarm/maintenance/production-masters.json' is already set

so the steps you need to run are something like this which could be added under here
http://hg.mozilla.org/build/tools/file/tip/buildfarm/maintenance/end_to_end_reconfig.sh#l487

cd /builds/buildbot/tests1-linux64/ (this varies by master, will have to generalize it)
python (path to )buildbot-configs/update-master-json.py ${production_masters_url} master/master_config.json

and then allow the regular checkconfig and reconfig steps to occur

This is one approach, let me know if you have questions

As a background, basically the purpose of a reconfig is to
checkout new code for the three releng repositories that reside on the master (buildbot-configs, buildbotcustom, tools) (update command)
check the configuration is okay (checkconfig commmand)
reload the new configuration on the masters (reconfig command)

The script also runs the show_revisions command so the version of code we have on the masters is reflected in the logs

Comment 2

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Attached you can find the patch

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 8697966 [details] [diff] [review]
bug1215294_end_to_end_reconfig.sh.patch

Did you test this?  I'm wondering if it works given that end_to_end_reconfig.sh runs on your local machine and 

python ${buildbotdir}/buildbot-configs/update-master-json.py ${production_masters_url} master/master_config.json

is designed to run on the remote machine in the buildbotdir directory

Comment 4

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Kim Moir [:kmoir] from comment #3)
> Comment on attachment 8697966 [details] [diff] [review]
> bug1215294_end_to_end_reconfig.sh.patch
> 
> Did you test this?  I'm wondering if it works given that
> end_to_end_reconfig.sh runs on your local machine and 
> 
> python ${buildbotdir}/buildbot-configs/update-master-json.py
> ${production_masters_url} master/master_config.json
> 
> is designed to run on the remote machine in the buildbotdir directory

I added the 'if' statement in order to run the python command only if the directory for buidbot exists.

Comment 5

[Kim Moir [:kmoir] ET]

Right, the if statement is good.  My point is that this reconfig script runs things from your local machine (or cruncher in the future) and this part

python ${buildbotdir}/buildbot-configs/update-master-json.py ${production_masters_url} master/master_config.json

assumes that it will run on the actual master itself.  If you look at this line later on
$(pwd)/manage_masters.py' -f '${production_masters_url}' -j16 -R scheduler -R build -R try -R tests show_revisions update"

it is running commands on the masters that are invoked via manage_masters.py

Comment 6

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Updated the patch

Comment 7

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: end_to_end_reconfig.sh.output

Attached the output command for the patch 8698900 . The script has been run with -p option. 

Kim can you provide us details how to run the script to check the new action (reconfig_buildbot_master) that I created

Comment 8

[Kim Moir [:kmoir] ET]

I would suggest that you test this on a staging master, not a real master (something on dev-master2.bb.releng.use1.mozilla.com).  As an aside, in the patch you call the action reconfig_buildbot_master.  You're not really running the reconfig in this step but rather update a file.

Comment 9

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Updated the patch by renaming the action.

I run the script ./end_to_end_reconfig.sh on dev-master2.bb.releng.use1.mozilla.com but I'm receiving the following error ERROR: Python version 2.7 not found - please make sure python 2.7 is in your PATH .I added python2.7 in the PATH but is not working

Comment 10

[Kim Moir [:kmoir] ET]

perhaps this has some hints on how to export your python env

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques

As an aside, asking for a review of code means that you have it working in staging and you think that it works to address the requirements that the bug requestor has asked you to implement.  So I'm happy to help if you are stuck, but please don't ask for review unless you have a working solution.

Comment 11

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Updated the patch

Comment 12

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: manage_masters.py.output

Attached you can find the output from the patch that I made. 
I run the manage_masters.py script by using the following action update_master_config and using my local master_config.json.test in order to do not test on production masters, also the action.py has been changed to use also the master_config file in stead of production masters

Comment 13

[Kim Moir [:kmoir] ET]

Can your run end_to_end_reconfig.sh and only have your dev-master in production-master.json.  The test output you provide isn't testing the shell script.

Comment 14

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_production-masters.json.patch

Attached is the patch in order to update production-masters.json by adding my master to test the new futures added in end_to_end_reconfig.sh

Comment 15

[Kim Moir [:kmoir] ET]

Comment on attachment 8700611 [details] [diff] [review]
bug1215294_production-masters.json.patch

We don't add staging servers to production-masters.json.  You can just test in staging with a copy of production-masters.json that only includes the information for your staging master.

Comment 16

[Vlad Ciobancai [:vladC] [:vciobancai]]

I created a production master json file on dev-master2 ( path /builds/buildbot/vlad.ciobancai/yosemite_thunderbird/master/production-masters-test.json ) in order to test the end to end script. 

I run the script with the following options -n -b -f but every time I'm receiving the following error 

Traceback (most recent call last):
  File "./manage_masters.py", line 6, in <module>
    from fabric.api import env
ImportError: No module named fabric.api

Comment 17

[Kim Moir [:kmoir] ET]

Is fabric installed locally on your machine?

pip freeze | grep -i fabric
Fabric==1.10.2


(version may be different)

Comment 18

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Kim Moir [:kmoir] from comment #17)
> Is fabric installed locally on your machine?
> 
> pip freeze | grep -i fabric
> Fabric==1.10.2
> 
> 
> (version may be different)

I run the script on dev-master2 not on my computer

Comment 19

[Kim Moir [:kmoir] ET]

okay,  so instead of installing the fabric libraries could you update your local copy of production masters.json on your machine and run the modified end_to_end_reconfig script that way?

Comment 20

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: end_to_end_reconfig.output

Attached is the output for the following command ./end_to_end_reconfig.sh -b -n -m -f

Comment 21

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Attached is the patch. In order to test I used a custom production-master.json where I defined my buildbot from dev-master2

Comment 22

[Kim Moir [:kmoir] ET]

Comment on attachment 8701494 [details] [diff] [review]
bug1215294_end_to_end_reconfig.sh.patch

I think you should move the order of the call to the update_master_config to be after the call to update on the masters.  Otherwise, you'll be running the code to update the file before you have the latest revision on the masters.

Comment 23

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Updated the bash script by using the suggestions received from :kmoir

Comment 24

[Vlad Ciobancai [:vladC] [:vciobancai]]

If I test again the script locally I'm receiving the following errros :
[dev-master2.bb.releng.use1.mozilla.com] out: (python buildbot-configs/update-master-json.py tools/buildfarm/maintenance/production-masters.json master/master_config.json)
[dev-master2.bb.releng.use1.mozilla.com] out: master not found in tools/buildfarm/maintenance/production-masters.json


In the past there was a buildbot master for tests in AWS. Can I use that buildbot master to test the entire process of the script ?

Comment 25

[Kim Moir [:kmoir] ET]

yes, I think you are testing that now :-)

Comment 26

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1215294_end_to_end_reconfig.sh.patch

Attached you can find the patch updated

Comment 27

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: end_to_end_reconfig.output

Attached you can find the output. In order to test I disabled the following actions: tests, show_revisions, update

Comment 28

[Kim Moir [:kmoir] ET]

Did you test running the script from a remote machine
i.e. run reconfig script from a machine other than bm69 to update bm69?

Comment 29

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Kim Moir [:kmoir] from comment #28)
> Did you test running the script from a remote machine
> i.e. run reconfig script from a machine other than bm69 to update bm69?

I run the script from my local computer. I updated the end to end script by specifying a custom production-masters.json where I defined only bm69 in stead of using the production file. I updated the end to end script by removing the following actions "tests, show_revisions, update" in order to test the following action update_master_config.

If you want me to run from different machine, please let me know

Comment 30

[Vlad Ciobancai [:vladC] [:vciobancai]]

in production