1080135 - Create mechanism to create a buildbot environment on a local machine

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Attachment: wip - create buildbot environment

In order to help with mozharness try (bug 791924), I'm trying to setup a buildbot environment.
However, I'm having trouble since every time the slave connects to my master it produces and error. [1]
Thus, not being able to take jobs.

I have attached the script that I'm using to setup my environments.
I decided to join both the buildbot and buildslave virtual environments since we need both for build slaves [2]

If one of you could please help me out it would be great.
Here's my environment [3]

Thanks!

[1]
2014-10-08 14:49:11-0400 [Broker,2,127.0.0.1] Got slaveinfo from 'tst-linux64-ec2-001'
2014-10-08 14:49:11-0400 [Broker,2,127.0.0.1] bot attached
2014-10-08 14:49:13-0400 [Broker,2,127.0.0.1] BuildSlave.sendBuilderList (<BuildSlave 'tst-linux64-ec2-001'>) failed
2014-10-08 14:49:13-0400 [Broker,2,127.0.0.1] Unhandled Error
	Traceback from remote host -- Traceback unavailable
	
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.00s starting
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.00s got 0 request(s)
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.01s requests for my builders: 0
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.02s builders with requests: 0
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.02s found 0 available of 0 connected slaves
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.02s builders with slaves: 0
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.03s prioritized 0 builder(s): []
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.03s using up to first 0 builders:
2014-10-08 14:49:13-0400 [-] prioritizeBuilders: 0.03s found 0 important builder(s): []


[2] http://hg.mozilla.org/build/puppet/file/ad32888ce123/modules/buildslave/manifests/install/version.pp#l27

[3]
(venv)armenzg-thinkpad community hg:[default!] $ buildbot --version
Buildbot version: 0.8.2-hg-c0c54553a432-production-0.8-hg-131bf6f503c4+-default
Twisted version: 10.2.0
(venv)armenzg-thinkpad community hg:[default!] $ buildslave --version
Buildslave version: 0.8.4-pre-moz2
Twisted version: 10.2.0
(venv)armenzg-thinkpad community hg:[default!] $ pip freeze
Jinja2==2.7.3
MarkupSafe==0.23
Twisted==10.2.0
argparse==1.2.1
buildbot==0.8.2-hg-c0c54553a432-production-0.8
buildbot-slave==0.8.4-pre-moz2
cffi==0.8.6
cryptography==0.5.4
pyOpenSSL==0.14
pyasn1==0.1.7
pycparser==2.10
pycrypto==2.6.1
simplejson==2.1.3
six==1.7.3
wsgiref==0.1.2
zope.interface==4.1.1

Comment 1

[Armen [:armenzg]]

Attachment: wip - create buildbot environment

I have adjusted the WIP so you can also reproduce the issue.
I know it is not great bash programming but I'm not asking for review.

STR:
* patch braindump with attached patch
* call setup_buildbot_environment.sh
* call create_community_slaves_and_masters.sh
* in one window call "tail -F ~/.mozilla/releng/masters/test_master/twistd.log"
** wait for the master to be ready ("configuration update complete" in log)
* in another window:
** source ~/.mozilla/releng/venv/bin/activate
** buildslave start ~/.mozilla/releng/slaves/test_slave

Comment 2

[Armen [:armenzg]]

Could anyone in here please paste the following?
- the buildbot versions and python packages of a master
- the buildbot, buildslave versions and python packages of a Linux 64-bit slave

Thanks!

Comment 3

[Armen [:armenzg]]

For some odd reason, adding this patch to my test master finally allowed the slave to connect [1]
For now I will focus on fixing bug 791924 while I have a connected slave rather than debug this.
To be honest it makes no sense since just 2 minutes before I could still reproduce the issue.

[1]
--- a/mozilla-tests/tests_localconfig.py
+++ b/mozilla-tests/tests_localconfig.py
@@ -33,2 +33,6 @@ ACTIVE_B2G_BRANCHES = b2g_config.BRANCHE
 ACTIVE_MOBILE_BRANCHES = mobile_config.BRANCHES.keys()
+ACTIVE_BRANCHES = ['try']
+ACTIVE_THUNDERBIRD_BRANCHES = []
+ACTIVE_B2G_BRANCHES = []
+ACTIVE_MOBILE_BRANCHES = []
 if 'limit_fx_platforms' in master_config:
@@ -59,2 +63,4 @@ ACTIVE_PROJECTS = PROJECTS.keys()
 ACTIVE_B2G_PROJECTS = B2G_PROJECTS.keys()
+ACTIVE_PROJECTS = []
+ACTIVE_B2G_PROJECTS = []

Comment 4

[Armen [:armenzg]]

Attachment: wip - create buildbot environment

Hi Pete,
IIRC you have a lot of shell knowledge.
Would you mind having a look at this wip and let me know which things you would structure differently to make it look nice?

You will also notice that I have some variables that are defined in setup_buildbot_environment.sh that I then redefine in the other scripts.
Is there a way that I could have a common file to load those variables?

Comment 5

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8503252 [details] [diff] [review]
wip - create buildbot environment

Review of attachment 8503252 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Armen!

Thanks for working on this, I'm sure it can be very useful for people. I guess you have also seen:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master

and I have a feeling I saw something from Hal about setting up a local slave+master environment, but can't find the wiki page I thought I had seen on it.

Another thing maybe worth looking at is:
https://wiki.mozilla.org/ReleaseEngineering/Tupperware

I haven't played with Tupperware myself, but my understanding was it was about setting up a release engineering dev environment including buildbot (slave and build/test/scheduler master?)

I had a look through the code - there were several places (I only highlighted a few) where an env variable potentially contained spaces, but wasn't quoted. I'd recommend explicitly testing with a custom working directory containing spaces, to catch all critical places where quotes were omitted.

Regarding sharing config between the different scripts, this is an excellent idea, and I'd recommend putting shared config in a shared file (something like config.sh) and sourcing that from the other scripts.

As I can see it is still very much in a dev state (with TODOs and hard coded paths etc) I have not gone too deep, just highlighted some points here and there. Looking forward to future iterations! =)

Thanks for working on this important topic!

Pete

::: buildbot-related/community/create_community_slaves_and_masters.sh
@@ +3,5 @@
> +# Purpose:  This script does the following:
> +#             - create generic build, try and test masters
> +#             - create generic slaves for each of the previous masters
> +#
> +# Usage:   ./create_community_slaves_and_masters.sh [-w alternate_workdir]

I would put the usage in the -h option, so if you run with -h you see it

@@ +5,5 @@
> +#             - create generic slaves for each of the previous masters
> +#
> +# Usage:   ./create_community_slaves_and_masters.sh [-w alternate_workdir]
> +#
> +while getopts j:w: opts; do

j not needed - can be removed

@@ +28,5 @@
> +bdump="$repos_dir/braindump"
> +tools="$repos_dir/tools"
> +path=$PATH
> +
> +for dir in $slaves_dir $masters_dir

for dir in "$slaves_dir" "$masters_dir" # quoted, since $workdir can contain spaces (e.g. if called with create_community_slaves_and_masters.sh -w '/home/fred/my dir with spaces')

@@ +30,5 @@
> +path=$PATH
> +
> +for dir in $slaves_dir $masters_dir
> +do
> +    if [ ! -d $dir ]:

if [ ! -d "$dir" ] # quoted, and also drop the :

@@ +32,5 @@
> +for dir in $slaves_dir $masters_dir
> +do
> +    if [ ! -d $dir ]:
> +    then
> +        mkdir $dir

mkdir "$dir"

@@ +47,5 @@
> +#    fi
> +#done
> +
> +# Load up the masters' environment
> +/bin/bash -c ". $venv/bin/activate"

no need for /bin/bash - can just call e.g.

source "${venv}/bin/activate"

@@ +55,5 @@
> +
> +for master_name in "test_master"
> +#for master_name in "build_master" "try_master" "test_master"
> +do
> +    if [ ! -d ${masters_dir}/${master_name} ]

"${masters_dir}/${master_name}" # in case of spaces

::: buildbot-related/community/setup_buildbot_environment.sh
@@ +16,5 @@
> +done
> +
> +echo $workdir
> +echo $quiet
> +exit

exit ?

Comment 6

[Pete Moore [:pmoore][:pete]]

Regarding sourcing the shared config, I mean including a line like:

    source "$(dirname "${0}")/lib/config.sh"

near the top of the script (before you have stepped into a different directory); this example would look for lib/config.sh relative to the location of the script being called.

I typically use:
    cd "$(dirname "${0}")"

near the top of a script if the script needs to cd into relative paths later on, to make sure I am in the directory containing the script, in case it is invoked from a different directory.

Comment 7

[Armen [:armenzg]]

Attachment: create buildbot environment

For the record, I gave up on using paths with white spaces because the activation of the venv + pip do not go well with each other.

Comment 8

[Armen [:armenzg]]

Attachment: disable_jacuzzi.diff

This patch is needed to stop querying the jacuzzi allocation.

pmoore, I landed my patch to be able to iterate on my local setup. I will follow up with any of your comments.

Comment 9

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8514541 [details] [diff] [review]
create buildbot environment

Review of attachment 8514541 [details] [diff] [review]:
-----------------------------------------------------------------

::: community/create_community_slaves_and_masters.sh
@@ +71,5 @@
> +    "$script_dir/setup_buildbot_environment.sh" -w "$workdir"
> +fi
> +
> +# Load up virtual environment
> +/bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +83,5 @@
> +        $bco/setup-master.py -j $bdu/community/community-masters.json ${masters_dir}/${master_name} ${master_name} || exit
> +    fi
> +done
> +
> +declare -A slaves=( 

trailing space

::: community/list_builder_differences.sh
@@ +1,1 @@
> +#!/bin/bash

Currently any errors would be skipped over, it would be better to use
#!/bin/bash -eu
With -e it means if something errors, it will exit, rather than continue in a non-deterministic way.
With -u it means any variables referenced that don't exist, will cause an error, and adds a little safety.

@@ +43,5 @@
> +    echo "Please pass the path to the patch you want information about with -j path_to_patch."
> +    exit 1
> +fi
> +
> +if [ ! -f $patch_to_test ]

"$patch_to_test" (with quotes - in case of spaces in the name, like you have in line 41)

@@ +45,5 @@
> +fi
> +
> +if [ ! -f $patch_to_test ]
> +then
> +    echo "We can't reach the file you specified: $patch_to_test"

May be useful here to output the current directory, something like:
echo "We can't reach the file you specified: '$patch_to_test'. Current directory is '$(pwd)'."

@@ +48,5 @@
> +then
> +    echo "We can't reach the file you specified: $patch_to_test"
> +    echo "Please make sure that it is an absolute path"
> +    exit 1
> +fi

Maybe would also be nice to allow -j - to allow passing content via standard in (i.e. so you can curl a patch and pipe it to this script). Other unix utilities offer this feature (e.g. patch command itself).

@@ +52,5 @@
> +fi
> +
> +if [ -z "$workdir" ]
> +then
> +    workdir="$default_workdir"

where is variable default_workdir defined?

@@ +59,5 @@
> +# Load common variables
> +. "$script_dir/buildbot_config.sh" -w "$workdir"
> +
> +# Let's setup the buildbot environment and update it
> +$script_dir/setup_buildbot_environment.sh -w $workdir -q

"$script_dir/setup_buildbot_environment.sh" -w "$workdir" -q

@@ +72,5 @@
> +cd $bco
> +hg up -r default
> +
> +# Activate the virtual environment
> +/bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +98,5 @@
> +    fi
> +fi
> +
> +# Create new list of builders
> +rm -f $with_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $with_patch 

trailing space

@@ +104,5 @@
> +if [ -z $faster ] # We are *not* asking for a faster run 
> +then
> +    # Create current list of builders
> +    hg up -C
> +    rm -f $without_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $without_patch 

trailing space

@@ +108,5 @@
> +    rm -f $without_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $without_patch 
> +fi
> +
> +rm -f $differences_path
> +$bdu/buildbot-related/diff_allthethings.py $without_patch $with_patch > $differences_path 

trailing space

@@ +109,5 @@
> +fi
> +
> +rm -f $differences_path
> +$bdu/buildbot-related/diff_allthethings.py $without_patch $with_patch > $differences_path 
> +echo "Here's the file that contains your changes: $differences_path"

Maybe also nice to output contents of file to console too?

::: community/setup_buildbot_environment.sh
@@ +47,5 @@
> +        hg clone -q http://hg.mozilla.org/build/$repo_name || exit
> +    fi
> +    # Let's update to the right branch
> +    cd $1
> +    hg up -C -q

This won't clean any added (non-tracked) files, and won't reset to correct branch, if the branch is changed locally.
Rather than rely on hg purge which might not be enabled, you could use:
hg status | while read Q file; do [ "${Q}" == '?' ] && rm -rf "${file}"; done
This should be done before the updating working dir, in case any of the locally untracked files are added in a version that you update to. i.e. add it after line 50 or 51.

@@ +48,5 @@
> +    fi
> +    # Let's update to the right branch
> +    cd $1
> +    hg up -C -q
> +    hg pull -q -u

-u not needed

@@ +50,5 @@
> +    cd $1
> +    hg up -C -q
> +    hg pull -q -u
> +    hg up -q -r $2
> +    if [ ! -z quiet ]; then echo "Repo info: $1 updated to `hg id`"; fi

This will always get called, because 'quiet' is a non-empty string - you need $quiet (or better still, "$quiet")

@@ +57,5 @@
> +
> +if [ ! -d "$venv" ]
> +then
> +    virtualenv --no-site-packages "$venv" || exit
> +    /bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +70,5 @@
> +    echo "$repos_dir" >> "$venv"/lib/python2.7/site-packages/releng.pth
> +    echo "$tools/lib/python" >> "$venv"/lib/python2.7/site-packages/releng.pth
> +fi
> +
> +if [ ! -z quiet ]

$quiet

Comment 10

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8514541 [details] [diff] [review]
create buildbot environment

Review of attachment 8514541 [details] [diff] [review]:
-----------------------------------------------------------------

::: community/create_community_slaves_and_masters.sh
@@ +71,5 @@
> +    "$script_dir/setup_buildbot_environment.sh" -w "$workdir"
> +fi
> +
> +# Load up virtual environment
> +/bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +83,5 @@
> +        $bco/setup-master.py -j $bdu/community/community-masters.json ${masters_dir}/${master_name} ${master_name} || exit
> +    fi
> +done
> +
> +declare -A slaves=( 

trailing space

::: community/list_builder_differences.sh
@@ +1,1 @@
> +#!/bin/bash

Currently any errors would be skipped over, it would be better to use
#!/bin/bash -eu
With -e it means if something errors, it will exit, rather than continue in a non-deterministic way.
With -u it means any variables referenced that don't exist, will cause an error, and adds a little safety.

@@ +43,5 @@
> +    echo "Please pass the path to the patch you want information about with -j path_to_patch."
> +    exit 1
> +fi
> +
> +if [ ! -f $patch_to_test ]

"$patch_to_test" (with quotes - in case of spaces in the name, like you have in line 41)

@@ +45,5 @@
> +fi
> +
> +if [ ! -f $patch_to_test ]
> +then
> +    echo "We can't reach the file you specified: $patch_to_test"

May be useful here to output the current directory, something like:
echo "We can't reach the file you specified: '$patch_to_test'. Current directory is '$(pwd)'."

@@ +48,5 @@
> +then
> +    echo "We can't reach the file you specified: $patch_to_test"
> +    echo "Please make sure that it is an absolute path"
> +    exit 1
> +fi

Maybe would also be nice to allow -j - to allow passing content via standard in (i.e. so you can curl a patch and pipe it to this script). Other unix utilities offer this feature (e.g. patch command itself).

@@ +52,5 @@
> +fi
> +
> +if [ -z "$workdir" ]
> +then
> +    workdir="$default_workdir"

where is variable default_workdir defined?

@@ +59,5 @@
> +# Load common variables
> +. "$script_dir/buildbot_config.sh" -w "$workdir"
> +
> +# Let's setup the buildbot environment and update it
> +$script_dir/setup_buildbot_environment.sh -w $workdir -q

"$script_dir/setup_buildbot_environment.sh" -w "$workdir" -q

@@ +72,5 @@
> +cd $bco
> +hg up -r default
> +
> +# Activate the virtual environment
> +/bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +98,5 @@
> +    fi
> +fi
> +
> +# Create new list of builders
> +rm -f $with_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $with_patch 

trailing space

@@ +104,5 @@
> +if [ -z $faster ] # We are *not* asking for a faster run 
> +then
> +    # Create current list of builders
> +    hg up -C
> +    rm -f $without_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $without_patch 

trailing space

@@ +108,5 @@
> +    rm -f $without_patch && cd $bco && $bdu/buildbot-related/dump_allthethings.sh $without_patch 
> +fi
> +
> +rm -f $differences_path
> +$bdu/buildbot-related/diff_allthethings.py $without_patch $with_patch > $differences_path 

trailing space

@@ +109,5 @@
> +fi
> +
> +rm -f $differences_path
> +$bdu/buildbot-related/diff_allthethings.py $without_patch $with_patch > $differences_path 
> +echo "Here's the file that contains your changes: $differences_path"

Maybe also nice to output contents of file to console too?

::: community/setup_buildbot_environment.sh
@@ +47,5 @@
> +        hg clone -q http://hg.mozilla.org/build/$repo_name || exit
> +    fi
> +    # Let's update to the right branch
> +    cd $1
> +    hg up -C -q

This won't clean any added (non-tracked) files, and won't reset to correct branch, if the branch is changed locally.
Rather than rely on hg purge which might not be enabled, you could use:
hg status | while read Q file; do [ "${Q}" == '?' ] && rm -rf "${file}"; done
This should be done before the updating working dir, in case any of the locally untracked files are added in a version that you update to. i.e. add it after line 50 or 51.

@@ +48,5 @@
> +    fi
> +    # Let's update to the right branch
> +    cd $1
> +    hg up -C -q
> +    hg pull -q -u

-u not needed

@@ +50,5 @@
> +    cd $1
> +    hg up -C -q
> +    hg pull -q -u
> +    hg up -q -r $2
> +    if [ ! -z quiet ]; then echo "Repo info: $1 updated to `hg id`"; fi

This will always get called, because 'quiet' is a non-empty string - you need $quiet (or better still, "$quiet")

@@ +57,5 @@
> +
> +if [ ! -d "$venv" ]
> +then
> +    virtualenv --no-site-packages "$venv" || exit
> +    /bin/bash -c ". $venv/bin/activate"

I think you can just run source "$venv/bin/activate" here (no need to invoke a separate bash process)

@@ +70,5 @@
> +    echo "$repos_dir" >> "$venv"/lib/python2.7/site-packages/releng.pth
> +    echo "$tools/lib/python" >> "$venv"/lib/python2.7/site-packages/releng.pth
> +fi
> +
> +if [ ! -z quiet ]

$quiet

Comment 11

[Pete Moore [:pmoore][:pete]]

Whoops, seemed to have posted that twice.

Comment 12

[Armen [:armenzg]]

Comment on attachment 8514541 [details] [diff] [review]
create buildbot environment

Review of attachment 8514541 [details] [diff] [review]:
-----------------------------------------------------------------

::: community/list_builder_differences.sh
@@ +48,5 @@
> +then
> +    echo "We can't reach the file you specified: $patch_to_test"
> +    echo "Please make sure that it is an absolute path"
> +    exit 1
> +fi

How can you accomplish this?

@@ +52,5 @@
> +fi
> +
> +if [ -z "$workdir" ]
> +then
> +    workdir="$default_workdir"

It used to be defined. I removed the block since calling buildbot_config.sh [1] works with $workdir being empty

[1]
. "$script_dir/buildbot_config.sh" -w "$workdir"

Comment 13

[Armen [:armenzg]]

Attachment: address issues

Comment 14

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8516148 [details] [diff] [review]
address issues

Review of attachment 8516148 [details] [diff] [review]:
-----------------------------------------------------------------

::: community/list_builder_differences.sh
@@ +105,5 @@
>  rm -f $differences_path
> +$bdu/buildbot-related/diff_allthethings.py $without_patch $with_patch > $differences_path
> +echo "\n"
> +cat $differences_path
> +echo "\nHere's the file that contains your changes: $differences_path"

echo -e "\\nHere's ...

::: community/setup_buildbot_environment.sh
@@ +52,5 @@
> +    cd $repo_path
> +    hg up -C $verbosity
> +    hg pull $verbosity
> +    hg up -r $branch $verbosity
> +    hg status -u -0 | xargs rm #Remove untracked files

xargs -0 rm

Comment 15

[Armen [:armenzg]]

I will be closing this once I write a blog post about it.

Any suggestion as to where to add some documentation about this on the Releng wiki pages?

Comment 16

[Armen [:armenzg]]

Done:
http://armenzg.blogspot.ca/2014/11/setting-buildbot-up-la-releng-create.html