1299421 - Switch to running restart_masters.sh on the buildduty-tools machine

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Alin Selagea [:aselagea]]

At the moment, the script is run by Coop from his personal account on dev-master2. It would be useful to move that to buildduty-tools so more releng people can run it (preferably during weekend).

Comment 1

[Chris Cooper [:coop] (he/him)]

The venv setup is as follows:

(restart_masters)[[e-mail] restart_masters]$ pip freeze
ecdsa==0.13
furl==0.4.8
keyring==7.3
orderedmultidict==0.7.5
paramiko==1.16.0
pycrypto==2.6.1
requests==2.9.0
simplejson==3.8.1
six==1.10.0

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: restart_masters.sh

Here's the shell script that I used to kick off the script. If supplying credentials from the command-line, we'll need to figure out how to log the output since piping logs will eat the credential prompts.

We should verify the flows required by this script actually work on the buildduty machine.

Comment 3

[Chris Cooper [:coop] (he/him)]

The script also relies on being able to hit the web interface for each master to be able to initiate a clean shutdown:

https://hg.mozilla.org/build/tools/file/8d577a74ae0f/buildfarm/maintenance/restart_masters.py#l278

Comment 4

[Chris Cooper [:coop] (he/him)]

We should also get python 2.7.12 installed on the buildduty-tools box as the default and re-create the venv(s). This will save us lots of insecure connection warnings in the logs.

Comment 5

[Chris Cooper [:coop] (he/him)]

It looks like the relevant netflows are in place now, i.e. I can ssh to buildbot masters in use1, usw2, and scl3, and can also hit the web interfaces for each.

We still need to provide credentials on the command-line until we figure out a way to automate this, but I think that's out of scope for this bug. We're just moving the script to run on a new host.

Comment 6

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Done switching restart_masters.sh to run on the buildduty-tools machine.
Some changes were made to the initial script in order to work properly:
-added a patch to import some libraries and log the work
-comment some configs that are added by puppet on all buildduty users (Host *mozilla.com *mozilla.org BatchMode yes) which prevented the script to have access to the masters under buildduty account.

The script is /home/buildduty/restart_masters.sh and the venv is /home/buildduty/restart_masters.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

When I ran the script last weekend I changed it a little so that it could log into papertrail while still prompting for credentials:

 def get_credentials_from_user():
-    credentials["ldap_username"] = raw_input("Enter LDAP username: ")
+    credentials["ldap_username"] = getpass.getpass("Enter LDAP username: ")

getpass doesn't echo to the prompt so typos are hidden, but it's smart enough to allow redirection like '2>&1 | logger -t restart_masters.sh' on the restart_masters.py call. [To be completely accurate I also used logging.basicConfig(stream=sys.stdout, ... and didn't redirect stderr, but I don't think that's necessary]

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Thanks Nick for the improvement,I added this part with getpass.getpass for username in the patch that is added to the script after cloning the repository.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #6)
> Done switching restart_masters.sh to run on the buildduty-tools machine.
> Some changes were made to the initial script in order to work properly:
> -added a patch to import some libraries and log the work
> -comment some configs that are added by puppet on all buildduty users (Host
> *mozilla.com *mozilla.org BatchMode yes) which prevented the script to have
> access to the masters under buildduty account.
> 
> The script is /home/buildduty/restart_masters.sh and the venv is
> /home/buildduty/restart_masters.

You set needinfo here, but there's no corresponding question. Did you want me to look at something in particular?

Comment 10

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Yes,I wanted a review on the new script that I have created in buildduty-tools (restart_masters.sh) and the enviornment, maybe is something that you want to change.
Sorry Coop for the need_info, should have requested a review or a feedback, just wanted someone to check before closing the bug.

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #10)
> Sorry Coop for the need_info, should have requested a review or a feedback,
> just wanted someone to check before closing the bug.

Is there a delta from the script I already attached? I want to treat these scripts as code (because they are) and do a proper review process if we're changing them.

It sounds like Nick has already validated the env, so let's get the script patches posted, reviewed, and close this out.

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: patch_restart_masters_bug1299421.pp

The patch which is attached to restart_masters.py just on this server buildduty-tools.

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: restart_masters_bug1299421.sh

The actual script, restart_masters.sh.

Comment 14

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: diff_restart_masters_bug1299421.pp

The differences between the old restart_masters.sh from dev-master2.bb.releng.use1.mozilla.com and the one from buildduty-tools.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8796527 [details]
patch_restart_masters_bug1299421.pp

> diff --git a/buildfarm/maintenance/restart_masters.py b/buildfarm/maintenance/restart_masters.py
>   --- a/buildfarm/maintenance/restart_masters.py
>   +++ b/buildfarm/maintenance/restart_masters.py
>   @@ -18,6 +18,10 @@
>    #  * on each interation, check for masters that have finished, pick a new master
>    #    from that same bucket, and begin the restart process for that master
>  
>   +import sys
>   +sys.path.append('/home/buildduty/restart_masters/slaveapi')
>  +import datetime
>  +

The change in the indent level implies this is an interdiff, or a copy and paste that went wrong. What's the story ?

>  @@ -33,7 +37,9 @@ from slaveapi.clients import ssh
>   from datetime import datetime
> 
>   import logging
>  +logging.basicConfig(filename='/home/buildduty/restart_masters/tools/buildfarm/maintenance/all_logs.txt',level=logging.DEBUG)
>   log = logging.getLogger(__name__)
>  +log.info("Testing logger")

Could you explain what this is for ? Switching from papertrail to local log ? If so, what's the motivation for that ? Perhaps it's only for testing, in which case I'd love to see the patch which would land in tools/.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

No need to add an 'import datetime' when 'from datetime import datetime' is lower down.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8796528 [details]
restart_masters_bug1299421.sh

Could use with some explanations around this one too. eg
* is the 'patch' call for production use or testing ? If testing I'd rather review the production version
* did you consider setting up ~/.ssh/known_hosts to avoid the sed on .ssh/config ? https://hg.mozilla.org/build/puppet/file/default/modules/ssh/templates/known_hosts.erb is the place to do that
* the change to the clone of slaveapi has the same effect, but in two lines

In the bigger picture, what will call this script ? A human with a calendar reminder so they can put in credentials ?

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

> >   +import sys
> >   +sys.path.append('/home/buildduty/restart_masters/slaveapi')
> >  +import datetime
> >  +
> 
> The change in the indent level implies this is an interdiff, or a copy and
> paste that went wrong. What's the story ?

Indeed these are not necessary so I removed them,that step is done in the bash script: "export PYTHONPATH=${WORKDIR}/slaveapi".


> >   import logging
> >  +logging.basicConfig(filename='/home/buildduty/restart_masters/tools/buildfarm/maintenance/all_logs.txt',level=logging.DEBUG)
> >   log = logging.getLogger(__name__)
> >  +log.info("Testing logger")
> 
> Could you explain what this is for ? Switching from papertrail to local log
> ? If so, what's the motivation for that ? Perhaps it's only for testing, in
> which case I'd love to see the patch which would land in tools/.

I wanted to create a file on this server in which we could have all the logs for the masters that we restarted ,if is not necessary I can remove this part,in both cases these logs will appear in Papertrail.

Comment 19

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: restart_masters_bug1299421.sh

I added here the production version of the restart_masters.sh
The reason for sed on .ssh/config is not for resolving known_hosts but to comment some configs that are added by puppet on all buildduty users (Host *mozilla.com *mozilla.org BatchMode yes) which prevent the script to have access to the masters under buildduty account.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8798430 [details]
restart_masters_bug1299421.sh

>        patch -p1 < /home/buildduty/restart_masters/patch_restart_masters.patch

Please fix up the tools patch so this isn't necessary.

>sed -i '4,8 s/^/#/' /home/buildduty/.ssh/config

While this works it really isn't the best way to do things - it works around around security we've set up. Please create a patch which adds all the non-build masters to https://hg.mozilla.org/build/puppet/file/default/modules/ssh/templates/known_hosts.erb. You can get the extra host signatures by doing this as buildduty@buildduty-tools:

masters=`python -c 'import sys, json; j=json.load(open(sys.argv[1])); print " ".join([m["hostname"] for m in j if m["enabled"] and m["role"] != "build"])' \
 ~/restart_masters/tools/buildfarm/maintenance/production-masters.json`
ssh-keyscan $masters 2> /dev/null

Comment 21

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Nick thank you for these,I pushed the changes in Tool repositoy so that the patch is no longer needed. 

About the known_hosts.erb,I don't think these will help us very much as at some point the script restart_masters.py will need to have access to the the masters using the password; for security measure this script is run manually because it requires sensitive credentials that shouldn't be stored on disk, so in order to run we will need to use the passwords.
Currently Puppet repository add (BatchMode yes )so that all buildduty users will not be able to  connect with password(check the links bellow):

https://dxr.mozilla.org/build-central/source/puppet/modules/users/manifests/buildduty/setup.pp#30
https://dxr.mozilla.org/build-central/source/puppet/modules/users/templates/builder-ssh-config.erb
http://hg/build/puppet/file/tip/modules/users/templates/buildduty-ssh-config.erb

This is why I use this part in restart_masters.sh:
>sed -i '4,8 s/^/#/' /home/buildduty/.ssh/config

Maybe it could be done in antoher way but I don't see how adding the non-build masters in the known_hosts.erb will help when connecting with the password.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

You're right, I hadn't checked BatchMode to find it disables password auth. I still don't think running sed over the config is right option though. 

BatchMode was first added so a mis-configured build machine (missing ssh keys!) would error out when uploading files, rather than hang for hours then error out. So it may be not appropriate for buildduty@buildduty-tools machine, and puppet could be changed to remove it. Alternatively we could provision the ssh keys for the buildduty account, and there's one less password to enter to run the script. It really depends what the long term plans are for this host, ie if we'll need ssh keys for something else, and what the security model is. I suggest taking that up with coop and/or kmoir. If we do add the keys then the known_hosts would need updating to avoid 'Host key verification failed.' errors.

Comment 23

[Chris Cooper [:coop] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #22)
> BatchMode was first added so a mis-configured build machine (missing ssh
> keys!) would error out when uploading files, rather than hang for hours then
> error out. So it may be not appropriate for buildduty@buildduty-tools
> machine, and puppet could be changed to remove it. Alternatively we could
> provision the ssh keys for the buildduty account, and there's one less
> password to enter to run the script. It really depends what the long term
> plans are for this host, ie if we'll need ssh keys for something else, and
> what the security model is. I suggest taking that up with coop and/or kmoir.
> If we do add the keys then the known_hosts would need updating to avoid
> 'Host key verification failed.' errors.

Yes, I think we can add the keys to this machine. 

We limit access to this machine for a reason, but it should be able to do the things we need it to do.

Comment 24

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: restart_masters_bug1299421.sh

The new changes for restart_master.sh ,where adding_non_build_masters_to_known_hosts.sh is :

#!/bin/bash
masters=`python -c 'import sys, json; j=json.load(open(sys.argv[1])); print " ".join([m["hostname"] for m in j if m["enabled"] and m["role"] != "build"])' \
 ~/restart_masters/tools/buildfarm/maintenance/production-masters.json`
ssh-keyscan $masters 2> /dev/null >> /home/buildduty/.ssh/known_hosts

Nick for the sed part (to be able to login with password as buildduty user) do you have some hints how we could change it.Also we can keep that part as we only use this script for buildduty-tools machine and we don't need it elsewhere.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

The script can't modify .ssh/known_hosts or .ssh/config - puppet will undo that within 30 minutes, and the master restart script runs for several hours. So we need the the .ssh directory set up before the script runs.

Here are two alternatives:

1, We keep using a password to access to cltbld@buildbot-masterNNN
Steps:
1a, modify modules/users/templates/buildduty-ssh-config.erb in the puppet repo [1], to remove the BatchMode restriction. NB it's a symlink at the moment, so turn it into a real file before removing the two lines related to BatchMode
1b, update known_hosts so it contains all the masters, otherwise we'll hit 'Host key verification failed.' errors. I've mentioned this in earlier comments, the file that needs changing is modules/ssh/templates/known_hosts.erb. Some masters are present already, but the test and try masters would need to be added. You could run the python I suggested earlier and figure out which entries need adding to known_hosts.erb

Steps 1a and 1b would need patch review.

2, Create a new buildduty ssh key, and use it connect to the masters. This is quite a bit more complicated, but basically uses the same model as releaserunner (bm81 -> build & scheduler masters). I'm not sure it's worth doing to avoid typing one or two passwords, and would recommend option 1.

[1] https://hg.mozilla.org/build/puppet/

Comment 26

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1299421_puppet.patch

The patch for puppet to add the non-build masters in known_hosts and remove BatchMode restriction for buildduty users.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8804628 [details] [diff] [review]
bug1299421_puppet.patch

Review of attachment 8804628 [details] [diff] [review]:
-----------------------------------------------------------------

This very close, but the buildduty-ssh-config.erb doesn't look quite right. I was expecting to see something like:

diff --git a/modules/users/templates/buildduty-ssh-config.erb b/modules/users/templates/buildduty-ssh-config.erb
old mode 120000
new mode 100644
--- a/modules/users/templates/buildduty-ssh-config.erb
+++ b/modules/users/templates/buildduty-ssh-config.erb
@@ -1,1 +1,6 @@
-builder-ssh-config.erb
\ No newline at end of file
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+Host github.com
+    IdentityFile ~/.ssh/ffxbld_rsa

That's from doing 'rm buildduty-ssh-config.erb && cp builder-ssh-config.erb buildduty-ssh-config.erb', and removing the lines for BatchMode. Are you using Windows ? I think it doesn't respect symlinks on checkout, and dereferences them (aka makes a copy).

Comment 28

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1299421_puppet_v2.patch

Version 2 patch,both have the same result when applied,also same diff is shown,tested on Linux.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8805513 [details] [diff] [review]
bug1299421_puppet_v2.patch

Review of attachment 8805513 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for updating the patch, looks good.

Comment 30

[Alin Selagea [:aselagea]]

Comment on attachment 8805513 [details] [diff] [review]
bug1299421_puppet_v2.patch

>diff --git a/modules/ssh/templates/known_hosts.erb b/modules/ssh/templates/known_hosts.erb
>--- a/modules/ssh/templates/known_hosts.erb
>+++ b/modules/ssh/templates/known_hosts.erb
>@@ -8,7 +8,21 @@
>   PLEASE KEEP THE CONTENTS OF THIS FILE SORTED!
>         In vim: :/^-%/,$!sort
> -%>
>+buildbot-master01.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApHAJ9DJ5feSpajW6PoMav1WcR8xClcdVFRQPsfZknPJrIsu8yGoZe2gY7qnmHUUdk3W3UmVhZxa4zOs2AjhA+LS62qRtnyAkm/Re/FZwvfzd4/+4Oeul3cJ4ZRNeP4NWrDJVmU0Ucv+f7a7qg+7DALnzlcSsDRwoElmBSaEalbrdq7HxduVrRpultNdXLMLV+w5kZn5LEmapVcu3tJY99/NkTag763u7Gi1+03qCqs0h89Zd6+r8dMEmrCccuKNHDwhqQieceAXZ2rSiWvf21lARoJ+ufJsFVxIzmdUlkzFXLCNiD8GCRPWk/8TzwHI4N7ctQJAjknqFyrOPsNyflQ==
>+buildbot-master02.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyWfH37siTUjUwbDt/rkx7KU1DdWZLqvD+yUbOF7AX33wfT27Mj2Tv57Aq6UFsmtA5yhTy10BWXcTWMe9I4c0oQbI9C1FCYhLOOlLzxbIsIEJQjff4Ohb0xBYeLUoQU8U6qLvAXHULv1ztpTK8xILdQnZiy77tf82v5bk38uW6rS1q+f9DLvjw6bEiubo3bbfk2UnQTyTIAnYVuOOeW5kd8XttX49fQWJfIJ/O5FgquntcItOJtp7Vs7Ul080/i50hYDpflRPQ8oWX6Fr3VxziObT02ilMkYlzQaOK2Sved3PjulbaG2FRgfQIHnjAsHj0gu0h58EZJ4tL4BFtCmArw==
>+buildbot-master03.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4j5XwRV/G55HodeYpIn+8VDE+zEEjtF8eiavkglIvPV6ZY0hyZ85LM1fyZxdeejcZJkBM2E0tbDxp4su0eAB0DTe10+NY92RNsVV4D+rFo7PHm8lfdmDEmRDAsefztxw3F06x+GxLRfgsjFJdqnBx9xPyW7PKEv8bABkCDz56bZwuWyhJeAA9TcAbU1XjrlddpUfS/v4bvGzdlDUpNaqeWy3cRC9VfdidLcClYPwTVJrVJByXnp0QYYgSty+70/44Yxhqa+rY8y9Ot4A1u+FetTIx5MhWqioYF5g+cS4pCzhtLEwlSae2GxR938m+uJbOgOG5yojvtJ7sy3YYxcgmQ==
>+buildbot-master04.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3rJNktPwkjd5J3Dqs1vZQ16e+BHV5knOnJBlYg5cvwhGt3UAOZjTChTuOYU+D9/PF+tgbGwociisQotRdsWNhmysq7v/0Q5AwGS1j9T+HIwa1e/ClLzpDHts0Q3gXglEuR6cWm91YfzhEKus4gXJ90P99qiLGlGgPBKa2bIN1VqDYe621QpPyqax8fNWCwoPDzcBAtqSamdiw3ThdVMTLLuGYpYtVAu1YibO68oPj0X/ZbuhHXbUdKWH3xKlo9/Yd4OyMq+D/dWOmND5ls7wIyoikChirgKIzA4ON1K6ZLsc/AHE3JYvuyCp5D7MedEo8o4ViJSRJc2FwYRFirDHUw==
>+buildbot-master05.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxCjTEFfDtMmyzxVIguCN5S5nG6KN2j6GdgUTqqGjysMKHmX3h4TiD2iigEK9h10rlUNAY54szz2JtKcz81ODrbnSc5r0I4ua0emVLk0etD2fAXKNWIdQcrHoBtEIQfHQsgFTiaCE1x8BEWVVHv5iPVZwj2hxF4LO5mTpNN6VsQTCdpakfXpikYUXiFa0e2KJa0Dc7zgEWn7wtQxNVWGmg48/XLh91yyRo+kHNRyEIAvMYnMD2mXkWAmYS+xy2niAmmUHfI56/q/ctDvdV+4RKC8Kwbg765dhBLpjnEc97HhmGdTJFnazJ17JhRjLN7FhyBBn/kyK95SxIoBWgT8E7Q==
>+buildbot-master06.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4vVPMCwHoXgdaguN5R3cT5+xtB+0f+iMVNdGElkkifbx8rlHjFuHj3VHqdwu5ttST01bFMe8IOVzD2uHXvajdsa4/+tCccqV9N+sPzxKwRb4aQIdi7hov4mpDCJ0JTddGhlY4RyYw1UBR8d/owH1f07oj+JYrFeGo4iftqCfhoSusB851espry16Jh5cs5XZpGvikMmHMamQJWQeQSfNnpBz5PB3JAdODDx9gcxJqc2TmLy1sO+qb+ESFNnQNw+qJvvgC0vVcb7gELwSqBDK1MsYK9M6NID8ATwzttOmV0SozHBJUH1Y4wgIAkuagcmZOfRjKrMRLv0PbkROhN6tnw==
>+buildbot-master07.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq00hxNTQQXiXxs0XnK/U1fxZOGdZtGnKExvB63lIPdgcwiK/AE01Rw+wZdI3ggzUJgkRzAaSeTuT4NuUjIAOooCtnRe5xYMiW607VZQMMOuGr4JGSqrb7/BAS1kpKOFgVoywTv4eSjMV24+Ms95uZIFrgYZMDGul5z4xszyozcC8dI0jrWi11lCYdIcRhczNah1WFC8rcYquBed0KwLQd+44pVPAam2hlwhXR+UVcXfdsV7UQM3Rb/L5qJcpdZ+1fvrULiEiI7VWY5taarGR0AW8GrPp5QLaSMueaRQLZX+4jAjixn7hl4RCvAXWBqut41VX27U8G2JW/3mL0osf8Q==
>+buildbot-master08.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3CRNu6gMafHiuk7+U20snQNjdzYsbTzSvE56ja6VIaavLMwe+K4aRH58MFkqIuUH+ZfNvehrltz4qU9BJXlw8B61WPLmii+G8bmso7xFWz+hGP8ecG4B2C7kfiBd1KxDCEKxmt3Eq8/AmMsMDelZgz9CLsaVaASy1MVeqF4ssbRZUNCDYLES2LBUgDOjUdgyz0XvBxZj2MPnltPpx051GXxe82RbG50s8LJMs/3/9Rs5v+v/Xzm/6sC5sZBvKox1wTuK48O5LbolewUDjE9OuAuti0YXdszIvqJmQ+/SudnVXuvAsUfPR7cFFf0NnUZMYt0maRsAKzlRnGdQiXaXRQ==
>+buildbot-master51.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx/QmEDU0Zdp72pTVnZqM8Yc1bwLp+AokfuisXdYPs4G2AqkRDVl5ItNfEExmkKeoa5fdjIcanrY1JbU+VBPuae+lBD8TVZB72nokJxyLjgJAqDb3fSz/tMmeqkdwp0TodBqLZ0hDykBRxY5H599ZXnzAEvvXh468ch7tp+Ppml5GA7kF1e03u/aIjaqVgeJSDSk4DNpH6s6H8DjwCbaqydWVruFYcqYWA9AiI9ijnvZNk53ZQ4VCC34MEitOZmZVL7WiqA6cOVTKw6FD8UXnWmI8Gh76+4ChgvKwAfN9Z0uktrMTf9N0z5dwgVjSjXcq2dx6JNbJC86H6sflyldcAw==
>+buildbot-master52.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubtdGqNf0dj8HVKCdeff57gQAoBKh/JSeTvjdEAZZo+fEOqcw0WQX+uFzxnwIN9q6fx7hCdiBjaq3kNKJYra6P4kij09kPFEo5nW92/aBVssaB3OzJIp5VP4zyEHcxIFEY5RwfvEAd/0YJvnLoiG8xm0txHZKLgTstg8xkGLTDZ6Xo6o6MV+MDJPmlQtIYJNu9taAkFE2DJIvm7Yu08WRzwFeec4PHmp3WDgUcaR6rdzNlo9xdXVLPTToRges2W5hCMp/UeMCDgCQTgmtsJIbGz2bCU43pdQzualfRBakwq3WmpNv/nxroB69RZ6XGUMCiPe/AbJOzOnKtUN6OpJ1w==
>+buildbot-master53.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEArcbGd/hW9ci1Wrb9yirY6vz9nmnKYO06Y9amvtGXihx+oFTDLsb5eSjiV3wrbxI+MTvtfnlzloj9erO5R34WZbv1lM1Ch02/QDoxmR7AWgQLugdO5C4zdpyp7LphEIyedCf56UV+oWtKVMfHEOzBIVA/EBfvy1uMdvLeNURpujBXxz2vA9PyWN430JoketNSV6OXWrxwOYj4bPYvmq66fKhI0Pr3SfMnVpBSb/gYgUViNXsMhjX+uSLK4lz5n188kalYDNjuQA0w9T65dVPiWXBB5n+Mzj81mPv6fwrduP5Q+yzJEzqWIDb9DkOfnJlbxo23zkxTy+tbY4jobxf96Q==
>+buildbot-master54.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAu5gckvqo0HLUSADvdpSxRHL3re2FQ31R7aKHH9E1aRacbEIpz82zofSeJ+sPf+3xNgeT+a4tQnu6taSiZLPbScj3+H3/Qx0Q0Rf9jWL+aErW1zcMZ+0tuNXnrp4KICwnq1EtsJSjbvkhedXIl1jK8IUu9WDTCEbQK8g0pS3jw9ef9t6v1zhXBm963F8jrJh9kEZ5At29asLw3m1olRmDjkoZ5fSGaLevV03ptiz8/nH6vDKKGJaXM1GZezUH9YLSyyawT6Tn62Dmz8peSOF4MQsewo+zwquo27fBY/6ocUmPO+q3y8dSWsMvSuIbXWtb0e1f36ps55OXLenJ+BjtmQ==
> buildbot-master66.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAm9TQVplmwEGmT6r46oXAtVpayeEfTkLeC8dQ6migXKZjNLP3Mw9j32C//BfTy3ZEAzhBFRKT05jwgA4LaIKDbCEa2UiQ43fR04aLASBE1p+ibQ3MDWLqcef2hsrhk4mK0xuHo+hEviWXjKY0ektEe30uWem+xLof1mj4gW+loOxhQp+nYFP/3wfI7ZMNWvUMsVymIE7Bp/gs7jyxP0sbM/iqL1CPoEOdFtsdc1Y3ixnkUdkeu84gSdqmEkMsQyIEsNFwBWL8wI+CAfKcy9ejQWAyZ8e4lXIVu2GwcJpVj6eqjU4g6iMrRd5HWIA4J6bGShP0sW2EUf19tNUUeyJdnw==
>+buildbot-master67.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxjqtBQE9FZ4lwvy4/ZQRgEpHHCUSN/moOUFqE3OBOuprHw+yNyQHOlAd6ZPlFpEnOsC7u8HVWcdHZz7uYMoFL2DnkwvwZ+sFoG7EuEv+Z8Yr4oVUhQgwZd/+dR/7y2Y+4/qDKGZ6nK0CKZ/yLWzNolHkFs1pJ1RL3UTcxEtJqEI/eoe0uRjWLoUfDPqr2DoA84/ia6XTAltPyMFhd7hx8/heUxZpPiEYHnCO+Qjv9nbM3dOjj/DSyH7X304Y4Fr6UdiclbKCtfK3ccMP3+IdfPoV0LHshSCoJlWTKmKdN0yWW/sySVQBQhNFrK4pTwsQYQx/2JqKl9ZeigA8/w9eZQ==
>+buildbot-master68.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3u8EmRSkSMs4XjzZxFYp+gF/44k1a38yFVgC5TY5EeLX0vUgQgPCqMUE3M3SUI2JE3Oy5HXUp47jGkLd2SiDZr1xmIC2MN5NlLf5cKhK6mQlO2F2tFC76FpkzIfOTYJd5R2VZKLuSXn4KNVNuTm3AmtjD+fZD12OEYJyKvkUOdrVFqEKZTFoKaUjei/7Cr2K6SKNdaH6DV5yCsbv7rmVxShGBZq/mBkMFGyWHT3FJcuUWv/2O4QPPxyTVRaH9ij10zitY+QVFWN7MuEIT+z8QyjWzm55f7JqZ04IgSnSaXgAUkTTmtZ1sLg3cWpZHprZ0UvOc/dYY/NbXYeqJIfl/w==
> buildbot-master69.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtTKiezEBKrtYV/IjV82K+Qlbl8J56cPCyYbUGs5Cq/I6tVZIT6uINC28VY7FdMCe7E6O/c2vLxbVDhHBjTnkaK0gaFVQDCUbytbJqUAvgEFX6UwTE/NtV7npnuYIJANkdId5pmB9+xBENfaSylUlzRQojxtwQjK7mbjpNeZbzPJCCjp4HTvBrN+upR9DXB1O/xMgmrbJU8bSZwDV0sPulI1lxK4CY3PQ29uhAxBgtNdUyrdqwenW28WB+cWRXaoMoO6pqLBIlCG3LK7/1cynunSeDWLoJnhepGVAWCYVrcnan5OO8f2DAHXA17nilqDyVaCFYJWMQiv/KfYwv7Z6rw==
> buildbot-master70.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA9C18dD2MYAehh+rGz7Wv7trbrmTIf4jIYzAuyc9dN4ljNeCOd+khDIjcivT/aygqHKBkNlAh21JF+hrRhvP7y4p+ZgPMjiGOETWrKyX7ZXF4+n5GAKqYvOSUXLSG2cHmaBwpD/ohiC3z3OYraner4vDVyQYoOpNHGe5BjYCJQzWo2jcSrJStSmkhm770qkHTq1U9EThc3cn3amB1pDDvtTMMtRwnrh/7MiJ5tei/edWu9Xnv0Ck1pmBX6k+hJFKh0IB6Wi1WrSYrKcgMlT3cnLATD/Edj1w92axBKIE8FrkxiZ/LgWxg5GB+MDY7gfo1sMypSlHv8tPJb91mnTgN+Q==
> buildbot-master71.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAn2zqO0yHp8RWTGu/wTSAxPXXKVYzHDh+t7V9kMUOttiB+uoCrYBSlz971iH4XEJj4aA62WLyX5VAmOyWW5n5ba+NXAhLnFdqzkH6S/e6n+TU59z4vMxZfmd1zZ03pBX3RXlMLE4/eWI42FPP13I87pvpyILmr1eMFctT7tuOCJdT6+n/aK6/VTdZGZeybpsqGOpQOgpxvaDtsUXWN3J4bBHzK+cTJKCFXEUZ69LoU7Y5lYfgRmbT4FRFH4HsR3E3aB0/YA4Cx5Y5aUvuTjNja0QaRL8OjuWOt+dGudutMYUapX8mi+cMQzKGNzoKpvvzXRl8wBnAYXReaEGENDmuuQ==
>@@ -29,6 +43,46 @@ buildbot-master86.bb.releng.scl3.mozilla
> buildbot-master87.bb.releng.scl3.mozilla.com,10.26.68.21 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzgPB9Z1yJNdAdpWyDw667FfNVLCn2DBoGopmMemt4FzLZCj/fV9v96tdT6U9Q+ZFTr50k0NYbXHpQiPhnckqCCAOzB0W0lSg6NnH2uIOZxLHCSSB4bLK2jGn5R5I45ILp5L9ASfYwQynMwWVys2TFxNdkkAwaxM8hgdV3VpzqY3k945jempmMlmUsPSmpJjOdAk57Y0Jy7f8ZEwlVB96p4sMQmsOgVwQcKaYYNDklQLagR+6/BB5rZc95+NCAXuHxwTibbqpFFHOqfKMfIwKz4zcDiTMlxh3eXiVUYSM6jswvcfj2JpAxgdD6EYzG+OsG2acre1R3vyd7xBtdmmpTw==
> buildbot-master91.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvZFjfzFjcwmi1mLoirt/B75VK0k8alimDr+pAdhQCr1mTwHwoYDra67GVjpJq3DXcDM6r2gpsZmyou2/ieERrXgpVIKa8cuntouYqwuyYanGrtI+/4Ff86HxkE8reP0kdCBVGpC0XyxPf0AIC+6sAv6X/n3Fpemov+aOG4ExE6LjZBKQ0Y/pgV6QQtDsfTgNZcqEo9udZUjdkdjsZs2wGpf9w7AN/sQGUj6xePjOrA2AjbpYUctol5fcW6OGBJQN689fsrgM61og9Y/vOrZYAaGxlGCN8B571lPAvtQXVkbvX5m/xY2aLhbcn9365Wj58oqt5tJALGnjemAD1cwR7w==
> buildbot-master94.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6eJSP7e5IgxG+jxSjUSC62gEyiTy71AQqYUQjZa8MUF6CoJYBAD1vYJdfquy/i7I0WUbcFK+0kTdqvACUsY/o3rc1D+DGrBpomTkWFjLGncHB3wdrwcgWBckkjSmt1rGHA1qx0gscVECeFfDQIWTQK8/DoCqhN/DM/LsXbKDffmVxTSpbiomKtgA31GeehWLP75LQzHYar2xQ/H1SAbD6XPmsD3AhcJb4tzQDzWa3vS2vJElJJVF9PrV1KAI6NbnSzFkNg1VdhteT8hc/Jvu69RAX79zrdis1THWYDVuNbuGNM98iZWFNLP1Cj354cMG5izUiLvzciE3V6D870bdmw==
>+buildbot-master103.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwGw0GYZ6wcAMcL+sRNUOpE7R2YwgVmJjhsHx6R/saChk/JifuQiZV8ZQtfmuip3hru3hrT1HO7Jy9t/hh3FljJjRjIGiYPBtPyxvc5JtxCEdmBsshVVRQ6xJJeDOYsQ+S39qBEItUnKhmBV+aIyQbEWGATKYSiQ2NIUAnGOrwonoejDrjW23OF1B2Mpq7ZFSokq7UCT7de1AON1gARIgYctF6ErRhqvTOIGKikoe0xzGwZVgE2EWYSWDk1KLhd5zWMUqSwaPF8fATf/JMlCXw34VXIKAERFA75bTuyz+v8OjahTp/BZlpTKEv03B2dUEdv7tqUX8jrCl9B38SNjaBw==
>+buildbot-master104.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuKRryPrc92RGllJQ2GeizDxdb2zBn1W9MGVZOORrS2/vDuu+8b9EJCZBkleWNa5b3+vXDaJ0gjRcemdvoWFHA5IcKIqmjD64tnz9wS5VYeUS1DTN/sgvNZo5W7JOuOBsNRKinkFNSvuohMI63HDw2y9NvkP4erd+fXBZYN7FapQXkMnWHAY9QKbuEsMqIXns3TlE7QOcRXwD2PzaxYxtRzsgFbno4nZHInIp7SmK0qDTSEDkJkqVG8hkcJ1biijiLRiqImDifaKMZ5IwAKZ3qRraDFfgl5zVfuYnDH5lYgQiQLDDTOL80xs2eAGa6vSAv4CrWZwxCilPCmGWDOi1bQ==
>+buildbot-master105.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAnOQnre+JTpzc1apJU42iUamd2zBrz6ferm/II5HPWh1RhEs88HyzwpImLZ15QTDx2rRJ211+uSX8IsvdEN8YsrbzbT3QrDVYkrbbwz2f9oKPEu1GB/4rrQAFeWHYCDSbZx0e+48NZ3Lnql7s+e6fX41EudOzs3Avfk0e/USINHgNmRY1BnYGU13/FNXKZx+I9KvyWUaSsq8WfFaHcygidh+1QrmvDXdwIN+iMxe7MNwWFkCIyKX18mNzgrNsjjbvPxOSsOcrP2h2WOiWV8K35p4znojaJAIjLvXLIFbzqHBlcFFGYJcHwL8PTFbRMV8fv+P89CELfMew9fHULbW8VQ==
>+buildbot-master106.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1lszqs5QKlCVur/dkXxMthAETIrKVezVqJ4LHAyVFU9R34VLodFW23quYgMThNUCHaK/H/h4xoRrj1D3wBb406hwadOjmBhIgx1c3WMHDYw4j1dlJ3uwQx6VWghwXPMa7uT6gjzkL92n16rWcOc78rlQMRlRjhYsvxSmwFqhQKTAdP2TyaCqYtySr/4QAfnsLoQnZn4UvZ834iy4QzAJpIFhv4OafXkVhN6OxfyHXyDmoyO0Ifes5wW3R03x0cvjrJ/0RiQ3oY3ETeVtFsqsWWQz8GGqMv2PaNOF27DCy8GL+ArVIRYcSl5nPlKgJM6YX+9Y5DlPWolHaByGXo5ZLw==
>+buildbot-master107.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvdOOkD2UgDVALpwFQmmyV+SHo+hpIrUSgxS3C/AADUUlLDHhFPxq9SrTc6H/hEdKV7MR3TtvZXIz1tnk1VQfY9pVNT31wmY84gsSrtLRY84CBMX768dGBAIq3OYXLnMZfLr/8x669lqqH1Zgq/suCSPj31DRySts7v2fiNRJTfORAGl/likoe4t1+BYPOLE8uc7jiiMh6f/XiApT8HixHlH/I77piHmveXxufJzs77lXUkJgkunv0vOCvyOKmL9ntFBx4INxACpSCWgEgXZhIRWiJpdMNOeTrxJ7VjpmBwVmAI4wYaeDBnPz2Dbf0tT19hWAPviqS95eppisVwHdgQ==
>+buildbot-master108.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyDda2YQIMvLFojt0ocPxGk0KHpYz39Y8nTBypTF8EtgU1ByIubasrdAofKzYZ2jkj/a58szs7mP7+d53s72P5mbxFhPijFalBMukJrGmqpSDCa5gZSBikocUYgMoP4bl4sDNlrzgN7HsxWmM9V3fM0ggR/zXXsBiaewk1WFIi6LkJtQLhgJAQZy6j/hCVIyyQyDwYGavBsYwC6giAt3400r7S92zvZz8kvogmILSG+PSoK5qewdozuM8Q0P/FhkB+4lFbs7CtnfbYRkGMm3TlJDR0zauUuPg/aQiao3bOwKklXvo8NqmLG4hhFaOneoBeG+OcR/9wUJz4TBUPO/zzQ==
>+buildbot-master109.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsSjArpUaDG22kX5GHEfaQwhnWENepIwmS3h8x89mCKhNhamMyzloNeulcEPX81bdTNu+A3kH104kGU0u0oLISheihnHcVIp6mjFxsrktFkU1OEB6sClN67goBB3HPaRCnm41IEA+x5iII+Rr6Cb/+umQDwkfB9sP6rMQWZ1P5qN3pa3eVzeLY3ssclPjcxeD/ZFbjEvm9JtSHDS6quzS9K+LRFHM5cVEFq75XotrAm48bnDGxYscMz/FgOVamfXzA0Ni8LyQw31tlxjwv/rc4MxlhFlGeMaEMCWugPiKAlh5anbAVlUbuxfg82UTtcIh2V77Bpg0X8/CDYD+pxzy7w==
>+buildbot-master110.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAu5IiCwiAcN+YbPl5Nm6R224dCedODbkdvAT7I+T2thbCjdPHNWevDYHMYZ/EGf272K1eI4Pag4LyZseVjYozTy/Utp+fZB3H299sR/dEJEsma13qg6r+5xNjhCCDdmABiHM/JYkq/2tyzRurWsbRzd6TlvM1KMQpJZCmjLncm8S5RFJkzLgY1n71kQZX1YZ7y6zJ1ND4p0100ms6tpatc7IOyZKph0zfsQeU/uJK27Lxr1GYtENMNBF7asxdZ/VZZZKlpJqIrAwOsPDaLIt4u9AnkrV4dVoQ3qKJJOTHHM+fSOFu9cqqxuFNEfVB1QpnPhkovDl6WuFqd+fe2k12Gw==
>+buildbot-master111.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwkSda19gF/HLYMmVTZ1p69UACjWzeXDFiHbiw2dcAIPjiw6V1uHx28UgQ7KGi4IuxK/1vn1hO/bbw+wNHK46K6xvGoS6ZubWKJRQ2SxdGRIwPxCd5bVLiXb21PDa358eOp/gBDfQsyoCkaI5YrOUmhK6YtWumQXAT41MoyrptWYJnjqDNtKAwg2vZlVPXZ2ZqXwCD6mI21iMKKIxl/RqmwPbhQo5Sz27HMKQrOKe4i94SOO3QLccuP9deljD/gdiXMvqHBvPBdSwPHGrU9C3YKj+/vO8RzgFzswabOC+g2QAnlAn5+xtCvboe2yoT3krXoYNjo5Lj+ebQ3jP9Zt14w==
>+buildbot-master112.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvzxWDFUJqMu3MCX5GNLzrxUNb7ZZtxXdj8QwJ4jxFi4XHwIMgv/KBwkYBsQW+0pb6WkOuEsYzZTBJ4vnOc+3QDKNzlxe/tYSvMrVk1tHlcd2/FNWz72kj8L2lSRPz7DJLCHHO5uNtAADIHEj47yDRnSgPsmdNF4ZlE02/nYr3GlCooDcQTWFyx4VGNpR3TR0myp1z++Iur94uVyGmNlOXPTdPDwJsN0NeidBrGaUP9fmlfVHyVFs/eBCM6/G4RU9YuOMqHoSv3XBmv09wxZEfyeDoKPBgwU9vrnqDD0gEo1ntlrUjgTCmcphuTuqATBUVOB0e6Xc42ICOjM6EMtLQw==
>+buildbot-master113.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzypyNT5QMJFNWV1kDT2VL6anYz0leMuLpJyJGd7jyjGdQ7Nw/OIl/ftCI+wAfuwWzbcusD3Il7tnUAhjTFlzrXzc3NRu+oNWqoLQAj1+LW/EAIKaKwzQDCs+Yc0m2RkhR3PlhhYGoHF8S3R1XRrn438yJ2+XpC450tDzVmE/7URWUNYiivOtdUuIPgyDvDA88vjNpvrwPiOdsAZUSVrJCwl8zmph+Lc9LgSaQ3nVYT0F9xKopBXn7S5gRnz3X7kCDzMiFBD4TcUw+bP7Tn9VK1eN158lzE/1vEK4XBwnnntzYw6Go6bnOzjhkg7X7hogx5jOviAjpl6807BsCN9G5w==
>+buildbot-master114.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAt7HlfAuE4CYuH9Ujw26b5qdz4lu3p6afecnX2Gx7Tw0PSAfVWtN5PlHpV501We0pStOOvU3vjTVb7As8qqAmGvYFxRfIWKRz3d0KEu7FpOsskMO7M7hNmoU/m6I+ccRCNkX/YMdceQAujKZitkYV2zHJ1ihzYPLfxusGNnO1uyAtNZ8ngcCcqbYRGhStQ2EbR+puWj/Z6VQ5LGg0Qj2DqVbDcpVl16q16a14/cOxjkHeHAUpG+VaFygHx32emrLqZHustnhYwgWtE7YpWko48aIYM6HgH+NYrqvu3RqKypIDEnBko77vXIqJyG/SvmNmHxpKf3k3MJLvHFjDFItaPw==
>+buildbot-master115.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3O1Je1ZRDFZ5byFLO36EvEfpDCfrtYqFPWngfQ1Oyl01vPqveBG/L92g15fMj4PXM5B4IVWhjdjZPyEd2zFuT82YEuq516LXc+1JXSchjdnAGGg34amcyqnVEbSxeTSSNyeUxCjQ7+RJ3Jc8LQCM5JpKb6nlpQrMGbPhiEGg91FdFacNk0cZk0AskUKn8M8VATP4yYiH9oQxSTgVccWt+QLwP+jts6yjMPSCSzPS4+CQfNmaTh97m6FcKDTFc3qyuy0meB47/9eCtFJlZhHKBBvetVJorf1Teac6Q3grVWGLTBJcv01jmInqPqexrsb4uBVYqW6YMm+OPWKiNueTSw==
>+buildbot-master116.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvedrKN/rqRpIbza6hwpdGR8J3lwyQJvnNzzHYtcISiscQXioOsV6y8l3uaA0vOliexuLxHxb+eMPM/L/1qNBDVskt1boYhowE3jPxViT2OGRmnDelVr9Z5zyfn3nPt/ZsmgTTGg+HyNcUNPbtSuaiRzZb/hbLpYFSu3xP68reBVXK15P22QXzUD/aiXegAwULVRfkYRslApfpZJrajJIqxVKSm+L5VwrP5FXLeyP6F4dwj1IE+7sxFHwtfSFKm9u9oBkdk7mK+qN6TaPVXsVgQgXYI22r3pyCy0PXYovKGFmXCY6/rFeEerVqYWQku/o13RtMwTZM/UtK0rioC2+qw==
>+buildbot-master117.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxFDswwvXfJjCXo3SXM8tYCJePQ9Uf7NjeCvlotIdG1Imma3p1FGKFEtUEAUEggc4qzQDxotkpvyBgOPC52lSztk/IE5rROgKthhDFuxLWFlInvPOmnpNWcfFkTBcSB/NR7hy1KtnEN7AAeVWmTz6XUMm2YVOCuG+996qsSrGP4exXp6dS7NWlWEMBt810vhlhux0YN6YFdC+8D8sfZ5iJyhIOy/e1W5kC7GK2t5loUhwVN8JQEP3zvUBegSwgVBAkKqTbQSEhQIVf1vngDTs+HQR62whlCL+gOK4pSBBuIzoowQE4LuAE1SUJ6qjfYG1NFIj/T2LinmHmy7jGApznw==
>+buildbot-master118.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApsVEOdTYCR+Lc7C3uu5OfnXPh/9n72uyVdgD2rSzW5iTbTZhJ85a4+nEQktlzZh6Uor+9Y1Vsegq19BG+NcIULbAIF7/hsl1IAzTdHV7qgaqPGVyH22iRDCHEft3h+2zO/O1PI0Suqmy5Dj5coZSeZx1Ezfl+cFwJwZvHdqEDDlgX4dl9+kVvLFMGt73AQEH/DM+8oY7IvAXR63dcS7iNmS6fIgYojIqSs3vNutfnRvp6JzUGvdlvPwRqqYEFDN2BrX89BIZvytKsSSI5kXzS3rRG2tE64b05gf5kcW5GlrOuW1BoHdaxxkFvdxPQ1RFEkPFUlqsGQZljpw4f9tYXQ==
>+buildbot-master119.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3cI3vxJRBfGGNUD2ChkjH38hpkFPyyrzhF6q1jbffofDgImlWt1zvxResNx/9NnmQ8JB1HurKrWibOIVVZAn4wwQ3bAKIEIIad/GSrU2bMsTjeRg/FnTijutxJ1lteH5VVh86ABfm+kSqO1hm9pJK/ilbh507O5mQkYdQAFngAHTkb6s0PP6T1z/8BCBe//0TzKWuJeUSY89536qdtkSUzM+HV0tMYGtOi2W9N5YcOL5y7VujW0Of57PQnbTZWrP7dh1Meh1W/2BYT31lApeEDdgdrTEWauEbvYZhdW2UBqkjDcMOMBbYRURGko0aougI3pu3nEFUZWFkQouEBePRQ==
>+buildbot-master120.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwJXhtFZJPd4BQ5hYP78ufKQHCMA0u7Oqgv3+HRgRf0ZcVIq1AzwNMLQYCREe5sM3Y4vOuD/bbkPrcCKWLWeEBObl3NPgWTjy8OhET8Ru/i4X5wG989TUWGt89ctUb2iMcjquPRBlzIjvrFKCB8ulWbRwmQjKJ5hFy2C0o5IGTSmS5viGJ11mJLSW89XnNB+iW5t/TXRg6s6VCGXuXQtyedK8k0fbKE737E7xYPx+2lZN1FJb47R2kdoltpZbtPWfDgK/99/bdSWxFtUBvDT3RyVDdJZ4aZHoFcGLuSI8Qq1ZQegc3tFdb2+TppCj49mhTd3KlRZWu3/9lLOqLeYNiQ==
>+buildbot-master121.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAt6K9NiJtvW2SA7DifsabhbJaF/28aAisRK0jQzM+faTP5g40SukmH4f1TsYWx3Bq9YtklEkGtfuQRJZVni2vDL+IfQWLl5E33Sq9uGVR6rdMpOPGFuO3DViTGl6lkvBj8+8g5H0tk5XeUR8KhbAqREiX30Iu19iKSCqrF9amoYwZGd/PsYorSIAEHQd3cLIE1rUHXUI+9aSHZAw0FO4AMlRlpxoR4WWBorh4WRGLAiexEAfiM+enpsdApAgDY2zlmvB9OzsMJ0rO5NS6+wv0Ep+yT5oQxys2okroQf3VcfVNXjCvPQg64jxI6FcpuNRg27tljP7uawBD0lzJ1CnK7Q==
>+buildbot-master122.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA48HTs+tzlRfBwd9JikBVR/0hWJT/lBtVnr7Z2YlpofaqZuAdVcdtW+L2uZAaE3s5zqeTytEw2vr0flNXqkxTTvgSVyBAdfwb/02QA1MCB2LfvgWFxsOHXomvgdH90Xqx4NNQ5ZTnGqobkmvA9tO4XdcWsjo5bQtUugfNtpWoKOd6wTpuuV+e+3GUUltsqaW02UrypfuDxpB5jJDd4G2ZY/pcoaxU976aUR2eSEjytsHl+UwEBJ0SLAJ8nYKVVtgNjbJvPLKXLDZK8QOYIGYcjhop9wHlsXaS0wafILN6xECnhVKxGmm7iNRH7I2LaQAjW78fGD0GbXgLcikCXKFZJQ==
>+buildbot-master123.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6H+iCTlH/y07M6wneoOmzkqBhNwVHelr2PeEU0YT+hpJpSbrOntqcR7mzycoXh4ukDPKrUTmmcLOUc6ZS6TiiIPx83nCdzya4ke0xzb/eeEefFlHeSZ4T0AJn4eX97R84G+ZsoKOReqw1MEq4UlaPG0T4j9J7MDI+D+qTjEs3iV5BDQyxcWAkazQrxMCkw6r6CVoOrKzdDRz8eGWBg+pLD90W8R1JIOByBqGLQi+Geru6lHoVfhfJQ+prfgfoYJoYexqrPy1sqV1exGHcaowmsImfGgYFQT2JWj0QjP23gh+EuoakkjTSVeFCnQYI8kUg2PPhMvQQIYxKJ2RPFWhuw==
>+buildbot-master124.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0OkkhqXqAf7DhubwAU8adACKuRjCbOFCrBP20xk9G604YeVgXAgfCQ41GspUp5YNnIVCvy/3X1VkWK5yuja6SWTmfNQwAxgHxQgcCbeKQ2IWK1mRnl7XQtWBuXIFyAWX5H3xrCpLP/h/i5wpQayknnrtNzsPQlRiILhLEeY9ipYrEHHnWthUJ7MGMRTWDRXX/UE+lFPmTOWh+RGgmZJWwSCvosebm4MetqYd7RsCDmN1kQoUmB+5U05vppuHBUkQeQNtBy/TRKFHrTCBV+7RRcO7sJUFN+3ypOxO4IY37WfknAZV5L632XvMccCdjun2pnBIaRuwdkBPU/QbRwL2rw==
>+buildbot-master125.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsiL6m3AqdhDc0T2zyKCk57doKz70y4p2ZiODbn37oKEZjkkwgwJZ6geEmlg/muvwBUqB0agKWLw0vjfUOxRTqc1gAy+7pSHVYHt3yrD0nFxLN0isftA5HzkgXA08GRAYLkaGR0rLn1jPH0ZC0hXrRAYICwrd8lCTJldWp9sAyoh82l3OVXH7VouobbECR3Zzi5+mHZB1kLe4Rd+cTs7jTRFL0CpzyckfxgeCWy4k8lISfxg9g+gii1VOUhmhyZgpImCBhbp234FzzwrjyuKAxb2GVKlKd/ydgSppOIlw3mbgZaoBaMYcUxSg83uacz6RqjqF2Xe5d/vPC5YfiLXWxw==
>+buildbot-master126.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzGD08BgZI/yXMnLLBe8mjwMJUcLZF2+P5N69KKE+ZJMZlOFiAFq7UqnOB6M2Bsm8tEHoLsyQrtiw2k1gISXZDfHPEJbNHoX7pJi+dNE6GDbsXwg29IlcADMyr4QzNfyIh4Ma4g/o6SnJgJnM72lJ2t2jok9YYy8Lq3rZZDT6ZAI8Q5Ztb2UH0hdHgH9udymdesfYBLFltjp8ENwE9n5KqTXk4IpTF696G1xv4mCBC3uuIDbZy4PinG+tdk4gS1h6YK+ruYwnleKSieY1rkFt9/Q4J76FJLGN6sLJ8S3QWDTLAvdZOs5bLJtqtLFg4AdXd/g1c3E3V9zNgOvTQcuYXw==
>+buildbot-master127.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA+j9nYtQgLelBYaqP9KjxSdd5GmG5Ur8IirTjbe/5CLl+A0jOXpiiAjzY/YPNMQU1GaIE7+UzXRmCawv3XcYrt6ja+YjcKXhJYxOPv0W8I1LG5lGk97gRRvo4GeqTLM0/+q9MSPXbqPD3AqCSjuHk55N68tTK36ONhVODutfPPaVh8azURdwuTvFQsmiu91f1YCC0D6JEOD8Do1vKx45K4RVYbbHRtDFHHPU71XtcJZCUwoV0NYteaqwTMQDV8ndvRIdecyPpLPs011WbVJzSmpBO6h7L0J5eVW9p0H0Xke1jIPet5ptP1W6R7Owt0QVPlJL1yzJ15KEqQo5zb9lqBw==
>+buildbot-master128.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA7bNPj9WdIi69OIF202vuNrzjHzfr5pEhP4ymlYTRnyQSc3q81WpTI/stKJh6czdyGAS9HARqtqWmgeez6SY4EWRorJsI3REtrjnn95jW6L8YsJBhg6PfhoFcfFxPd1C+ejLrmxSbpk9dyfotY0aW9EQ2ilRCMRjmU+n2aBQF79ff1Bun/Ds64QlJnqDv1jUG36+NkGrkBMWQBIsUyBnp/j4mIzYqzU9traPQVWPrG29I5iPbPCXUCUDKhL+58NxAGlOYO7OrazUiEWZLF70sZGX2u2BDQ51XHBr0YJW0VgF/lXMGmgRWpq7hRCNDvlENoy4Usxc4BVyU97rwoljN8w==
>+buildbot-master129.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA87AL6t2gJGELI2xkecWoVpWrIc7Ljz6rDVfFPG7VLoHLiKIemx5HbNWmXe2AF83fNdAp9tUiR+C8lZgJ50Jzt8P/C0oZePzBgPYXJqn22CWEQJP+xuwrhQMTKNBhoYWMhTS3Z4RkWTdPJeXz+PQLDGnh+D7nwEgR/sIlDyX7jIc14SkD7bYN9gVQ+bCzk4wj4OOBPmVAGQTTzkgQnXG+b5SoW+rk8ndTho0DWmBKz2BZqNnmHNbNUQekvskJU4oEOSn/gkXoZEX6JpdjcITN90xUnjlwEAdfz+Vv43rdsNIVkKe5BZBghoiUx7WIVF1es7TZ5hz64muAc1Nj+nCmZw==
>+buildbot-master130.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvmvPDYzWbgnLukOnqnmOQUicWBVbp10bkwK0xSd4mRVsIsJQduKJqn3kfbxy+NgZx7aclfdGyofcJIypYbSL8O0t9DJmzdTYYBQve5pvVsU7eF5EpL6oPi618F1HD/SM45kIHMC2w0tfkKwgS+8PGMbZGQC3rk3eoK0nfR6+BCJwIBXRBxQaDvdR0tRjQdLm3/n61/7sPbyMx6y+h6sXNMq2NDeqbCqenTOspqso9WPuvA+5yYvtFXj6UPkMQv1WbmO9Ht+oWi/Dj5bWYATefT5deg6ldyckWyTXV5JuB4AvUhUd54E3L5wWxGGh4by9AauoASTk7sRu/GxOwMmDKw==
>+buildbot-master131.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1MK7EeKR5nRvbHs6+zir3jmlXHQUWUIj63Dulelm6RuAs3/HpwMI2RgaAY9sr/XVEQdEvHtxxu4Ax7dHgxUrmVTMwFq1LwOs/dYldrAAEyVcvTTJOh5DGq3UZn6VMLsbJuPj/3akQiZIVt3qqKDv8uLREI6w2nGupaBv8IykCUsdbPwCdPoILBsjnyRKsMLOaqGmUHykOfCFR91qpb9MHKScy0rqOHFCfUGUSlhzCvgrPW1YmhjYgCq8VF74H5ZiSkRaJsimZOk8C6Y9ehjhfrlhHNx7kpd/lngbteAzxsg8JdRrayuuM0+JDumOU6SvIB2F4zjxsvGWe0XIaxnjVw==
>+buildbot-master132.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuluMsC9ml9nhGimV5dnjw8mxnn81uutwBVzSBEaIbpVrIEoxG8YD3jgCM3PLJgi0KiXWwFWnbn9hqQwnYGmtYmR+ja9BnglAu7mnQtmuLTDY3kHk3bfg3kKV8c2ymfbpDSqY9RF7ECJGDEfXLxfPeZf3wPl6T1xzYF19gVjtP52EAXIG8MDuZ1Hz+irOS0phvd/G5rf6KyxbPVVqQpWn/oJnwT4mFIfnctO5xYTqwXtZXlqJBJ6NSvQXlwfoKP9zbuznaQ0q8T5yMr5p9fIEjexLGlG3OmI5DMpzcY/NdTXI/uXaN9nYdJJ2NFhiIN4mUK8BA/TlOixaq32N0zhzmw==
>+buildbot-master133.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvJUTSzjBfx85wutNLkRmWfpRS+eNvfHMceOxdl2RQhqUds7rJI23mujQpFcTAysDBucbiGPiC0jeS8eAu3vsV1kY8W/umMQNKFnlHtcf2B+/5PNmAVaEuOD6ZQPWTKqWX9GY1bGnQGFMsZ/Aexo0l+2E3/GjOyUzE/ppzw0kUkN4xEIbJqPlUQVNQrPhWtKw2nt2vLjoXZVYpUNVoXTeYVuD7oPMFLsNyXgH0uHZiCHKohUyXouj5WZXeMQdDgdnaY0bzTMQbxRaBUVKLlwY7SB5dK1AZggP7HpKfAE+yplCjP9+eU0+xKNjj1/aoVqD9LsAZDK1osMrq1q3I5K+RQ==
>+buildbot-master134.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwyMHaRw5RQWkP7Yu8anSB9aTUFtw2dv2IqM+OVIMdcg914G6tPncrcT4Vo/gGSTaf1FgX0lFnQoRFmrbYpvpKkdHdSpmmwh0jFKaRAnqFp0tL3uTJGu9t+KxD5V9cwQOPQQWUi+cvvn1nZOlsdajEiTO4PFkrxdYw2j8r37olZIAFUimkhtqyFq4YOvIOKgGtd/WgjhkkrxEmDIt/YYaFR72Zlc0Ebpj3+Syw/19sMVnIGgkEH75yYynQevuxS0AZiIyr5p3UA7v0JRMEglKY/cZ+TmiH+BoONnOaVXwyOXQMqCZZj7DnI3csc7rJgk6xoAUg94hZ9j2sj2nIaTNKQ==
>+buildbot-master135.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuBqpOQgHAuOEvvC7dTNB3UgreWGoUwlYQPZblH0EPYiYbTLH0gjJQYtAFuHJ8X/A9G6XXc8tXK7Zogo3rzhzU8OdMp0URgQOVCe9QBqHb5/Pi2Ep/FR+kTJcM+nM6hG3eFki9vvUsDHLuUBRR1f9+aYEs9LdqfA0SCSuYWNqB3GZn2QZzS4PnyNJ34P+stJqBDIIXDQ8+yxAnupWx9xl+Oukec3zx7KtuyAsXL/ZzEt6q5dYtrKoQEV0f5Xa5sOdgRy4S2T2oiZ59PvHqWzEAoL2pJM2gyQKBW082n05A/wMZdXORX0LE15a7zUPpSbRJ7Xmh+wDv0/CoqnMkq0Y2Q==
>+buildbot-master136.bb.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAufwCLFRdZf5qB8nINu3jSu1ejHa+5Of56HY5ZUovyYPAwqBajWgc0NuCArPGWiRRHxwd7uvAwVtn7n2BjLfSshbjrL77DoxogvIn6lIB1xMBRCcFIKK53Jx5XF5aLSvjFSzlgU5jYCrJZhCIHv2HJTOIB8VdFlOKVOg53KNw3oUCv6N698klOWUZ3vflCk3OC6OeBtWg/m2JBAZxjzBNuXfCofwtCWditwyIPogTeCMRYYntu/1/hoTKWwAFYO0NylKaVCUf/I3As+P/sr6X5j8lLeP72fViPYwMgruEuircbNqLMYOaDieuoCzaLSBjE2B6P+SbIvow1kdZHzuIzQ==
>+buildbot-master139.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuZlAaBXYU2p+Q1WiSczP0sqXbGoTLdSZC+i/0UOSn0gBjNAyzo4rDDTspgm7LWFVNbRS7zfe+vnDfhd7WXrz6OwGC9Bil/dQQ0Q75JtoQ46aNEBEuxb3lVZak0BBBx5WquWvlQZEQp7X3++5nHXmykfYKX39uH6rvsZwSeUbwk8Hmgr1PiwrbNMwD/PLBnWw6u3S2LsyWV/V2QI+jCg5gFSRc+AOR+bi+Mhu+DsvpqpH76iCc789nZxc8LUwyJF6rMMhfCF9WiI8D68LwgVYnzyRuqpU+UaHVkSdQn/iSxmkRJgmTfmu7u7CQ1SY6AQGAWRBCXGZ0o1Iv7M7T4RFCw==
>+buildbot-master140.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsZsyGmsWWCZjUMZWMgTE7tBO1VyzKrYJVr349SdhUAT8YjPhqMckVBlDU/r7Z4cwZbvTlHcjSbT+T6WsP2zZ7IZEHRY1QdOeH9+EqK+UgwF3u+wulL1eSM8nfB7fcbh3Ebb6eP1nF/MzZP31/wXcVvQCdv1LJsuth7q8HezVQi+F6koG654mtTvmUp2TJaezcG7gXLKarislrL+t7ukEjCT1XlyuJ1rlKGEOgBvY9mFPXcXFHjDQccAf1DBzjufgfB0ioaQh30VldbwfBMDQJrDxSSrwbFXWniBPivNuDiTubL2gCUvDEp+pKw0WqqkDj+6ytUKelTs3NT/60j3U4w==
>+buildbot-master137.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwut9J9u7kTfy+JduzNS3lcMlaX0vKPE3kawArk4cPmZSlT6et3mhQB+3PHXUvg3Ge3dAGUtpevpSH94IKhg2FGmuRD46Zc0REfhPxeitceqnWgn5RVgpfDPMcgAOcBTcxLSiVt0n9cZPjJFsO8wuk/X6EUO03EjRjT1QITO8j2Nb8yec7aJYLcJbqmRARDQPihLsRhDphKIZ52EyRI+JrLS6mus/7FzXs1fhgcQIjSUMN6jnM2RSIPJ/yhTPoNstmCnYit126F2gmjibQqQY3Zty6HRXJI6OIM3+qoC2FwvpL28Eo2MJxNkLaVBXjA1Evab7uj76Vy3x3xfQuVhAqQ==
>+buildbot-master138.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4ieNcmmrMgEnCsHzTABWw8QQt30KJtyHbww+5rsYuw6ouLOWSvmRDEhEK782fyMndp8mzZ71Vbi7hUJ17ioTCDNubGAktU1NKT8JjpOhOPIxWMLKxZp0KtzooSqPpHsiXynh0kA0jtkwnYSHIHORpaJ0i9IZK1+3Yq74hbwrutwGug/qSsmaxhNLVhTtsYldlZpXtVRIlyeGT6wOnb7YHmTduxCLvcYNyU5UTdpAVsIhA7iKag6yzel1Y0qRcxBw1A/v6hLVFixPEGj/BaRiubKATffTHvkPI1ybtTAHk7WP4fTrpA4TU0/EebgLGQM1G/x7guIkqTZ4ajB+zEyIEQ==
>+buildbot-master141.bb.releng.use1.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxwzLt6ysHh3EbjkK45MNlaG7I25LmTs6Uo/79ZhAYtMdDcszT9jzorrpxotB9uhIdQGjakBCnM6/+/fZDUNs5uxVklgkTY+fq9F2Hp97Pij+itLeAX1vCD6Rk6GDDQevguJhNQaHurddZz4c4WY7OvzoxO1Z2/6wnnX7tya/JBClqi2KitqOPdT6awtG0qqWJUuek6HWKmyY4zgkCs3vHz32vEx1wv/4lmgtXaC5ZjCvzVSRZnEH1AynQnHL4l1X02kgUzfpm2ZyVmPaK6PUrilaP6zHxMwxVt8sA1aWtHN4L2+PoJje1TtH57r7EDgmqyLSnwunntN71BoVQE79Ow==
>+buildbot-master142.bb.releng.usw2.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAra/+1fwmnNNEmtKtD+pDZkHbls5F6GdM9JzD2BHmLxeobC05LGqXCT5sAPZhQzN32js/op2l6OpXP3+yony5Nomf5mNUpZa48kvdynwrfCHMe4BCcWrsv7hG+SFFpXzTxfj1RQAJ3eP+IxZnEEpKPpczq1ID2beJTrLldH8NclVcBPdPuKFb/uj+Hi2s95V52py8KkCx9CuYg4wJR8JnNpbIQQwz8XzRxdluKKtTtUI+aFRIrLZb3WY4p8ZEJfpR2Z/lUv9V26wYnXsr5VTPXDrCVQzMM+yiFuAWhdJl9v5qqLqD6CoTUJxEJ5DbyxUzxb7hRRrxMTqqaWO+fmvqdw==
> cruncher.build.mozilla.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4avDG9htwaW3tTlL5JE71a9RgAFCAHGF5uIjWqydRBKyhtLVkAqx3v5GOVQI/5E7L2vUz6AnLxOAadqMZSP2l9VNaZA9xJvS5fNiFmBqvsLAxFUvapT3PCXj5s4D2WxbXoM7Rq8WdOwBGTtUsCHezJxv4ihRu70M3sgZAcpUI8FqX93n5CPVrypbQAL8IarkjPZeC4BkoXsg/1n1M+AfU+98oWtKYlrzgjxVE1UsmqbdKLX1q9KVkARr4XY/oUisjW98qOBkfCSWQqF3OXocahaX2pfFJdfJxrlp2D1TVPpDNLqyiH/E/sZkB2GIGHIbNtQgyx5Y8xlHNpd7G0g8LQ==
> cruncher.srv.releng.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4avDG9htwaW3tTlL5JE71a9RgAFCAHGF5uIjWqydRBKyhtLVkAqx3v5GOVQI/5E7L2vUz6AnLxOAadqMZSP2l9VNaZA9xJvS5fNiFmBqvsLAxFUvapT3PCXj5s4D2WxbXoM7Rq8WdOwBGTtUsCHezJxv4ihRu70M3sgZAcpUI8FqX93n5CPVrypbQAL8IarkjPZeC4BkoXsg/1n1M+AfU+98oWtKYlrzgjxVE1UsmqbdKLX1q9KVkARr4XY/oUisjW98qOBkfCSWQqF3OXocahaX2pfFJdfJxrlp2D1TVPpDNLqyiH/E/sZkB2GIGHIbNtQgyx5Y8xlHNpd7G0g8LQ==
> dev-stage01.srv.releng.scl3.mozilla.com,10.26.48.44 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA80JxNARSnHo9ym5FI2vt/WQPkLTePgHxE9Iwafwz2SvAKs/bcz1rUZm+s1YUJ6Xm4i3w984T6q/KSEqOWcYY3HjZ8rntcehGO8zXIVQRI4U1v1TxCvrwsJ12z9NHNrzX+JGtjyb+5RgODNQm2A6klbu/sgVdwbCxTkvImNb281erYyNuwaZ7CvF2yYFzMYtuR0dGMO02aIzIO3phM2PM7bLtWgDZ/zjiFZcuCVcM6vY75sLYyoXPj+q+oezeU5/cksyInVvcwFJ7Jn5WwRD4anqWrIrKjlVNY6XQbPm5OHdw4mXzct6hrLaYdXloT9tgA/ktZ7NDqL6kxeBjplTQ4Q==
>diff --git a/modules/users/templates/buildduty-ssh-config.erb b/modules/users/templates/buildduty-ssh-config.erb
>old mode 120000
>new mode 100644
>--- a/modules/users/templates/buildduty-ssh-config.erb
>+++ b/modules/users/templates/buildduty-ssh-config.erb
>@@ -1,1 +1,6 @@
>-builder-ssh-config.erb
>\ No newline at end of file
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/. 
>+
>+Host github.com
>+    IdentityFile ~/.ssh/ffxbld_rsa

Comment 31

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Restarting buildbot masters from buildduty-tools machine finished successfully.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

I've tweaked buildduty@buildduty-tools:~/restart_masters.sh to add '| logger -t restart_masters.sh' to the restart_masters.py call, so that the logs go to papertrail. Also streamlined the docs at https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_Buildbot_Masters#By_script.