support py38 on the mac signing pool
Categories
(Infrastructure & Operations :: RelOps: Puppet, task)
Tracking
(Not tracked)
People
(Reporter: mozilla, Assigned: dividehex)
Details
Attachments
(2 files)
I'm not sure how we're installing python3.7 currently -- it looks like the standard solution is homebrew, which we're not using on the signing pool. Is this from the virtualenv_python3_s3 package?
If we can get py38 support, I can roll that out to the pool. Thanks!
|
Reporter | |
Comment 1•5 years ago
|
||
catlee 15:10
have we looked into using pyenv for mac?
aki 15:14
i use it on my laptop
15:14
that may be a relatively easy way to install and keep python up to date
|
Assignee | |
Comment 2•5 years ago
•
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #0)
I'm not sure how we're installing python3.7 currently -- it looks like the standard solution is homebrew, which we're not using on the signing pool. Is this from the
virtualenv_python3_s3package?If we can get py38 support, I can roll that out to the pool. Thanks!
Python3 is being called from: https://github.com/mozilla-platform-ops/ronin_puppet/blob/master/modules/scriptworker_prereqs/manifests/init.pp#L5
I removed the homebrew dependence from signing awhile back since it is something we really don't want to use. The preferred method for installing python is to simply pull the maintainers package from s3. I've added it the package s3 bucket already.
Although pyenv is nice and I also use it on my personal env, I assume it to suffer the same dependency issues homebrew has and would probably be pushed back on from security.
|
|
Assignee | |
Comment 3•5 years ago
|
||
|
|
Assignee | |
Comment 4•5 years ago
|
||
Aki, do you want to try to deploy this? I think a simple puppet run should update python to 3.8.
|
|
Reporter | |
Comment 5•5 years ago
|
||
I'm imaging mac-v3-signing13.srv.releng.mdc2.mozilla.com, and it seems to not have my user and/or .ssh/authorized_keys set up. I was under the impression we would auto-puppetize. I'm not sure how to log in without user setup working. Can you tell if puppetization is broken on mac-v3-signing13?
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
I'll check mac-v3-signing13.srv.releng.mdc2.mozilla.com right now and fix it (and check the reimaging to see why the user setup did not happen)
The bootstrap stopped because no /etc/puppet_role file was found. So something broke in the imaging:
-rw-r--r-- 1 root wheel 60 Jun 16 04:44 bootstrap_mojave.log
-rwx------ 1 root wheel 8580 Jun 15 13:25 bootstrap_mojave.sh
mac-v3-signing13:~ root# tail bootstrap_mojave.log
Failed to find puppet role file /etc/puppet_role
Hanging...
mac-v3-signing13:~ root# date
Tue Jun 16 11:34:32 PDT 2020
mac-v3-signing13:~ root# cat /etc/puppet_role
cat: /etc/puppet_role: No such file or directory
The deploystudio run was marked as failed, but I did not find any errors within the log.
The workflow 'Deploy Mojave Signing v1' was launched on the computer C07Q73JHG1J2 (name: mac-v3-signing13, ip: 10.51.48.38, mac: 38:c9:86:04:c2:bd) with a FAILED termination status.
The puppet_role file is created by the vault_agent.sh script, which is configured to run during the workflow and not on first boot. However it does not log anything (maybe there is a warning or error I missed in the log). The copy_bootstrap_signing.sh appears right before the vault_agent.sh in the workflow, and it must have run for the bootstrap to be in place. But it also does not show in the log. So I think that part of the log is missing.
The other files created by the vault script are:
/usr/local/bin
/usr/local/bin/vault
/etc/vault_role_id
[...]
And none of those exist on the created machine.
The vault_setup.sh script is executable.
The copy_bootstrap_signing.sh created files all exist. This script ends with a chmod 600 to the vault.yaml and then an exit 0 (weird but maybe the chmod gives a non-zero sometimes).
bash -n syntax check on the vault_setup.sh passes
The scripts are using environment variables, set in the computer group's "Custom properties". These are set for the "mac_v3_signing_ff_prod" group, PUPPET_ROLE and DATACENTER are correct.
|
|
||
Comment 10•5 years ago
|
||
ds_finalize.log shows the ds_finalize script ran on first boot of the host, and correctly set the hostname and turned on ssh (etc).
|
|
||
Comment 11•5 years ago
|
||
The ds_finalize does show as failed in the system log:
Jun 15 13:28:59 mac-v3-signing13 com.apple.xpc.launchd[1] (com.deploystudio.finalizeCleanup[82]): Service exited with abnormal code: 127
|
|
||
Comment 12•5 years ago
|
||
I'll retry the reimaging to see if the failure repeats. I did not find a specific failure, but there are no logs captured during the end of the reimage before the first boot. Before I alter the scripts or workflow to try to capture the logging, I will re-run it to see if it was a fluke failure first.
|
|
||
Comment 13•5 years ago
|
||
The failure repeats on a retry of reimaging. (same bootstrap result of not /etc/puppet_role).
I'm trying to get better logging during the reimage workflow.
|
|
||
Comment 14•5 years ago
|
||
no luck so far. I'm not sure still what is failing
|
|
Reporter | |
Comment 15•5 years ago
|
||
Looks like mac-v3-signing12 has mac_v3_signing in /etc/puppet_role. I'm perfectly fine with us manually setting that, puppetizing, and then debugging why /etc/puppet_role population is broken later. We'll have to fix that before we can puppetize the rest of the macs, though.
|
|
||
Comment 16•5 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #15)
Looks like mac-v3-signing12 has
mac_v3_signingin/etc/puppet_role. I'm perfectly fine with us manually setting that, puppetizing, and then debugging why/etc/puppet_rolepopulation is broken later. We'll have to fix that before we can puppetize the rest of the macs, though.
I'm trying with the previous workflow for signing (rebuilt comparing with install2.mdc1's versions) that does not use the deploystudio environment variables to set the puppet_role.
The workflow was changed since these were puppetized. And based on testing with #13, the new workflow does not set them up correctly.
|
|
||
Comment 17•5 years ago
|
||
mac-v3-signing13.srv.releng.mdc2.mozilla.com reimaged correctly on the old workflow
on the bootstrap run of puppet, it failed to find virtualenv
Notice: /Stage[main]/Roles_profiles::Profiles::Mac_v3_signing/Signing_worker[signing_worker_cltbld]/Python::Virtualenv[signingworker_cltbld]/File[/builds/scriptworker/virtualenv]/ensure: created
Error: sh: virtualenv: command not found
Error: /Stage[main]/Roles_profiles::Profiles::Mac_v3_signing/Signing_worker[signing_worker_cltbld]/Python::Virtualenv[signingworker_cltbld]/Exec[python_virtualenv_/builds/scriptworker/virtualenv]/returns: change from 'notrun' to ['0'] failed: sh: virtualenv: command not found
Notice: /Stage[main]/Roles_profiles::Profiles::Mac_v3_signing/Signing_worker[signing_worker_cltbld]/Python::Virtualenv[signingworker_cltbld]/Exec[python_requirements_initial_install_/builds/scriptworker/requirements.txt_/builds/scriptworker/virtualenv]: Dependency Exec[python_virtualenv_/builds/scriptworker/virtualenv] has failures: true
Warning: /Stage[main]/Roles_profiles::Profiles::Mac_v3_signing/Signing_worker[signing_worker_cltbld]/Python::Virtualenv[signingworker_cltbld]/Exec[python_requirements_initial_install_/builds/scriptworker/requirements.txt_/builds/scriptworker/virtualenv]: Skipping because of failed dependencies
|
|
||
Comment 18•5 years ago
|
||
Retries did not fix the virtualenv dep error. I switched over to the python3 module from python3_s3 (because the python3 module is now not using homebrew also), and that works.
|
|
||
Comment 19•5 years ago
|
||
I'm re-trying with the python3.8 pkg. The master puppet config for signing has py38 in it, and it was having the virtualenv dep failure. The python 3.7.4 install did not have this problem.
|
|
||
Comment 20•5 years ago
|
||
I had missed the virtualenv install from the signing_worker module's init.pp. I removed that require of the python3_s3 module, and I'm re-testing the full reimage/bootstrap (after confirming it worked on a puppet re-run on #13) to make sure it works from base/nothing.
|
|
||
Comment 21•5 years ago
|
||
The reimage+bootstrap from this branch completed successfully on #13.
|
|
||
Comment 22•5 years ago
|
||
|
|
||
Comment 23•5 years ago
|
||
:aki the #13 worker is ready for you to setup or start and the reimaging workflow is working (and puppet fixed with py38 with the PR https://github.com/mozilla-platform-ops/ronin_puppet/pull/199)
|
|
Reporter | |
Comment 24•5 years ago
|
||
Set up, started, unquarantined signing13.
Updated https://github.com/mozilla-releng/scriptworker-scripts/wiki/Manual-Rollout-with-Puppet#firefox-production .
Just missed the afternoon nightlies; we'll see how it goes tonight.
|
|
Reporter | |
Comment 25•5 years ago
|
||
The mac-v3-signing13 signing tasks last night went green. Thank you!
|
|
||
Comment 26•5 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #25)
The mac-v3-signing13 signing tasks last night went green. Thank you!
Great, thank you for finding the problem and getting it started up!
The puppet change is merged now. So new/reimage puppet runs for signing will reach the correct state.
For reimaging the signing macs, mdc2 is good (all current machines are set to the "-revert" workflow that #13 ran) but I did not change/fix mdc1. When we reimage signing next on mdc1 we may need to revert to the "v3" signing workflow from the current "v4" that uses the environment variables.
For future python upgrades, we can put a new pkg in s3, point to the version in the import in scriptworker_prereqs/manifests/init.pp, and we'll need to update the path at https://github.com/davehouse/ronin_puppet/blob/ac02a9ee1b2a316509f58d8328cd52122d1390a7/modules/signing_worker/manifests/init.pp#L106 when we reach 3.9 or 4:
path => [ '/bin', '/usr/bin', '/usr/sbin', '/usr/local/bin', '/Library/Frameworks/Python.framework/Versions/3.8/bin'],
|
|
Reporter | |
Comment 27•5 years ago
|
||
I updated https://github.com/mozilla-releng/scriptworker-scripts/wiki/Manual-Rollout-with-Puppet with a link to comment #26.
(In reply to Dave House [:dhouse] from comment #26)
For reimaging the signing macs, mdc2 is good (all current machines are set to the "-revert" workflow that #13 ran) but I did not change/fix mdc1. When we reimage signing next on mdc1 we may need to revert to the "v3" signing workflow from the current "v4" that uses the environment variables.
Should we fix mdc1 sooner, while we have context?
It's still on my list to minimize the post-puppet steps, support the puppetizing the notarization poller, and potentially allow for python package updates / maintenance without having to ssh in. That may slip into H2.
|
|
||
Comment 28•5 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #27)
I updated https://github.com/mozilla-releng/scriptworker-scripts/wiki/Manual-Rollout-with-Puppet with a link to comment #26.
(In reply to Dave House [:dhouse] from comment #26)
For reimaging the signing macs, mdc2 is good (all current machines are set to the "-revert" workflow that #13 ran) but I did not change/fix mdc1. When we reimage signing next on mdc1 we may need to revert to the "v3" signing workflow from the current "v4" that uses the environment variables.
Should we fix mdc1 sooner, while we have context?
It's still on my list to minimize the post-puppet steps, support the puppetizing the notarization poller, and potentially allow for python package updates / maintenance without having to ssh in. That may slip into H2.
Fixing mdc1's reimaging sooner will be better so it doesn't block us next time. Is there an mdc1 signing worker that I could test reimaging with?
|
|
Reporter | |
Comment 29•5 years ago
|
||
From channel: mac-v3-signing7. <3
|
|
||
Comment 30•5 years ago
|
||
looking at mac-v3-signing7,
it was last imaged July 31, 2019, and ran 'Deploy Mojave Signing v3' (I copied the log for this run to compare with if we have failures).
Deploystudio db has the machine set to run the 'Deploy Mojave Signing v4' workflow.
|
|
||
Comment 31•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #30)
looking at mac-v3-signing7,
it was last imaged July 31, 2019, and ran 'Deploy Mojave Signing v3' (I copied the log for this run to compare with if we have failures).
Deploystudio db has the machine set to run the 'Deploy Mojave Signing v4' workflow.
The imaging workflow completed without problems, the puppet_role and other files were placed correctly, and the puppet bootstrap completed without errors.
|
|
Reporter | |
Comment 32•5 years ago
|
||
Finished setup and unquarantined mac-v3-signing7. Thanks again!
Description
•