1462607 - Run-Puppet :: puppet agent summary contains error#015 on y-2008-ec2-golden

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task, P1)

Description

[Adrian Pop]

While monitoring #buildduty channel, the following alert appeared :
relengbot> [sns alert] May 18 10:16:46 y-2008-ec2-golden.try.releng.use1.mozilla.com Userdata: Run-Puppet :: puppet agent summary contains error#015

Comment 1

[Adrian Pop]

the process to regenerate AMI has been started, waiting to finish.

Comment 2

[Adrian Pop]

the ami creation script is stuck at
 
2018-05-18 04:40:47,989 - INFO - Sanity checking DNS entries...
2018-05-18 04:40:47,990 - INFO - Checking name conflicts for y-2008-ec2-golden
2018-05-18 04:40:52,031 - INFO - waiting for workers
2018-05-18 04:40:52,177 - INFO - Using IP 10.134.48.10
2018-05-18 04:40:52,742 - INFO - subnet subnet-33a98358
2018-05-18 04:40:52,742 - INFO - ignore_subnet_check, using subnet-33a98358
2018-05-18 04:40:53,913 - INFO - instance Instance:i-07d4222794d7e245f created, waiting to come up
2018-05-18 04:41:05,015 - INFO - assimilating Instance:i-07d4222794d7e245f
2018-05-18 04:41:05,170 - INFO - waiting for instance to shut down

After two hours is still the same and the problem still persists

[sns alert] May 18 11:53:08 y-2008-ec2-golden.try.releng.use1.mozilla.com Userdata: Run-Puppet :: puppet agent summary contains error#015

Can someone check or point me to the right person to help us solve the problem ?

Comment 3

[Attila Craciun [:arny]]

The script is running on aws-manager2 in screen y-2008-ec2.

Comment 4

[Adrian Pop]

currently, I have ran the script again on aws-manager2 in screen 32375.y-2008

Comment 5

[Mark Cornmesser [:markco] OOO 2024/04/15]

Here is the error:

Failed to apply catalog: Could not find dependency Exec[rebuild c:\mozilla-build\buildbotve] for Exec[virtualenv c:\mozilla-build\buildbotve] at /etc/puppet/production/modules/python/manifests/virtualenv.pp:105


bhearsum: Any thoughts?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Mark Cornmesser [:markco] from comment #5)
> Here is the error:
> 
> Failed to apply catalog: Could not find dependency Exec[rebuild
> c:\mozilla-build\buildbotve] for Exec[virtualenv
> c:\mozilla-build\buildbotve] at
> /etc/puppet/production/modules/python/manifests/virtualenv.pp:105
> 
> 
> bhearsum: Any thoughts?

Yeah, this is my fault. Patch incoming.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: don't define rebuild requirement on windows

I wasn't able to figure out how to test this against the golden machine, but it works fine against a buildbot master (where it's a no-op at the moment).

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/c7d40f1d8bee
don't define rebuild requirement on windows. r=aki