1043777 - run puppet on webheads after a release

Status: RESOLVED WONTFIX

(Socorro :: Infra, task)

Description

[Lonnen :lonnen]

Puppet fixes the permissions problem caused by shipping a new copy of crash-stats. It runs so infrequently, though, that we're very likely to see the symbols cron job alert and fail in production before it can clean things up.

Ideally the webhead update script would fix this, and there's another bug on file for that. That script is outside of the parts of puppet most of us can modify, though. Instead, we can trigger a puppet run on just the web heads after we push new code.

Comment 1

[Lonnen :lonnen]

Attachment: 1043777.patch

Comment 2

[Robert Helmer [:rhelmer]]

Comment on attachment 8462304 [details] [diff] [review]
1043777.patch

I believe that this is a fine thing to do in any case - it'd be nice to make /data/crashstats/src/crash-stats.mozilla.org/update set the proper permission I'd think (although I agree that the lag between install and puppet finishing leaves very little room for the symbol upload to fail, and it'll recover anyway)

Comment 3

[Lonnen :lonnen]

Committed revision 90904.


I can't get at the right file to fix but this will cut down on spurious problems in the meanwhile.

Comment 4

[Lonnen :lonnen]

$ svn commit -m "set the correct multi-command group to run puppet against"
Sending        files/prod/usr-local-bin/update-socorro-all.sh
Transmitting file data .
Committed revision 90905.

got an error. `web` was an outdated target for the old php webheads. `crashstats` is the correct target for production.

Comment 5

[Lonnen :lonnen]

downside of this change -- now releases take a lot longer. There's a step at the end where we wait for puppet, which can take 5 - 15 minutes.

Comment 6

[Lonnen :lonnen]

mega downside -- the upgrade seems to hang after the puppet run finishes until you send sig term. reverting change.

Comment 7

[Lonnen :lonnen]

reverted by revision 90908

Comment 8

[Lonnen :lonnen]

Changed the deploy script to set permissions correctly instead of this.