1150580 - [Air Mozilla] dev chief on the fritz

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Peter Bengtsson [:peterbe]]

The #airmozilla-dev channel is flooded with failing builds [0].

It seems it basically has nothing to commit (because nothing has changed since yesterday) but it keeps trying all the time. Or something :)

[0] http://genericadm.private.phx1.mozilla.com/chief/air.dev/history

Comment 1

[C. Liang [:cyliang]]

It looks like these attempts to commit are coming from a cron job.  [1]  Is that cron job still needed given that you can push out updates to Air Mozilla dev via chief?


I'm not sure why this job did not spit out errors to the channel before.  



[1]
4,14,24,34,44,54 * * * * root /usr/bin/flock -w 10 /var/lock/air-dev /data/genericrhel6-dev/src/air-dev.allizom.org/update > /dev/null 2>&1

Comment 2

[Peter Bengtsson [:peterbe]]

(In reply to C. Liang [:cyliang] from comment #1)
> It looks like these attempts to commit are coming from a cron job.  [1]  Is
> that cron job still needed given that you can push out updates to Air
> Mozilla dev via chief?
> 

Some sort of automation is good. I don't want to have to execute the chief deployment every time something lands on master. 

Ideally, I'd like to be able to do this in a GitHub webhook on pushes if that's possible.

Comment 3

[C. Liang [:cyliang]]

A quick question: Are there still messages about failed builds going to the channel?

It looks like, in the course of troubleshooting something else, JD deleted a file that was preventing dev pushes from taking place.  If that's the case, I check to see if there are any technical or policy barriers for putting in a Github webhook to do a dev deploy after pushes are made.

Comment 4

[Peter Bengtsson [:peterbe]]

(In reply to C. Liang [:cyliang] from comment #3)
> A quick question: Are there still messages about failed builds going to the
> channel?
> 
No. I can't remember when it stopped. 

> It looks like, in the course of troubleshooting something else, JD deleted a
> file that was preventing dev pushes from taking place.  If that's the case,
> I check to see if there are any technical or policy barriers for putting in
> a Github webhook to do a dev deploy after pushes are made.

I'm in your hands to help this being set up. It would be a big boost to team productivity to have dev being updated with 0 delay.

Comment 5

[C. Liang [:cyliang]]

tl,dr;

While technically possible, having a Github webhook push out a dev deploy is not recommended since it has not proven to be stable.


Long form: 

Another (publicly accessible) web service needs to be run to respond to the github web hook.  At Moz, this has historically been done through freddo (https://github.com/oremj/freddo).   Jake recalled that a number of teams used to use this (MDN and SUMO among them),  but they *stopped* using it because the service was not terribly reliable.  

With the impending shift to AWS and (most likely) a completely different deployment model, I'm not sure it's worth investing in getting freddo set up for Air Mozilla.

Comment 6

[C. Liang [:cyliang]]

Closing this bug for now.   If there is a strong impetus for doing attempting having a Github webhook do a deploy before moving to AWS, please open a new bug.