1131983 - Code for testdaybot is no longer updated via crontab on mozqa.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With bug 918266 our testdaybot was setup on mozqa.com. As I have seen in the last couple of weeks, there is no pull happening from the github repository anymore. Maybe the crontab script is broken?

Comment 1

[C. Liang [:cyliang]]

I'm having a hard time finding a cron job that updates the testdaybot automatically (either in puppet or on the server itself).   Do you know the last time testdaybot was updated automatically?  Does it make sense to have a job try to automatically pull in changes?  (Most of the places we do that are on dev or stage servers because it limits the damage if there is a bad commit.)

It looks like the last commit pulled onto mozqa1.qa.scl3.mozilla.com is #65, which was done on February 3rd.  The logs seem to show that this initiated by a manual pull from you.   =) 

The only cron job that I can see does an hg pull and updates files in /var/ftp/data.  (https://mana.mozilla.org/wiki/display/websites/mozqa.com#mozqa.com-Cronjobs)  There is also a job that goes a git pull in /data/www/mozqa.com/mciconf.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The last time was probably Sep last year. So you may be right if there is no crontab entry for testdaybot.

In regards of pulling I could suggest the following. We can create a release branch in that repository which we can use to pull the data from. Once we have something to release we will push it to this branch. When we get the new data we should restart the testdaybot too.

Would that work better for you?

Comment 3

[C. Liang [:cyliang]]

That makes sense.  Once you've created the release branch, let me know and I'll work on a script to pull from it and restart testdaybot.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The release branch has been created. So you can pull the code for testdaybot from here:
https://github.com/mozilla/testdaybot/tree/release

For the script I would propose to check the other one for mciconf. We have seen that initially there was no hard reset, and no updates were done because someone locally modified a file. Thanks!

Comment 5

[C. Liang [:cyliang]]

@whimboo:  I think I've got a script + cron job combination that does what you want.  Would it be possible to arrange a time for a test?  (This would involve  having you make a change to the release repository and seeing if the testdaybot is correctly restarted afterwards.) 

I've updated /usr/local/testdaybot so that it now tracks the release branch.  There is a script /data/bin/testdaybot-github-pull.sh checks to see if there is anything to be fetched and (if so) will restart the testdaybot.  There is a  cron job (defined in /etc/cron.d/testdaybot-github-pull) that runs the  testdaybot-github-pull.sh every 15 minutes.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Great to hear that! I have a couple of PRs which need to be landed soon. I will let you know when I merged those to the release branch. Thanks for the work!

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think we have a problem here. Our bot is using a config.json file which is located in the same folder as the code. But a git reset --hard will remote it. :/ One option would be to add this file in .gitignore on the repo so we do not delete it. A working version of the file can be found at /home/hskupin/testdaybot/config.json.

C. Liang, do you have another idea what to do? IMO the above should be the easiest solution.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have checked the git config and I think we should not spoof the release branch while we are on the master branch locally. Instead lets ensure to checkout the release branch locally and pull from the remote branch appropriately.

As mentioned in my last comment I have added the file config.json to the .gitignore file, so we won't remove it anymore when doing a local `git reset`. All of the latest changes have been rebased to the release branch, and pushed to the remote branch.

I checked if files are getting updated but that is not the case. Latest update for /usr/local/testdaybot still shows Feb 23rd but not today. There is still something wrong.

Comment 9

[C. Liang [:cyliang]]

Looks like I got tripped up an empty if and a missing path for a command.  I've fixed both and altered the scripts so that it dumps output to /tmp/testdaybot-run.log  

Would you be willing to make a small commit / change to see if the whole process (checking to see if a git pull is needed, pulling + restart of the testday bot) works cleanly?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hey. So I pushed a little update to the readme and rebased the change to the release branch. When I logged into mozqa.com it accurately pulled the latest changes, and restarted the bot with keeping the config from the last run. So I think all is looking fine here. But maybe you want to have a look yourself to the log file. Please close if all is fine. Thanks!

Comment 11

[C. Liang [:cyliang]]

Things look good from my end.  Thank you for your patience. =)