957852 - Add files and cronjob for mciconf on mozqa1.qa.scl3

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Brandon Burton [:solarce]]

Add files and cronjob for mciconf on mozqa1.qa.scl3

* source: https://github.com/mozilla/mciconf/tree/release
* desired URL:  http://www.mozqa.com/mciconf

Comment 1

[Brandon Burton [:solarce]]

[[e-mail]3 ~]# cd /data/www/mozqa.com/
[[e-mail]3 mozqa.com]# ls
ca  data  index.html  mozmill-env  qa-testcase-data  webapi-permissions-tests
[[e-mail]3 mozqa.com]# git clone https://github.com/mozilla/mciconf
Cloning into 'mciconf'...
remote: Reusing existing pack: 699, done.
remote: Total 699 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (699/699), 3.84 MiB | 1.79 MiB/s, done.
Resolving deltas: 100% (302/302), done.

-> % svn ci -m "cronjob and script for mciconf, bug 957852"
Adding         webapp/files/mozqa/data-bin/mciconfig-github-pull.sh
Adding         webapp/files/mozqa/etc-crond/mciconf-github-pull
Transmitting file data ..
Committed revision 80363.

Info: Applying configuration version '80362'
Notice: /File[/data/bin/mciconfig-github-pull.sh]/ensure: defined content as '{md5}51f5eb527e01850eeb280fa9bd361830'
Notice: /File[/etc/cron.d/mciconf-github-pull]/ensure: defined content as '{md5}48bd0c481f14f9d67e78ba4ac8fd6b82'

Comment 2

[Brandon Burton [:solarce]]

Confirmed working, so RF!

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Correct. Thanks!

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sorry for reopening this, but somehing has not been correctly added to cron here. You did the initial checkout on Jan 8th, which is still the date of the dirs and files right now. None of the pushing we did on the release branch in the repository ended up on mozqa.com.

Can someone please check that ASAP, so that we can use the tool for the upcoming release update tests? Thanks a lot!

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can we please get someone to check that? It's somewhat important for us to be able to push code to our production server. Thanks.

Comment 6

[Ashish Vijayaram [:ashish]]

Dropping this down a notch so that we could get someone in the morning...

Comment 7

[Ashish Vijayaram [:ashish]]

Meh that still paged. I'll take this and find someone in the morning.

Comment 8

[Ashish Vijayaram [:ashish]]

Ok, I took a look at this and it should be fixed now. The script didn't have +x set... Next cron run in 2 mins.

Comment 9

[Ashish Vijayaram [:ashish]]

> Notice: /Stage[main]/Webapp::Mozqa/File[/data/bin/mciconfig-github-pull.sh]/mode: mode changed '0644' to '0775'

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There are some more problems here:

* There are local changes, so the git pull command in the cron job doesn't work because we don't force a local reset before the pull. Because of that we never get updates

* Some tool is modifying our permissions on all the files from the repository to 644 -> 755. May this be puppet? We should stop it from doing so.

* The current branch we are pulling from is gh-pages. But that is our staging instance. So this was never correctly setup, even mentioned so in comment 0. 'release' is the correct branch here.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can we get an update here? Any ETA when this can be solved?

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

If it takes forever for IT to get those things fixed, we will consider to move mozqa1 to puppetagain, which is also open, and we could work on fixes ourselves.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

2 months without an update here. :( Jake, when can this finally be corrected?

Comment 14

[Ryan Watson [:w0ts0n]]

Paging on call. I'll take this until the morning and try and hand over to someone from webops.

Comment 15

[Chris Turra [:cturra]]

looks like there were local changes that were causing git to exit in error.

[[e-mail]3 ~]# bash -x /data/bin/mciconfig-github-pull.sh
+ DATA_DIR=/data/www/mozqa.com/mciconf
+ cd /data/www/mozqa.com/mciconf
+ git pull -q
error: Your local changes to the following files would be overwritten by merge:
	app/css/app.css
	app/data/platforms.json
	app/js/controllers/appController.js
	app/js/directives/directives.js
	app/templates/document.html
	app/templates/form.html
Please, commit your changes or stash them before you can merge.
Aborting


i git stash[ed] these changes and ran again. can you confirm everything looks as expected now?

[[e-mail]3 mciconf]# bash -x /data/bin/mciconfig-github-pull.sh
+ DATA_DIR=/data/www/mozqa.com/mciconf
+ cd /data/www/mozqa.com/mciconf
+ git pull -q

[[e-mail]3 mciconf]# git log -1
commit 913854925bb14cf4f455c5dec195dd51ae3b79ce
Author: Henrik Skupin <[e-mail]>
Date:   Wed Aug 27 22:22:18 2014 +0200

    Non-released versions of Firefox are not listed even when candidate builds exist (#24)

Comment 16

[Chris Turra [:cturra]]

i am going to mark this bug as r/fixed. :whimboo - if there is still an issue here, please feel free to reopen.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think at the moment it looks fine code wise. Thanks Chris. I have actually no idea why those files have been changed locally. All what we do is grabbing updates via the cron job.

So can I propose an update to the bash script? Something I would like to see that in those cases we always force an update, and local changes should be discarded first. Can we make the bash file look like:

DATA_DIR=/data/www/mozqa.com/mciconf
cd $DATA_DIR
git reset --hard
git pull -q

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, and I have seen that git is operating on the wrong branch!

$ git branch -a
* gh-pages
  remotes/origin/HEAD -> origin/gh-pages
  remotes/origin/gh-pages
  remotes/origin/master
  remotes/origin/release

As stated in comment 0 the code has to be based of branch release!

Comment 19

[Chris Turra [:cturra]]

i have updated the cron as requested and did a `git checkout` of the release branch.

[[e-mail]3 mciconf]# git checkout release
Branch release set up to track remote branch release from origin.
Switched to a new branch 'release'

[[e-mail]3 mciconf]# git pull
Already up-to-date.

[[e-mail]3 mciconf]# git log -1
commit ce2a02757300568f4dc108ce9e6c56600a70dedc
Author: Cosmin Malutan <[e-mail]>
Date:   Tue Feb 18 17:59:04 2014 +0200

    Warn the user when invalid platform gets parsed from uploaded config file(#12)

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Chris. That seems to work fine now. So I rebased the latest changes from gh-pages to the release branch, and pushed them to upstream. Sadly I didn't see them updated within the 15min period, which we usually used to pull our web related repositories on this host. Checking the crontab entry for mciconf-github-pull I noticed that there is no execution each 15min but once per hour.

So can you please do the last remaining change here from "15 * ..." to "*/15 * ..." in the crontab file? Once that has been done we can really close this bug as fixed!

Comment 21

[Chris Turra [:cturra]]

updated cron pushed out via puppet and should be available within 30 minutes. marking this as r/fixed again :)

$ svn diff
sIndex: modules/webapp/files/mozqa/etc-crond/mciconf-github-pull
===================================================================
--- modules/webapp/files/mozqa/etc-crond/mciconf-github-pull	(revision 92963)
+++ modules/webapp/files/mozqa/etc-crond/mciconf-github-pull	(working copy)
@@ -1,2 +1,2 @@
 MAILTO="[e-mail]"
-15 * * * * root /data/bin/mciconfig-github-pull.sh
+*/15 * * * * root /data/bin/mciconfig-github-pull.sh

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Alright. After all this ping pong :) this is finally fixed now. Thanks Chris for all the updates.