879732 - Please push treestatus 0c429f093afa9a591c3f0ab0b791582abfd748b5 to production

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ed Morley [:emorley]]

Please can treestatus master head be pushed to staging (https://github.com/mozilla/treestatus/commit/0c429f093afa9a591c3f0ab0b791582abfd748b5), ready for confirmation & then prod push.

Changelog:
https://github.com/mozilla/treestatus/compare/85138c89b59e6cb3c0328301e9b26e5d17430038...0c429f093afa9a591c3f0ab0b791582abfd748b5#files_bucket

Thank you :-)

Comment 1

[Chris Turra [:cturra]]

push to stage complete. pls give it a test and let me know when you're happy for it to go to production.

[[e-mail]1 treestatus.allizom.org]# bash -x update
+ CODE_DIR=/data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ VENDOR_DIR=/data/genericrhel6-stage/src/treestatus.allizom.org/tree-status/vendor
+ update_code
+ echo -e 'Updating code...'
Updating code...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ git fetch origin -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git checkout -f origin/master -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule sync -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule update --init --recursive -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Updating vendor submodules...'
Updating vendor submodules...
+ git submodule update --init --recursive
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Cleaning gitignore and pyc files...'
Cleaning gitignore and pyc files...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ find . -type f -name .gitignore -or -name '*.pyc' -delete
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Storing revision information...'
Storing revision information...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ git rev-parse HEAD
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ touch /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status/treestatus.wsgi
+ echo -e 'Deploying code...'
Deploying code...
+ '[' xterm == dumb ']'
+ /data/genericrhel6-stage/deploy treestatus.allizom.org
[2013-06-05 09:35:38] Running rsync_project
[2013-06-05 09:35:38] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/treestatus.allizom.org/ /data/genericrhel6-stage/www/treestatus.allizom.org/
[2013-06-05 09:35:38] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/treestatus.allizom.org/ /data/genericrhel6-stage/www/treestatus.allizom.org/ (0.316s)
[2013-06-05 09:35:38] Finished rsync_project (0.317s)
[2013-06-05 09:35:38] Running commit_www
[2013-06-05 09:35:38] [localhost] running: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.allizom.org']'
[2013-06-05 09:36:13] [localhost] finished: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.allizom.org']' (35.291s)
[localhost] out: [master 47337c0] deploy [treestatus.allizom.org]
[localhost] out: 5 files changed, 127 insertions(+), 39 deletions(-)
[2013-06-05 09:36:13] Finished commit_www (35.292s)

Comment 2

[Ed Morley [:emorley]]

There seems to be some weirdness going on with treestatus.allizom.org's DB.

By what means is it populated? It appears to be mirroring production - to the extent that a change written to the DB momentarily appeared in the production site (albeit cleared again upon a ctrl+shift+R - perhaps something up with the caching) - however shortly after both prod and treestatus.allizom.org partially forgot the change.

Comment 3

[Ed Morley [:emorley]]

(To clarify: I'm not comfortable pushing until we know what's going on)

Comment 4

[Chris Turra [:cturra]]

treestatus stage has it's own database on our dev database cluster. 

 mysql://treestatus_stage:<REMOVED>@dev-zeus-rw.db.phx1.mozilla.com/treestatus_allizom_org


any further questions about the database behavior are going to need to be answered by one of our DBAs.

Comment 5

[Ed Morley [:emorley]]

Sheeri, do you know if there is any mirroring set up between the production and staging treestatus DBs? For some reason we're experiencing some overlap between the two (ie staging is showing data that should only exist in production). Not sure if this is a DB issue or a caching problem.

Staging DB location in comment 4.

Comment 6

[Chris Turra [:cturra]]

:edmorley - one thought that i have is if stage and prod are sharing memcache keys? i see in the treestatus.wsgi file we define the memcached.servers, but don't see any memcache key config. is it possible both these environments are using the same keys? and if so, can we add a config option to prepend the keys with treestatus_<env> or something similar?

Comment 7

[Ed Morley [:emorley]]

That sounds plausible - I'd have to ask catlee to be sure, since I'm not familiar with anything other than the local dev treestatus setup.

(In reply to Chris Turra [:cturra] from comment #6)
> :edmorley - one thought that i have is if stage and prod are sharing
> memcache keys? i see in the treestatus.wsgi file we define the
> memcached.servers, but don't see any memcache key config. is it possible
> both these environments are using the same keys? and if so, can we add a
> config option to prepend the keys with treestatus_<env> or something similar?

Comment 8

[Sheeri Cabral [:sheeri]]

The production and staging databases are not linked at all. It does sound like something weird was happening in your browser. :( Sorry I'm not more helpful, but prod and dev/stage are definitely not related at all.

Comment 9

[Ed Morley [:emorley]]

Thank you :-)

Comment 10

[Chris AtLee [:catlee]]

The memcached keys do support prefixing, but it's not exposed in a useful way right now. I assumed the memcached servers for dev/stage/prod were all separate.

It shouldn't be too hard to support a new config value in the .wsgi that gets passed down to modify memcachePrefix.

https://github.com/mozilla/treestatus/blob/master/treestatus/app.py#L27

Comment 11

[Chris Turra [:cturra]]

do we want to wait on a patch so the memcache key can be configured, or continue onto production with this deployment?

Comment 12

[Ed Morley [:emorley]]

Lets continue please - I've tested locally & the bit of testing on staging not interfered with by the memcache issue seems fine. That plus no schema changes means a rollback would be easy (but I doubt it would even come to that).

Comment 13

[Chris Turra [:cturra]]

roger that. i have completed the push to prod as requested.

[[e-mail]1 treestatus.mozilla.org]# bash -x update
+ CODE_DIR=/data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ VENDOR_DIR=/data/genericrhel6/src/treestatus.mozilla.org/tree-status/vendor
+ update_code
+ echo -e 'Updating code...'
Updating code...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ git pull -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule sync -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule update --init --recursive -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Updating vendor submodules...'
Updating vendor submodules...
+ git submodule update --init --recursive
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Cleaning gitignore and pyc files...'
Cleaning gitignore and pyc files...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ find . -type f -name .gitignore -or -name '*.pyc' -delete
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Storing revision information...'
Storing revision information...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ git rev-parse HEAD
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ touch /data/genericrhel6/src/treestatus.mozilla.org/tree-status/treestatus.wsgi
+ echo -e 'Deploying code...'
Deploying code...
+ '[' xterm == dumb ']'
+ /data/genericrhel6/deploy treestatus.mozilla.org
[2013-06-05 14:04:31] Running rsync_project
[2013-06-05 14:04:31] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/treestatus.mozilla.org/ /data/genericrhel6/www/treestatus.mozilla.org/
[2013-06-05 14:04:32] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/treestatus.mozilla.org/ /data/genericrhel6/www/treestatus.mozilla.org/ (1.181s)
[2013-06-05 14:04:32] Finished rsync_project (1.182s)
[2013-06-05 14:04:32] Running commit_www
[2013-06-05 14:04:32] [localhost] running: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.mozilla.org']'
[2013-06-05 14:04:38] [localhost] finished: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.mozilla.org']' (5.293s)
[localhost] out: [master eb5fc39] deploy [treestatus.mozilla.org]
[localhost] out: 5 files changed, 127 insertions(+), 39 deletions(-)
[2013-06-05 14:04:38] Finished commit_www (5.293s)

Comment 14

[Ed Morley [:emorley]]

Thank you - looks good.

Will file a separate bug to configure the memcache key.

Comment 15

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley UTC+1] from comment #14)
> Will file a separate bug to configure the memcache key.

Bug 880173