Please push treestatus 0c429f093afa9a591c3f0ab0b791582abfd748b5 to production

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: emorley, Assigned: cturra)

Tracking

Details

(Whiteboard: [push interrupt])

Please can treestatus master head be pushed to staging (https://github.com/mozilla/treestatus/commit/0c429f093afa9a591c3f0ab0b791582abfd748b5), ready for confirmation & then prod push.

Changelog:
https://github.com/mozilla/treestatus/compare/85138c89b59e6cb3c0328301e9b26e5d17430038...0c429f093afa9a591c3f0ab0b791582abfd748b5#files_bucket

Thank you :-)
(Assignee)

Comment 1

5 years ago
push to stage complete. pls give it a test and let me know when you're happy for it to go to production.

[root@genericadm.private.phx1 treestatus.allizom.org]# bash -x update
+ CODE_DIR=/data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ VENDOR_DIR=/data/genericrhel6-stage/src/treestatus.allizom.org/tree-status/vendor
+ update_code
+ echo -e 'Updating code...'
Updating code...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ git fetch origin -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git checkout -f origin/master -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule sync -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule update --init --recursive -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Updating vendor submodules...'
Updating vendor submodules...
+ git submodule update --init --recursive
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Cleaning gitignore and pyc files...'
Cleaning gitignore and pyc files...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ find . -type f -name .gitignore -or -name '*.pyc' -delete
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Storing revision information...'
Storing revision information...
+ cd /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status
+ git rev-parse HEAD
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ touch /data/genericrhel6-stage/src/treestatus.allizom.org/tree-status/treestatus.wsgi
+ echo -e 'Deploying code...'
Deploying code...
+ '[' xterm == dumb ']'
+ /data/genericrhel6-stage/deploy treestatus.allizom.org
[2013-06-05 09:35:38] Running rsync_project
[2013-06-05 09:35:38] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/treestatus.allizom.org/ /data/genericrhel6-stage/www/treestatus.allizom.org/
[2013-06-05 09:35:38] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/treestatus.allizom.org/ /data/genericrhel6-stage/www/treestatus.allizom.org/ (0.316s)
[2013-06-05 09:35:38] Finished rsync_project (0.317s)
[2013-06-05 09:35:38] Running commit_www
[2013-06-05 09:35:38] [localhost] running: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.allizom.org']'
[2013-06-05 09:36:13] [localhost] finished: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.allizom.org']' (35.291s)
[localhost] out: [master 47337c0] deploy [treestatus.allizom.org]
[localhost] out: 5 files changed, 127 insertions(+), 39 deletions(-)
[2013-06-05 09:36:13] Finished commit_www (35.292s)
Assignee: server-ops-webops → cturra
Flags: needinfo?(emorley)
Whiteboard: [push interrupt]
There seems to be some weirdness going on with treestatus.allizom.org's DB.

By what means is it populated? It appears to be mirroring production - to the extent that a change written to the DB momentarily appeared in the production site (albeit cleared again upon a ctrl+shift+R - perhaps something up with the caching) - however shortly after both prod and treestatus.allizom.org partially forgot the change.
Flags: needinfo?(emorley)
(To clarify: I'm not comfortable pushing until we know what's going on)
(Assignee)

Comment 4

5 years ago
treestatus stage has it's own database on our dev database cluster. 

 mysql://treestatus_stage:<REMOVED>@dev-zeus-rw.db.phx1.mozilla.com/treestatus_allizom_org


any further questions about the database behavior are going to need to be answered by one of our DBAs.
Sheeri, do you know if there is any mirroring set up between the production and staging treestatus DBs? For some reason we're experiencing some overlap between the two (ie staging is showing data that should only exist in production). Not sure if this is a DB issue or a caching problem.

Staging DB location in comment 4.
Flags: needinfo?(scabral)
(Assignee)

Comment 6

5 years ago
:edmorley - one thought that i have is if stage and prod are sharing memcache keys? i see in the treestatus.wsgi file we define the memcached.servers, but don't see any memcache key config. is it possible both these environments are using the same keys? and if so, can we add a config option to prepend the keys with treestatus_<env> or something similar?
Flags: needinfo?(emorley)
That sounds plausible - I'd have to ask catlee to be sure, since I'm not familiar with anything other than the local dev treestatus setup.

(In reply to Chris Turra [:cturra] from comment #6)
> :edmorley - one thought that i have is if stage and prod are sharing
> memcache keys? i see in the treestatus.wsgi file we define the
> memcached.servers, but don't see any memcache key config. is it possible
> both these environments are using the same keys? and if so, can we add a
> config option to prepend the keys with treestatus_<env> or something similar?
Flags: needinfo?(emorley) → needinfo?(catlee)
The production and staging databases are not linked at all. It does sound like something weird was happening in your browser. :( Sorry I'm not more helpful, but prod and dev/stage are definitely not related at all.
Thank you :-)
Flags: needinfo?(scabral)
The memcached keys do support prefixing, but it's not exposed in a useful way right now. I assumed the memcached servers for dev/stage/prod were all separate.

It shouldn't be too hard to support a new config value in the .wsgi that gets passed down to modify memcachePrefix.

https://github.com/mozilla/treestatus/blob/master/treestatus/app.py#L27
Flags: needinfo?(catlee)
(Assignee)

Comment 11

5 years ago
do we want to wait on a patch so the memcache key can be configured, or continue onto production with this deployment?
Flags: needinfo?(emorley)
Lets continue please - I've tested locally & the bit of testing on staging not interfered with by the memcache issue seems fine. That plus no schema changes means a rollback would be easy (but I doubt it would even come to that).
Flags: needinfo?(emorley)
(Assignee)

Comment 13

5 years ago
roger that. i have completed the push to prod as requested.

[root@genericadm.private.phx1 treestatus.mozilla.org]# bash -x update
+ CODE_DIR=/data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ VENDOR_DIR=/data/genericrhel6/src/treestatus.mozilla.org/tree-status/vendor
+ update_code
+ echo -e 'Updating code...'
Updating code...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ git pull -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule sync -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ git submodule update --init --recursive -q
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Updating vendor submodules...'
Updating vendor submodules...
+ git submodule update --init --recursive
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Cleaning gitignore and pyc files...'
Cleaning gitignore and pyc files...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ find . -type f -name .gitignore -or -name '*.pyc' -delete
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ echo -e 'Storing revision information...'
Storing revision information...
+ cd /data/genericrhel6/src/treestatus.mozilla.org/tree-status
+ git rev-parse HEAD
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ touch /data/genericrhel6/src/treestatus.mozilla.org/tree-status/treestatus.wsgi
+ echo -e 'Deploying code...'
Deploying code...
+ '[' xterm == dumb ']'
+ /data/genericrhel6/deploy treestatus.mozilla.org
[2013-06-05 14:04:31] Running rsync_project
[2013-06-05 14:04:31] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/treestatus.mozilla.org/ /data/genericrhel6/www/treestatus.mozilla.org/
[2013-06-05 14:04:32] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/treestatus.mozilla.org/ /data/genericrhel6/www/treestatus.mozilla.org/ (1.181s)
[2013-06-05 14:04:32] Finished rsync_project (1.182s)
[2013-06-05 14:04:32] Running commit_www
[2013-06-05 14:04:32] [localhost] running: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.mozilla.org']'
[2013-06-05 14:04:38] [localhost] finished: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['treestatus.mozilla.org']' (5.293s)
[localhost] out: [master eb5fc39] deploy [treestatus.mozilla.org]
[localhost] out: 5 files changed, 127 insertions(+), 39 deletions(-)
[2013-06-05 14:04:38] Finished commit_www (5.293s)
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Thank you - looks good.

Will file a separate bug to configure the memcache key.
(In reply to Ed Morley [:edmorley UTC+1] from comment #14)
> Will file a separate bug to configure the memcache key.

Bug 880173
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.