966080 - SUMO chief is returning an Internal Server Error

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Ricky Rosario [:rrosario, :r1cky]]

One more time.

+++ This bug was initially created as a clone of Bug #939797 +++

(Again)

+++ This bug was initially created as a clone of Bug #932380 +++

+++ This bug was initially created as a clone of Bug #925080 +++

+++ This bug was initially created as a clone of Bug #917357 +++

When trying to deploy, chief is failing me with an Internal Server Error.

http://supportadm.private.phx1.mozilla.com/chief/support.stage


I didn't try prod. Obviously, this blocks us from deploying.

Comment 1

[Chris Turra [:cturra]]

looks like the redis process had hung completely. the redis-memory check script in place could not check the state correctly because redis was running, but not responding. i went ahead and manually kicked the redis process, which should get you back up and running.


[[e-mail]1 ~]# bash -x /data/bin/redis-memory-check.sh
+ REDIS_CLI=/usr/bin/redis-cli
+ GREP=/bin/grep
+ DATE=/bin/date
+ LOGGER=/usr/bin/logger
+ KILL=/bin/kill
+ REDIS_SERVICE_NAME=/etc/init.d/redis-support-updater
+ MEM_MAX=99
++ redis-cli -p 6388 info
++ grep used_memory_human
++ awk -F : '{print $2}'
++ awk -F . '{print $1}'
Could not connect to Redis at 127.0.0.1:6388: Cannot assign requested address
+ MEM_USED=
++ redis-cli -p 6388 info
++ grep used_memory_human
++ awk -F : '{print $2}'
++ awk -F . '{print $2}'
++ tr -d '\r'
Could not connect to Redis at 127.0.0.1:6388: Cannot assign requested address
+ MEM_UNIT=
+ [[ '' = *M* ]]

[[e-mail]1 ~]# /etc/init.d/redis-support-updater status
redis-server (pid  28195) is running...

[[e-mail]1 ~]# /etc/init.d/redis-support-updater stop
Shutdown may take a while; redis needs to save the entire database
Shutting down redis-server: Could not connect to Redis at 127.0.0.1:6388: Cannot assign requested address
                                                           [FAILED]
[[e-mail]1 ~]# ps aux | pgrep redis
28195

[[e-mail]1 ~]# kill 28195

[[e-mail]1 ~]# /etc/init.d/redis-support-updater start
Starting redis-server:                                     [  OK  ]

[[e-mail]1 ~]# bash -x /data/bin/redis-memory-check.sh
+ REDIS_CLI=/usr/bin/redis-cli
+ GREP=/bin/grep
+ DATE=/bin/date
+ LOGGER=/usr/bin/logger
+ KILL=/bin/kill
+ REDIS_SERVICE_NAME=/etc/init.d/redis-support-updater
+ MEM_MAX=99
++ awk -F : '{print $2}'
++ awk -F . '{print $1}'
++ grep used_memory_human
++ redis-cli -p 6388 info
+ MEM_USED=826
++ awk -F : '{print $2}'
++ awk -F . '{print $2}'
++ tr -d '\r'
++ grep used_memory_human
++ redis-cli -p 6388 info
+ MEM_UNIT=78K
+ [[ 78K = *M* ]]

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

I was able to deploy now. Thanks cturra!!

Comment 3

[Chris Turra [:cturra]]

i actually found a bug in my redis-memory-check script that is likely the root cause here. it identified when the service needed a force restart, but couldn't do it b/c my variable was incorrect.

i just committed a fix for that which should resolve this permanently.

$ svn diff
sIndex: modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb
===================================================================
--- modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb	(revision 81626)
+++ modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb	(working copy)
@@ -11,7 +11,7 @@
 function force_restart {
   REDIS_PID=`pgrep redis-server`
   $KILL -9 $REDIS_PID
-  /etc/init.d/$REDIS_SERVICE_NAME start > /dev/null
+  $REDIS_SERVICE_NAME start > /dev/null
 }

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

Seems busto again :-/

Comment 5

[Chris Turra [:cturra]]

hah! i found another bug :( 

i had added some login a while ago to ensure that the memory check only looked when Mb's were used, not kb's. as it turns out, i applied that check /before/ looking to see if the service had packed it in. doh! this has now been committed to puppet and resolved on the sumo chief instance.

sorry for the delay in solving this completely!


$ svn diff
svn Index: modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb
===================================================================
--- modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb	(revision 81800)
+++ modules/webapp/templates/admin/data-bin/redis-memory-check.sh.erb	(working copy)
@@ -19,16 +19,16 @@
 MEM_USED=`redis-cli -p 6388 info | grep used_memory_human | awk -F ":" '{print $2}' | awk -F "." '{print $1}'`
 MEM_UNIT=`redis-cli -p 6388 info | grep used_memory_human | awk -F ":" '{print $2}' | awk -F "." '{print $2}' | tr -d '\r'`

+# if blank is returned for MEM_USED, redis has hung.
+if [ "$MEM_USED" == "" ]; then
+  $LOGGER -p user.notice "[redis-memory-checker] kill/start redis process b/c it has stopped responding."
+  force_restart
+  exit 1
+fi
+
 # check that Mb are consumed, not Kb
 if [[ $MEM_UNIT = *M* ]]; then

-  # if blank is returned for MEM_USED, redis has hung.
-  if [ "$MEM_USED" == "" ]; then
-    $LOGGER -p user.notice "[redis-memory-checker] kill/start redis process b/c it has stopped responding."
-    force_restart
-    exit 1
-  fi
-
   if [ $MEM_USED -gt $MEM_MAX ]; then