1048512 - msisdn-gateway load test is failing with JSON errors and 404s/499s

Status: VERIFIED FIXED

(Cloud Services Graveyard :: MobileID, defect)

Description

[James Bonacci [:jbonacci]]

Trying to run a simple 'make test' against Stage, I am seeing this:

Failures: 1

1 occurrences of: 
    AssertionError: No JSON has been returned:     Traceback: 
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "loadtest.py", line 35, in test_all
    self.register()
  File "loadtest.py", line 101, in register
    self.fail("No JSON has been returned: %s" % resp.content)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)


Slowest URL: https://msisdn.stage.mozaws.net/register 	Average Request Time: 60.029453

Stats by URLs:
- https://msisdn.stage.mozaws.net/register	Average request time: 60.029453	Hits success rate: 0.0
- https://msisdn.stage.mozaws.net/discover	Average request time: 0.819449	Hits success rate: 1.0

Custom metrics:
- register : 1
make: *** [test] Error 1



Checking the nginx access log, I see, for each test, a 200 followed by either a 404 or a 499


The Stage server and the Mock server appear healthy to me.

Comment 1

[James Bonacci [:jbonacci]]

Ah, ok, so in the msisdn-gateway_err.log file, I get traceback:

...etc...
events.js:72
        throw er; // Unhandled 'error' event
              ^
TypeError: Cannot call method 'toString' of undefined
    at SecretBox.self.decrypt (/data/msisdn-gateway/node_modules/sodium/lib/secretbox.js:149:30)
    at Object.decrypt (/data/msisdn-gateway/msisdn-gateway/encrypt.js:40:14)
    at /data/msisdn-gateway/msisdn-gateway/index.js:606:34
    at /data/msisdn-gateway/msisdn-gateway/storage/redis.js:211:13
    at try_callback (/data/msisdn-gateway/node_modules/redis/index.js:573:9)
    at RedisClient.return_reply (/data/msisdn-gateway/node_modules/redis/index.js:661:13)
    at HiredisReplyParser.<anonymous> (/data/msisdn-gateway/node_modules/redis/index.js:309:14)
    at HiredisReplyParser.emit (events.js:95:17)
    at HiredisReplyParser.execute (/data/msisdn-gateway/node_modules/redis/lib/parser/hiredis.js:43:18)
    at RedisClient.on_data (/data/msisdn-gateway/node_modules/redis/index.js:534:27)
    at Socket.<anonymous> (/data/msisdn-gateway/node_modules/redis/index.js:91:14)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:748:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:410:10)
    at emitReadable (_stream_readable.js:406:5)
    at readableAddChunk (_stream_readable.js:168:9)
    at Socket.Readable.push (_stream_readable.js:130:10)
    at TCP.onread (net.js:528:21)

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: Redis connection to msi-ca-qqwbm7dres8z.jk7o1z.0001.use1.cache.amazonaws.com:6379 failed - read ETIMEDOUT
    at RedisClient.on_error (/data/msisdn-gateway/node_modules/redis/index.js:185:24)
    at Socket.<anonymous> (/data/msisdn-gateway/node_modules/redis/index.js:95:14)
    at Socket.emit (events.js:95:17)
    at net.js:440:14
    at process._tickCallback (node.js:419:13)

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: Redis connection to msi-ca-qqwbm7dres8z.jk7o1z.0001.use1.cache.amazonaws.com:6379 failed - read ETIMEDOUT
    at RedisClient.on_error (/data/msisdn-gateway/node_modules/redis/index.js:185:24)
    at Socket.<anonymous> (/data/msisdn-gateway/node_modules/redis/index.js:95:14)
    at Socket.emit (events.js:95:17)
    at net.js:440:14
    at process._tickCallback (node.js:419:13)