625209 - watch hourly processing rate as smoke testing and transition of PHX data center rolls out.

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[chris hofmann]

as an end to end test we need to keep an eye on how PHX and changes in architecture are keeping up with current loads.

I'll attach an hourly profile of processing volume for jan1-jan11 to help understanding what the current flow looks like.  

This also can be the bug to make sure we have tools ready to watch end-to-end processing rates down to the hour or minute so we can stay on top of any slowdowns that might show up.

Comment 1

[chris hofmann]

Attachment: hourly processed rate generated from the .csv files showing flow over 24 hour cycles.

Looks like we are hitting peaks of just over 18,000 per hour, and valleys of just under 8,000 per hour in the normal incoming flow over the last several days.

Comment 2

[chris hofmann]

Looks like the hourly profile for the last day on SJC looked like this:

with 8-18k reports processed per hour.

./hourly-volume.sh 20110121*
date_proce 1
2011012100 10339
2011012101 10961
2011012102 11744
2011012103 12922
2011012104 14570
2011012105 16006
2011012106 17328
2011012107 17926
2011012108 18718
2011012109 18429
2011012110 18224
2011012111 17432
2011012112 16336
2011012113 15753
2011012114 13422
2011012115 11826
2011012116 10513
2011012117 9974
2011012118 9265
2011012119 8945
2011012120 8489
2011012121 8280
2011012122 8297
2011012123 9129

on the first day of PHX it looks like peak rate of 14-15k was all that we achieved.  Maybe volume was lower because of it being Saturday, but we should have seen higher hourly rates due to the creation of a backlog.

 ./hourly-volume.sh 20110122*
date_proce 1
2011012200 10117
2011012201 11400
2011012202 12359
2011012203 13320
2011012204 14476
2011012205 6739
2011012206 8280
2011012207 9226
2011012209 10584
2011012210 1936
2011012211 9830
2011012212 15270
2011012213 14215
2011012214 12888
2011012215 11498
2011012216 10063
2011012217 9516
2011012218 8880
2011012219 8637
2011012220 8143
2011012221 7757
2011012222 7812
2011012223 8493

Comment 3

[chris hofmann]

back up to around 17,500 reports per hour between 9a-11a yesterday.

./hourly-volume.sh 20110123*
date_proce 1
2011012300 9657
2011012301 11283
2011012302 12577
2011012303 13011
2011012304 14162
2011012305 15405
2011012306 16138
2011012307 16553
2011012308 15935
2011012309 17443
2011012310 17594
2011012311 17437
2011012312 16596
2011012313 14750
2011012314 12732
2011012315 10742
2011012316 10189
2011012317 9454
2011012318 9251
2011012319 8602
2011012320 8304
2011012321 8127
2011012322 8488
2011012323 9413