Tiles webheads need to be tested to make sure there is no loss of data due to the increased payload sizes. We use UDP as the transport to rsyslog within the same machine. There are potential issues of message sizes growing beyond buffer sizes, therefore, this should be tested I will write a script to help kthiessen test
Code to generate data in review: https://github.com/oyiptong/onyx/pull/12 kthiessen: to run this, you'd type: ONYX_SETTINGS=onyx_config.OnyxConfig manage.py log flood_user_event 1000 -s 10 1000 here stands for the number of messages the parameter to -s is the number of kilobytes one message should be
will update when the code is deployed in staging
The code is deployed in staging. The staging box is at: ec2-54-172-198-19.compute-1.amazonaws.com. How do we run manage.py from the stage environment? I tried: > cd /data/onyx > ONYX_SETTINGS=onyx_config.OnyxConfig ./bin/python scripts/manage.py log flood_user_event 1 -s 10 Traceback (most recent call last): File "scripts/manage.py", line 3, in <module> from onyx.utils import environment_manager_create, GunicornServerCommand, LogCommand ImportError: No module named onyx.utils
The correct command to execute it is: > $ ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10 To make that work in the current environment, onyx had to be installed in the virtualenv's site-packages. To do so: > cd /data/onyx > sudo ./bin/python setup.py install
Actually, that won't work. This will: > $ PYTHONPATH=/data/onyx ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10 This can run from any cwd
Conclusions from testing: Max safe message size is 3KB. When message payloads are larger we start seeing message drops. Message loss starts happening around 4KB. We generated 20,000 3KB messages and got 100% delivery. The delivery rate was ~330 messages/second. The rate was limited by CPU (c3.large). As long as the new telemetry messages are <3KB we should see no affect on message delivery at the onyx layer.