Test local UDP transfers when bigger payloads are sent by Telemetry Experiments



Content Services Graveyard
Tiles: Data Processing
3 years ago
3 years ago


(Reporter: oyiptong, Assigned: kthiessen)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: .002a)

User Story

Tiles webheads need to be tested to make sure there is no loss of data due to the increased payload sizes.

We use UDP as the transport to rsyslog within the same machine. There are potential issues of message sizes growing beyond buffer sizes, therefore, this should be tested

I will write a script to help kthiessen test
Comment hidden (empty)


3 years ago
User Story: (updated)

Comment 1

3 years ago
Code to generate data in review: https://github.com/oyiptong/onyx/pull/12

kthiessen: to run this, you'd type:

ONYX_SETTINGS=onyx_config.OnyxConfig manage.py log flood_user_event 1000 -s 10

1000 here stands for the number of messages
the parameter to -s is the number of kilobytes one message should be

Comment 2

3 years ago
will update when the code is deployed in staging
The code is deployed in staging. The staging box is at: ec2-54-172-198-19.compute-1.amazonaws.com. 

How do we run manage.py from the stage environment? 

I tried: 

> cd /data/onyx
> ONYX_SETTINGS=onyx_config.OnyxConfig ./bin/python scripts/manage.py log flood_user_event 1 -s 10
Traceback (most recent call last):
  File "scripts/manage.py", line 3, in <module>
    from onyx.utils import environment_manager_create, GunicornServerCommand, LogCommand
ImportError: No module named onyx.utils

Comment 4

3 years ago
The correct command to execute it is:

> $ ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10

To make that work in the current environment, onyx had to be installed in the virtualenv's site-packages.

To do so:

> cd /data/onyx
> sudo ./bin/python setup.py install

Comment 5

3 years ago
Actually, that won't work.

This will:

> $ PYTHONPATH=/data/onyx ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10

This can run from any cwd
Conclusions from testing: 

Max safe message size is 3KB. When message payloads are larger we start seeing message drops. Message loss starts happening around 4KB. 

We generated 20,000 3KB messages and got 100% delivery. The delivery rate was ~330 messages/second. The rate was limited by CPU (c3.large). 

As long as the new telemetry messages are <3KB we should see no affect on message delivery at the onyx layer.
Blocks: 1113256
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.