Closed Bug 1111733 Opened 10 years ago Closed 10 years ago

Test local UDP transfers when bigger payloads are sent by Telemetry Experiments

Categories

(Content Services Graveyard :: Tiles: Data Processing, defect)

Product:
Content Services Graveyard
Component:
Tiles: Data Processing
Type:
defect
Priority:
Not set
Severity:
normal
Points:
3

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Iteration:
37.2

People

(Reporter: oyiptong, Assigned: kthiessen)

References

Details

(Whiteboard: .002a)

User Story

Tiles webheads need to be tested to make sure there is no loss of data due to the increased payload sizes.

We use UDP as the transport to rsyslog within the same machine. There are potential issues of message sizes growing beyond buffer sizes, therefore, this should be tested

I will write a script to help kthiessen test
      No description provided.
User Story: (updated)
Code to generate data in review: https://github.com/oyiptong/onyx/pull/12

kthiessen: to run this, you'd type:

ONYX_SETTINGS=onyx_config.OnyxConfig manage.py log flood_user_event 1000 -s 10

1000 here stands for the number of messages
the parameter to -s is the number of kilobytes one message should be
will update when the code is deployed in staging
The code is deployed in staging. The staging box is at: ec2-54-172-198-19.compute-1.amazonaws.com. 

How do we run manage.py from the stage environment? 

I tried: 

> cd /data/onyx
> ONYX_SETTINGS=onyx_config.OnyxConfig ./bin/python scripts/manage.py log flood_user_event 1 -s 10
Traceback (most recent call last):
  File "scripts/manage.py", line 3, in <module>
    from onyx.utils import environment_manager_create, GunicornServerCommand, LogCommand
ImportError: No module named onyx.utils
The correct command to execute it is:

> $ ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10

To make that work in the current environment, onyx had to be installed in the virtualenv's site-packages.

To do so:

> cd /data/onyx
> sudo ./bin/python setup.py install
Actually, that won't work.

This will:

> $ PYTHONPATH=/data/onyx ONYX_SETTINGS=onyx_config.OnyxConfig /data/onyx/bin/manage.py log flood_user_event 1 -s 10

This can run from any cwd
Conclusions from testing: 

Max safe message size is 3KB. When message payloads are larger we start seeing message drops. Message loss starts happening around 4KB. 

We generated 20,000 3KB messages and got 100% delivery. The delivery rate was ~330 messages/second. The rate was limited by CPU (c3.large). 

As long as the new telemetry messages are <3KB we should see no affect on message delivery at the onyx layer.
Blocks: 1113256
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.