1054452 - Datazilla Repository Update

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Jonathan Eads ( :jeads )]

A new datazilla repository is ready to push to stage. One of the main things included in this release is  fix for a data corruption issue resulting from overloading the device type with memory information in a single property in the json sent to datazilla. This release requires running a specialized manage.py command to reset some corrupted machine/test relationships.

Once the repository is updated on stage, please run the following manage.py commands:

python manage.py fix_b2g_machines --project b2g --min_id 354404 >b2g_fix_out.txt

python manage.py fix_b2g_machines --project b2gtw --min_id 354404 >b2gtw_fix_out.txt

Note: fix_b2g_machines runs an ALTER statement modifying the UNIQUE index on the b2g_perftest_1.machine and b2gtw_perftest_1.machine tables so the user running the command will need to have sufficient DB privileges.

The first command will run for awhile, the second will be quick. If everything goes as planned we won't need the output files but if something goes wrong, they will give me some ideas as to what is going on.

This release includes the following changes:

1.) Addresses Bug 1046056, Bug 1050179, and Bug 1047324

The manage.py command fix_b2g_machines iterates through raw json objects stored in b2g_perftest_1/b2gtw_perftest_1 and builds an index in memory using the machine name (mac address), operating system id, and device type as a unique key. It then compares this index to the one in the machine table, inserting missing combinations, and reusing the correct ones that have been inserted to re-associate the correct machine_id to the corresponding test_run.id using the new version of the machine table.

2.) Addresses Bug 1007774

Adds a target line to the main graph for device/test combination listed in https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/config.json#L14. This is done dynamically by ingesting the config.json file and matching device/test names.

3.) Addresses Bug 1011577 

Application load defaults are now set to:

Test: startup_>_moz-app-visually-complete
Time Range: 7 Days
Gaia Branch: master
Device: flame-319MB

4.) Addresses Bug 1047306

Added ingestion alerts for flame-319MB.

Comment 1

[Chris Turra [:cturra]]

Attachment: datazilla_fix_out.tgz

i have updated stage from the repo and attached you will find the output from both these manage.py commands.

Comment 2

[Jonathan Eads ( :jeads )]

Looks good! The manage command looks like it resolved the data integrity issue. Please push this to production, there may be some follow up changes next week but we can address those in a different bug. Thanks for the quick turn around.

Comment 3

[Chris Turra [:cturra]]

great! i have gone ahead and pushed these changes out to prod as requested and will mark this bug as r/fixed.