456529 - Create new cgi script to replace bulk.cgi & collect.cgi for new db schema

Status: RESOLVED FIXED

(Webtools Graveyard :: Graph Server, defect)

Description

[Ryan Doherty (:rdoherty)]

With the new db schema, we'll need to rewrite bulk.cgi and collect.cgi. While we're there, let's merge them.

Comment 2

[Mark Smith [:xb95]]

As discussed today:

These scripts have to update both the old database form and the new one for now, until we've figured out migration of everything to the new data.

In other words, Talos submits points of data, and we insert into both databases.

Comment 3

[alice nodelman [:alice] [:anode]]

Here's my first pass at covering both how Talos currently passes data and what needs to change to use the new schema:

https://wiki.mozilla.org/Perfomatic:SendingData

Comment 4

[alice nodelman [:alice] [:anode]]

Attachment: test tool for old collect.cgi/bulk.cgi and new collect.cgi

This tool sends mock data to a graph server - the data itself isn't very interesting but it is a quick way of checking to see if graph server collection scripts are operating correctly.

I'm not sure where this should be checked in, but it should be put into hg somewhere.  Also, post_file.py is checked in with the talos code (it is how talos sends data to the graph server) so it might be bad policy to check it in again?

Comment 5

[Ryan Doherty (:rdoherty)]

Comment on attachment 353303 [details]
test tool for old collect.cgi/bulk.cgi and new collect.cgi

I have no way to test this, but because it's a tool I'll r+ it and we can fix it if needed.

It should be checked in in /scripts, which is where utilities are put. I don't think it's a big deal to check in post_file.py twice.