We would like to consolidate download-stats.m.o into our normal ingestion infrastructure. The DSMO service is currently a one-off that uses a combination of hindsight and heka, because at the time we set it up hindsight did not have all the functionality we needed (and neither did heka). DSMO uses GET requests where the URL contains all the information we care about for downstream parsing. Therefore, it would be nice to use nginx-moz-ingest with a location-specific directive allowing GETs, so we get the usual landfill and kafka-forwarding capabilities. We don't care about URL args (since DSMO doesn't use them), so the current behavior of parsing them into their own field(s) would be fine. As there won't be a POST body, the content field could be elided. Now that hindsight has a tail output, the other option is to use that to read the nginx logs on the edge and do the landfill and kafka forwarding, which is similar to what we are doing now (but on the normal ingestion infra). However,it seems silly to run hindsight and nginx-moz-ingest in essentially the same capacity. I think adding an option to support GET is a good solution.
Component: Metrics: Pipeline → Pipeline Ingestion
Product: Cloud Services → Data Platform and Tools
You need to log in before you can comment on or make changes to this bug.