Consider adding location-specific directive for allowing the GET method to nginx-moz-ingest

NEW
Unassigned

Status

Data Platform and Tools
Pipeline Ingestion
P3
normal
10 months ago
9 months ago

People

(Reporter: whd, Unassigned)

Tracking

(Blocks: 1 bug)

Details

(Reporter)

Description

10 months ago
We would like to consolidate download-stats.m.o into our normal ingestion infrastructure. The DSMO service is currently a one-off that uses a combination of hindsight and heka, because at the time we set it up hindsight did not have all the functionality we needed (and neither did heka).

DSMO uses GET requests where the URL contains all the information we care about for downstream parsing. Therefore, it would be nice to use nginx-moz-ingest with a location-specific directive allowing GETs, so we get the usual landfill and kafka-forwarding capabilities. We don't care about URL args (since DSMO doesn't use them), so the current behavior of parsing them into their own field(s) would be fine. As there won't be a POST body, the content field could be elided.

Now that hindsight has a tail output, the other option is to use that to read the nginx logs on the edge and do the landfill and kafka forwarding, which is similar to what we are doing now (but on the normal ingestion infra). However,it seems silly to run hindsight and nginx-moz-ingest in essentially the same capacity. I think adding an option to support GET is a good solution.
(Reporter)

Updated

10 months ago
Blocks: 1357257

Updated

10 months ago
Component: Metrics: Pipeline → Pipeline Ingestion
Product: Cloud Services → Data Platform and Tools

Updated

9 months ago
Points: --- → 2
You need to log in before you can comment on or make changes to this bug.