1452166 - Create HTTP Edge node that responds with 400 on invalid payloads

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Frank Bertsch [:frank]]

Currently we respond with some pieces about improper messages on the production edge server, but if the JSON Validation (or something else) fails the client doesn't know.

This server wouldn't necessarily be hooked in to the rest of the pipeline (though, a CEP attached to it would be nice). It would run messages through validation, and let the client know if things were not validated.

Comment 1

[Mark Reid [:mreid]]

This seems like it's within the purview of the validation service :amiyaguchi is working on in bug 1454062. Anthony, does this seem like something that would fit that project, possibly in v2?

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

This definitely fits in the flavor of the validation service. One nice thing about the framework I'm using (flask) is that functionality can be modularized into packages called blueprints.[1] A flask blueprint for validating a single ping on-demand can be stood up separately and then integrated into the larger schema validation API. 

However, this may not be the right choice if the number of requests per second is above roughly 10k req/sec.[2] 


[1] http://flask.pocoo.org/docs/0.12/blueprints/
[2] https://medium.com/@tschundeee/express-vs-flask-vs-go-acc0879c2122

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

The main task of this service would be to map the incoming document and version to the schema in the mozilla-pipeline-schema. This logic will also need to be implemented in the v2 release of the validation CI service, so implementing and deploying this would cover a lot of ground.

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

A work in progress exists at [1]. I've manually tested processed landfill pings from several doctypes with the correct response using schemas from the mozilla-pipeline-schemas HEAD. I will include scripts for syncing data and integration tests for deployment.

uWSGI with nginx is potential deployment option. 


[1] https://github.com/acmiyaguchi/edge-validator

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

Dependency for testing data.

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: v1 release #1

I've set up an endpoint that should 200 or 400. It implements the proposed generic ingestion spec (except for the optional docid that doesn't make sense in this context).[1]

The schemas repo is a submodule within the project, so the submodule can be updated using git commands to update the schemas in the endpoint.


:frank I've tagged you as a reviewer, let me know if this works.

[1] https://docs.google.com/document/d/1PqiF1rF2fCk_kQuGSwGwildDf4Crg9MJTY44E6N5DSk/edit#heading=h.74qlucdvwdg0

Comment 7

[Frank Bertsch [:frank]]

Anthony, next steps here would be:

1. Have somebody try this locally with the docker setup you have and a locally-built Fx instance
2. Deploy this to incoming-test.telemetry.mozilla.org (or alternate)

Does that sound right to you?

Comment 8

[Anthony Miyaguchi [:amiyaguchi]]

Those sounds like good next steps. I've filed a bug with ops for (2) in bug 1462048, which should hopefully be a straightforward process.

Comment 9

[Anthony Miyaguchi [:amiyaguchi]]

This is ready-to-use locally. See the follow-up bugs for deployment.