1527906 - Deploy Webpagetest schema

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement)

Description

[Arkadiusz Komarzewski [:akomar]]

ref: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/216

Comment 1

[Wesley Dawson [:whd]]

I've deployed this but no pings are making it through:

https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.moz_generic_error_monitor.webpagetest.html
https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.moz_generic_webpagetest_webpagetest_run_1_pings.errors.txt

The error is not particularly helpful (:trink has in-flight work for making the error monitor more useful), but I dug up an example ping from landfill which makes the issue pretty clear:

name: uri type: 0 representation: <nil> value: /submit/webpagetest/webpagetest-run/1/UUID
name: content type: 1 representation: <nil> value: appName=chrome&channel=canary&[...]

Namely, the submission is not JSON and appears instead to be encoded as query parameters.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks for deploying, Wes! Sorry 'bout the errors.

I'll take a look, but would appreciate any and all pointers/PRs/gists; the code is https://github.com/mozilla/wpt-api/blob/6a035b1b69426164dff40c01305a8d2219586787/send_to_telemetry.py.

Comment 3

[Wesley Dawson [:whd]]

(In reply to Stephen Donner [:stephend] from comment #2)

Thanks for deploying, Wes! Sorry 'bout the errors.

I'll take a look, but would appreciate any and all pointers/PRs/gists; the code is https://github.com/mozilla/wpt-api/blob/6a035b1b69426164dff40c01305a8d2219586787/send_to_telemetry.py.

https://github.com/mozilla/wpt-api/blob/6a035b1b69426164dff40c01305a8d2219586787/send_to_telemetry.py#L84

You probably want data=json.dumps(asdict(result)) or similar.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Sorry, all - this should now be fixed, in: https://github.com/mozilla/wpt-api/commit/e524adda32cb731a5b1ad6282ee0b8806894adee

Wesley/Akomar, can either of you confirm?

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screen Shot 2019-02-18 at 11.47.07 AM.png

Looks like it's now ingesting?

Comment 6

[Wesley Dawson [:whd]]

Also see: https://sql.telemetry.mozilla.org/queries/61530/source

Comment 7

[Arkadiusz Komarzewski [:akomar]]

I made an error in Parquet schema resulting in null metric values in the output, we'll need to redeploy once this is merged: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/264

Comment 8

[Arkadiusz Komarzewski [:akomar]]

:whd - can you redeploy updated schema[1]?

[1] https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/264

Comment 9

[Wesley Dawson [:whd]]

(In reply to Arkadiusz Komarzewski [:akomar] from comment #8)

:whd - can you redeploy updated schema[1]?

[1] https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/264

Done. Please close this when you've verified the fix.

Comment 10

[Wesley Dawson [:whd]]

Er, keeping open until the fix has been verified by someone other than me.

Comment 11

[Arkadiusz Komarzewski [:akomar]]

Looks good so far.

:whd - before we close this, can you delete all the data from this dataset where submission_date_s3<='20190219'? Schema changed a little after the last fix and this would make consumers' life easier.

Comment 12

[Wesley Dawson [:whd]]

(In reply to Arkadiusz Komarzewski [:akomar] from comment #11)

:whd - before we close this, can you delete all the data from this dataset where submission_date_s3<='20190219'? Schema changed a little after the last fix and this would make consumers' life easier.

This has been done. Closing.