1291340 - Port Sync server metrics to production

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Danny Coates [:dcoates]]

Sync metrics[1] are currently running in the dev environment (cloudservices-aws-dev) using my one-off method[2]

This bug tracks the work required to transition these metrics to production.


[1] https://sql.telemetry.mozilla.org/dashboard/sync
[2] https://github.com/dannycoates/smt

Comment 1

[Mark Reid [:mreid]]

Per a couple of discussions this week, we need to either run the conversion + export code in the prod IAM, or store a pii-scrubbed version of the data that we can access from the dev IAM (specifically from analysis.telemetry.mozilla.org- and airflow-launched instances).

I would prefer to be able to run the import / rollup code via airflow, but if it's significantly easier to run it in prod, let's do that.

Wesley, what do you think?

Comment 2

[Wesley Dawson [:whd]]

As discussed in IRC/vidyo, if the analysis doesn't need access to the PII fields I can refactor https://github.com/mozilla-services/puppet-config/pull/2031/files to facilitate this.

Comment 3

[Mark Reid [:mreid]]

Danny, can you confirm that we don't need access to any PII fields?

Comment 4

[Danny Coates [:dcoates]]

Here are the fields we need:

  uid CHAR(32) NOT NULL encode lzo,    -- a sha256-hashed Firefox Account (FxA) user id
  s_uid CHAR(32) encode lzo,           -- a surrogate user id (generated by token server)
  dev CHAR(32) NOT NULL encode lzo,    -- a sha256-hashed FxA device id
  s_dev CHAR(32) encode lzo,           -- a surrogate device id (generated by token server)
  ts TIMESTAMP NOT NULL encode lzo,    -- timestamp of the request
  method VARCHAR(32) encode lzo,       -- request method (GET, POST, etc)
  code SMALLINT encode lzo,            -- http status code of the response
  bucket VARCHAR(255) encode bytedict, -- sync bucket name (bookmarks, history, etc)
  t INTEGER encode bytedict,           -- request time in milliseconds
  ua_browser VARCHAR(255) encode lzo,  -- request User Agent browser
  ua_version INTEGER encode lzo,       -- request User Agent browser version
  ua_os VARCHAR(255) encode lzo,       -- request User Agent OS
  host VARCHAR(255) encode lzo         -- server hostname that handled the request

None of those are immediately PII, though with enough access (to our internal production systems) one could probably decipher what the actual fxa uid and dev are.

Comment 5

[Mark Reid [:mreid]]

@whd, does this list look ok?

Comment 6

[Wesley Dawson [:whd]]

(In reply to Mark Reid [:mreid] from comment #5)
> @whd, does this list look ok?

To me, yes.

Comment 7

[Wesley Dawson [:whd]]

Taking this to make the data available to ATMO, whereupon :mreid will take over.

Comment 8

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/puppet-config/pull/2263

Data should be available in s3://net-mozaws-prod-us-west-2-pipeline-analysis/sync-metrics/data, which should be accessible from ATMO etc. Back to :mreid.

Comment 9

[Alex Davis [:adavis]]

Just checking in to see what's left to get this in production. A lot of work was put into this up until now and I feel like we're really close to the finish line.

Until it lands in prod, all of Danny's beautiful dashboards are reporting back inaccurate numbers because data is struggling to refresh in the dev environment. https://sql.telemetry.mozilla.org/dashboard/sync

Thanks in advance for updating me on this status.

Comment 10

[Mark Reid [:mreid]]

I'm back on this as my #1 item this week. I should have something preliminary in the next few days.

Comment 11

[Alex Davis [:adavis]]

(In reply to Mark Reid [:mreid] from comment #10)
> I'm back on this as my #1 item this week. I should have something
> preliminary in the next few days.

Great news! Thanks for update!

Comment 12

[Mark Reid [:mreid]]

Attachment: Notebook for importing sync log data & rollups to parquet form

r? @Danny for adherence to the previous conversion and rollup logic.
r? @Roberto for partitioning, Dataset mangling, and general Spark stuff (feel free to redirect to others).

Comment 13

[Mark Reid [:mreid]]

Attachment: Notebook for importing sync log data & rollups to parquet form

Comment 14

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mreid-moz:sync_bug1291340 > mozilla:master