1736466 - Support GCLB custom TLS headers in the data pipeline

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P2)

Description

[Wesley Dawson [:whd]]

Our TAM informed us that https://cloud.google.com/load-balancing/docs/https/custom-headers can be used to pass some information from GCLB to our backend application. This is presently the only way to get access to some information that will be useful for us to monitor the Fx 95 rollout of bug #1734262, and to have a better general understanding of the TLS spread from clients.

See https://cloud.google.com/load-balancing/docs/https/custom-headers#variables

For specifically TLS monitoring, the following values are most interesting:
tls_version
tls_cipher_suite

Some other potentially interesting headers:
tls_sni_hostname: an additional bit of information to help us weed out really old or otherwise misconfigured clients (e.g. clients not hitting incoming.tmo but one of the older names like data.m.c)
client_*: we use maxmind for geoip but GCP provides similar information
origin_request_header: potentially useful for glean-js based submissions, see bug #1676676

My tentative proposal is to add two new headers:

X-TLS-Version: maps to tls_version
X-TLS-Cipher-Suite: maps to tls_cipher_suite

Note that apart from X-User-IP, CDN-Loop or any header starting with X-Google, X-Goog-, X-GFE or X-Amz-, we control the header mapping and could even e.g. combine the two values into into a single header if we wanted.

It might be worth adding a prefix e.g. X-[GC]LB-TLS-Version to indicate that this is not a client-header and is generated by the load balancer and not the application.

IIRC the deployment of a new header usually involves two PRs, similar to the following:

https://github.com/mozilla/gcp-ingestion/pull/1699/files
https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/688

And deploys that can happen in any order:
MPS deploy
edge, decoder, and sink deploys

In addition, the configuration for adding these headers needs to be added to our GCLBs. Since we don't generate these resources via GCP directly we will need to modify our backendconfig to add these headers and test that in stage.

EDIT: Perhaps worthy of consideration is X-Source-Tags style header to avoid header proliferation. Unfortunately it would be difficult to combine gclb-generated tags with client-generated tags so a custom header would likely be required. But perhaps just the one e.g. X-[GC]LB-Tags.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Given that there's not a real existing standard for how to encode this, consolidating to a single tags-style header seems reasonable, and we can modify the format in the future. I'm tempted to make this a key=value style unlike X-Source-Tags which is just values, but that's probably overkill. It's probably fine for order to be significant here.

So my preference would be:

--custom-request-header='X-LB-Tags:{tls_version}, {tls_cipher_suite}'

Example value:

TLSv1.3, 009C

And this could potentially be extended to something like:

TLSv1.3, 009C, incoming.telemetry.mozilla.org

I don't know much about origin request header, but seems like that should be split to a separate value if we need to hold it.

Comment 2

[Wesley Dawson [:whd]]

Some thoughts here on ergonomics. I think it makes sense to update global metadata normalization to parse X-LB-Tags similarly to X-Source-Tags. It's an open question as to whether to make the result of that normalization more struct-like as tag order is significant.

The encoding of cipher suites GCLB uses comes from https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4, which includes a (newline+carriage-return delimited) CSV. This can be loaded directly into BQ, so I'm thinking we should import this into static directly for joining with views on top of the parsed header metadata. It might make sense to have the metadata normalization udf be directly responsible for turning e.g. 009C into TLS_RSA_WITH_AES_128_GCM_SHA256 using the static dataset. Recommended is potentially an interesting field the RFC CSV provides as we could monitor for usage of cipher suites that are no longer recommended.

It's also possible we'll only want to use this with specific monitoring views on pbd and avoid updating the general metadata normalization logic. For now I will work on MPS, gcp-ingestion, and cloudops-infra PRs for adding the tag and a bq-etl PR for ergonomics can come later.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/706

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1880

Comment 5

[Wesley Dawson [:whd]]

I deployed the ingestion changes today (though the prod edge will take a few hours to propagate) and see data in PBD and live tables, so starting UTC 10th or 11th this data should be relatively complete.

One thing to note: views deploys showed a lot of output such as:

Could not update field descriptions for moz-fx-data-shared-prod.org_mozilla_mozregression.baseline_clients_daily: /tmp/sql/moz-fx-data-shared-prod/org_mozilla_mozregression/baseline_clients_daily/schema.yaml is not a valid YAML schema file.

I believe this will be auto-fixed when next bqetl updates generated-sql to include the new header.

I'm going to split out analysis ergonomics (comment #2) into a separate bug.