2048276 - Schedule BHR aggregation on CI/TaskCluster

Status: RESOLVED FIXED

(Core :: XPCOM, enhancement)

Description

[Chutian (Sky) Ning]

With the new BHR aggregation living in Firefox codebase instead of externally on python_mozetl and the perfteam taking over the ownership now, we should be able to replace the daily job on AirFlow with TaskCluster.
This allows much faster future iteration and feature enhancement.

Comment 1

[Chutian (Sky) Ning]

Design update after security review with :sparky (Perf), :jlorenzo and
:hgueniot (RelEng/Firefox-CI), and :benwu (Data Eng).

Scope change: no try access

The original plan made the task try-runnable for iteration. That is dropped.
Try is level 1 and runs contributor code, so it must never hold a production
BigQuery credential. Production data access is limited to the reviewed level-3
cron on mozilla-central. Developers who need real data locally use their own
GCP credentials via ./mach bhr-aggregate.

Credential

The task reads a GCP service-account key from the dedicated TaskCluster secret
project/bhr/aggregation-gcp-key (bug 2056977), rather than reusing the shared
perftest namespace. RelEng granted secrets:get:project/bhr/* to
cron:bhr-aggregate (trust domain gecko, level 3, mozilla-central) in
fxci-config PR #1097.

The secret name is a fixed constant with no {level} templating on either the
kind's scope or the script's default. Previously both interpolated
MOZ_SCM_LEVEL, so a level-1 run resolved to a real level-1 secret name. Now no
level-1 name exists and anything other than the level-3 cron fails closed on a
scope it cannot hold.

Keyless auth (Workload Identity Federation) was investigated first and is
preferred, but RelEng confirmed there is no supported path for a Taskcluster
task to obtain short-lived GCP credentials today. A stored key scoped to
level 3 was accepted as the fallback.

BigQuery access

Service account: [e-mail]

Grants verified directly as the service account on 2026-07-28:

mozdata.firefox_desktop.hang_report readable
moz-fx-data-shared-prod.firefox_desktop_stable.hang_report_v1 denied
mozdata.firefox_desktop.hang_report_redacted denied (not created)
mozdata:tmp write OK

Note that a missing submission_timestamp filter returns HTTP 400, not 403,
which reads like a denial without being one.

Verified end to end

Ran the full aggregation authenticated as the service account (not developer
credentials) for build date 2026-04-01 at sample size 0.01: 5686 pings after
filtering, 16363 hangs, 1428 modules symbolicated, 38098 frames resolved, 8261
signatures, artifact written. Row counts were identical to the same run through
the base table under developer credentials, confirming the view is a faithful
projection for every column the job reads.

Untested surface is now limited to TaskCluster plumbing: secret fetch through
taskcluster-proxy, the in-tree docker image, artifact upload, and the index
route.

Landing order

_BQ_TABLE must be mozdata.firefox_desktop.hang_report until DENG-11394 lands,
then mozdata.firefox_desktop.hang_report_redacted. Bug 2052962 and bug 2056370
currently point at firefox_desktop_stable.hang_report_v1, which the service
account cannot read; those must be corrected before either lands or the first
cron run will 403.

Comment 2

[Chutian (Sky) Ning]

Attachment: Bug 2048276 - Schedule BHR aggregation daily on TaskCluster r=dthayer

Add a cron-scheduled TaskCluster job that runs the BHR aggregation in a
custom Docker image. The container authenticates to BigQuery headlessly
with a GCP service-account JSON pulled from a TaskCluster secret via the
taskcluster-proxy (no interactive gcloud login), and publishes the
dashboard JSON as a public artifact indexed at
gecko.v2.<project>.latest.firefox.bhr-aggregate.

Pieces:

• taskcluster/scripts/bhr-aggregate.py: container entry point
• taskcluster/docker/bhr-aggregate/: image + python deps
• taskcluster/kinds/bhr-aggregate/kind.yml: the task
• docker-image + target_tasks + .cron.yml: registration and daily schedule
• build/sparse-profiles/bhr-aggregate: minimal sparse checkout

The secret is project/bhr/aggregation-gcp-key, created for this job in
bug 2056977. RelEng granted secrets:get:project/bhr/* to cron:bhr-aggregate
(trust domain gecko, level 3, mozilla-central) in fxci-config PR #1097.

The secret name is a fixed constant on both the kind's scope and the
script's default, deliberately not templated on MOZ_SCM_LEVEL. No level-1
secret name exists, so anything other than the level-3 mozilla-central
cron fails closed on a scope it cannot hold rather than resolving to some
other secret. This job must never be reachable from try: try is level 1
and runs contributor code, and the credential reads production data.

Comment 3

[Chutian (Sky) Ning]

Attachment: Bug 2048276 - Read BHR pings from the firefox_desktop.hang_report view r=dthayer

Data Engineering manages BigQuery access for the CI service account through
the firefox_desktop.hang_report view rather than the underlying
firefox_desktop_stable.hang_report_v1 table. The view has the same schema, so
the query is unchanged.

Comment 4

[Chutian (Sky) Ning]

Attachment: Bug 2048276 - Request an explicit scope when checking GCP credentials r=dthayer

google.auth.default() returns a service-account key unscoped, and
refreshing an unscoped service account raises RefreshError, so the
preflight check reported "credentials are missing or expired" for a
credential that was in fact valid. Only user credentials from
gcloud auth application-default login worked, because those arrive
already scoped.

This matters for testing the TaskCluster cron: pointing
GOOGLE_APPLICATION_CREDENTIALS at the job's service-account key is how
you verify the credential the cron will actually use, and that path was
unreachable. The error message also pointed at
gcloud auth application-default login, which would overwrite the
developer's own credentials without addressing the problem.

Only the preflight was affected; bigquery.Client() resolves its own
scopes, so the aggregation itself always worked.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/457b915ba271
https://hg.mozilla.org/integration/autoland/rev/0bb84a194bd7
Schedule BHR aggregation daily on TaskCluster r=dthayer
https://github.com/mozilla-firefox/firefox/commit/20a50d10332d
https://hg.mozilla.org/integration/autoland/rev/58f1ca55ddd1
Read BHR pings from the firefox_desktop.hang_report view r=dthayer
https://github.com/mozilla-firefox/firefox/commit/970f45000d5d
https://hg.mozilla.org/integration/autoland/rev/8a62beb9089e
Request an explicit scope when checking GCP credentials r=dthayer

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0bb84a194bd7
https://hg.mozilla.org/mozilla-central/rev/58f1ca55ddd1
https://hg.mozilla.org/mozilla-central/rev/8a62beb9089e