1693305 - Request to provision Rally Study: News & Disinformation

Status: RESOLVED FIXED

(Firefox :: Pioneer, task)

Description

[Ted Han]

Hey folks,

We're going to re-launch the News & Disinformation study as the first study on Rally. We'd like to provision a new schema+env+keys just as if this were a new study.

This study's schemas should be identical to the study already running in about:ion, so let's just copy those over. https://github.com/mozilla-services/mozilla-pipeline-schemas/tree/master/schemas/pioneer-citp-news-disinfo

Likewise, the analysis env for the new study should have the same set of users as the previous set.

Comment 1

[Alessio Placitelli [:Dexter]]

Important note: while this is a "re-launch", this is a different add-on, but the same schemas apply, as Ted said.

Comment 2

[Wesley Dawson [:whd]]

I need some clarification WRT schemas and study names, as AFAIK no work except for bug #1691807 has been done to support Rally in our ingestion infrastructure. In particular, I haven't done any work such as routing based on the rally- prefix that is discussed in bug #1677567. I'm marking that bug as a blocker for provisioning this study, but if indeed we're going with e.g. pioneer_id in rally studies, this may be a simple operation.

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Wesley Dawson [:whd] from comment #2)

I need some clarification WRT schemas and study names, as AFAIK no work except for bug #1691807 has been done to support Rally in our ingestion infrastructure. In particular, I haven't done any work such as routing based on the rally- prefix that is discussed in bug #1677567. I'm marking that bug as a blocker for provisioning this study, but if indeed we're going with e.g. pioneer_id in rally studies, this may be a simple operation.

This does not require the work of bug 1677567. Rally, right now, is a bit of a conflated term for the product. For our data purposes, we have two major deadlines:

a) Rally v1: the whole data ingestion infrastructure is the same of Pioneer/Ion, which means that's using legacy telemetry under the hood. We don't require the "rally" prefix, we can have the "pioneer" prefix for this. Whatever works with the current infrastructure. We don't need changes with respect to what was already working in the previous study that was already deployed.
b) Beyond Rally v1: we'll integrate Glean.js and use that for submitting encrypted pings. We'll stop using legacy telemetry and use the rally prefix for data.

The current study will work with (A). Wesley, Anthony, does this clarify/make sense?

Comment 4

[Ted Han]

Hey Wesley,

Apologies i totally forgot about the preceding work outlined in the ticket you noted. For the sake of re-launching on rally, i'm not stressed about the schema name (although we should make certain to document where to find the schemas on the Rally end, and follow up on the work in the ticket).

If the new schema is called pioneer-citp-news-disinfo-2 that's fine for both my and CITP's purposes (or another name of your suggestion)

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

This makes sense to me, it sounds like we want to make a clear distinction between the underlying addons that are collecting the data by creating a new study project.

Comment 6

[Wesley Dawson [:whd]]

Thanks for the clarification. I don't follow Rally work closely and I wasn't aware we were going to deploy any more pioneer-style addons. I've pieced together from bugs and discussions that the "two major deadlines" are beginning of March and end of March, but please correct me if I'm wrong. I'm going to assume unless corrected that this work is for the former deadline and https://bugzilla.mozilla.org/show_bug.cgi?id=1677567#c9 the latter.

We unfortunately don't support numbers in namespaces so we're going to have to come up with a different name for this study.

Comment 7

[Ted Han]

Yeah, sorry, i'll back up to fill in the jargon gap that I've been using.

We're making a product distinction between Ion and Rally. From the data platform perspective, they're just the different front ends for installing study addons + sending pings. Ion is the about:ion page and components to serve it. Rally is the website+core-addon driven experience that we're launching. Study implementation differs a bit from the developer's experience (Rally studies must use Rally.js, and Rally.js manages sending data to the Core Addon, which in turn submits pings to the telemetry pipeline).

We would like to migrate away from the pioneer name, but (as should probably be clear in this ticket!) I had mentally decoupled that migration from the user-facing changes we're making.

Separately from that, we're looking at migrating to a Glean-based model of sending data.

For right now, we're interested in launching a new version News & Disinformation study, one which uses Rally.js, and is installed through the Rally front end. If we have to put up w/ tables and routing that still use pioneer prefixes, that's okay (and i know that will commit us to keeping the pioneer prefix around until those studies are done).

Also if we don't support numerics in names, i am perfectly happy with pioneer-citp-news-disinfo-two ;) If a -two suffix is aesthetically unpleasant, happy to use -rally as a suffix instead.

Comment 8

[Alessio Placitelli [:Dexter]]

(In reply to Wesley Dawson [:whd] from comment #6)

I'm going to assume unless corrected that this work is for the former deadline and https://bugzilla.mozilla.org/show_bug.cgi?id=1677567#c9 the latter.

You are correct.

Comment 9

[Wesley Dawson [:whd]]

pioneer-citp-news-disinfo-two it shall be.

I've provisioned the key for the study:

{
    "crv": "P-256",
    "kid": "citp-news-disinfo-two",
    "kty": "EC",
    "x": "Q20tsJdrryWJeuPXTM27wIPb_YbsdYPpkK2N9O6aXwM",
    "y": "1onW1swaCcN1jkmkIwhXpCm55aMP8GRJln5E8WQKLJk"
}

I'm planning to land the schemas and analysis project on Monday. Depending on whether we wait for GCP to enable DNS restrictions for the project, the analysis environment will be available then or a few days later.

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/658

Comment 11

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/cloudops-infra/pull/2894

The analysis project is moz-fx-data-pioneer-citp-f757. DNS restrictions were enabled today as well.