Open Bug 1409964 Opened 7 years ago Updated 5 years ago

Add templates to the data-review flag.

Categories

(bugzilla.mozilla.org :: Administration, task)

Product:
bugzilla.mozilla.org
Component:
Administration
Version:
Production
Type:
task
Priority:
Not set
Severity:
normal

Tracking

()

People

(Reporter: liuche, Unassigned)

References

Details

For the Data Review process, we'd like to add a data-r? flag that auto-populates a comment with a template of questions, much like uplift requests like the approval-beta? flag. Currently for data review (and we're working to improve this process), people flag data peers with some combination of r?/f?/NI flags. Data peers look at the documentation, and there's usually some back and forth on bugs to ask questions about why the data is being collected, how it is going to be used, and what alternatives have been considered. As part of making the data review process more transparent and clear and faster for those requesting data review (and to remove uncertainty about what the requirements are to get an r+), we'd like to ask those questions up front when people request data review, and have developed a template of questions. In testing this process out, we're asking people who request data review to fill out and attach a document of these template questions to the bug. The general goal is so that somewhere down the line, if we revisit telemetry probes, there is clear documentation for why this data is being collected and how it was intended to be used. Here's an example of a data review, where the template document of questions needed to be filled out for data review: https://bugzilla.mozilla.org/show_bug.cgi?id=1402267#c17 Can we add a data-r? flag, with a comment with something along the lines of the following template questions? (These are kind of long, so if you have feedback, let's talk about it.) Data Review Comment: [What are the motivating questions you want to answer with this data?] [Why does Mozilla need answers to this question? Are there benefits for users? Do we need this information to address business requirements?] [List all proposed measurements. Refer to data collection types for Firefox on the Mozilla wiki and indicate types of measurements for each one.] [How long will this data be collected? until date XXXXXX / 6 months initially / (someone's name) will permanently monitor this data] [What populations do you need to these measurements for? Which release channels? Which countries? Which locales? Any other filters? Please describe in detail below.] [General description of how you will analyze this data] [What alternative methods did you consider to answer this question? Why were they not sufficient?] [Can current instrumentation answer this question?] [Where do you intend to share the results of your analysis?]
This isn't immediately urgent, and somewhere along the timeline of "after 57 release" is fine, but wanted to surface this since we'll be announcing data review updates soon, and I wanted to know what we'd need to do to do this (instead of having people attach google docs).
Flags: needinfo?(ehumphries)
Summary: Add data-r? flag with template → Add data-r? flag with comment template questions
Which products should the flag be available for?
Flags: needinfo?(ehumphries) → needinfo?(liuche)
Dylan, since this is like the release tracking flag, this will be a code change?
Assignee: administration → nobody
Flags: needinfo?(liuche) → needinfo?(dylan)
Product: Bugzilla → bugzilla.mozilla.org
QA Contact: default-qa
Version: unspecified → Production
Also, is there an existing group that would grant the flag, or do you need a new group for that. And who would manage and be in that group?
Flags: needinfo?(liuche)
This would be mainly for Firefox* components, where most reviews come in. There isn't an existing group who could "+" these, but this would be the Data Stewards (with data peer rweiss). Right now, we have a github repo, and this is the current set of questions for data review: https://github.com/mozilla/data-review/blob/master/request.md I'm going to be on PTO but you can flag rweiss with questions, or I will be back in two weeks!
Flags: needinfo?(liuche)
I think this is just about comment templates, which are part of the design for tracking flags.
Flags: needinfo?(dylan)
So, is this still needed? if so give it P1 and I'll discuss it with dkl as I think I was confused to the functionality.
Flags: needinfo?(ehumphries)
:rweiss, is this still needed?
Flags: needinfo?(ehumphries) → needinfo?(rweiss)
I'm hesitant to add new flags right now. There may be better ways to do this, particularly if this is on a per-patch basis and not a per-bug basis, since code review is moving to Phabricator. So if this general idea is still needed I'd like to go back to setting requirements for it.
This slipped. I'll resurrect with data stewards in our next monthly and revisit this bug afterwards.
Blocks: phab-switch
Priority: -- → P1
Assignee: nobody → glob

chutten: as per our email conversation are you ok with us creating a data-review flag, leaving the templating as-is?

Flags: needinfo?(chutten)

Yes. Please spin out a bug for the flag and we'll keep this one for the template discussion.

Flags: needinfo?(chutten)

created bug 1526952 for creating the flag, morphed this bug into just the template.

No longer blocks: phab-switch
Flags: needinfo?(rweiss)
See Also: → 1526952
Summary: Add data-r? flag with comment template questions → Add templates to the data-review flag.
Priority: P1 → --
Assignee: glob → nobody

Is this work still needed?

Flags: needinfo?(chutten)

Yes and no. Yes in that Data Collection Reviews come along with decent frequency and smoothing out the review process might improve efficiency and accuracy. No in that it is unclear if this will bring us closer to an ideal expression of the Data Collection Review process using BMO tools.

Let me bring this up at the Data Stewards meeting on Tuesday, and then I'll get you a better answer : )

Flags: needinfo?(chutten)
Flags: needinfo?(chutten)

We met about this late in 2019 (meeting notes) with the result being that the UI templates in use for uplift approvals would be a lovely time-saving addition for Data Review requestors. But unfortunately that UI is very custom so our use of it was predecated on that UI customization being made more widely available.

At least I think that's what it was. ni?emceeaich to see if this matches her recollection.

Flags: needinfo?(chutten) → needinfo?(ehumphries)

I set up a "data-review" flag on bugzilla-dev that triggers comment templates to see if that would work as an alternative.

Flags: needinfo?(ehumphries)
You need to log in before you can comment on or make changes to this bug.