Closed Bug 1506144 (bmo-stmo) Opened Last year Closed 8 months ago

Access bugzilla security bug metadata via STMO

Categories

(bugzilla.mozilla.org :: General, enhancement, P1)

Production
enhancement

Tracking

()

RESOLVED FIXED

People

(Reporter: psiinon, Assigned: dylan)

References

Details

(Keywords: bmo-goal)

Attachments

(1 file, 3 obsolete files)

We (the Firefox security teams) would like to be able to access security bug metadata via STMO so that we can create live dashboards similar to the static info Dan is now emailing.

The plan is to use the Secops metrics pipeline - that already gathers a growing set of security metrics from a variety of sources and makes them accessible via STMO.

The current pipleline is:
* Scripts run in the Ops Jenkins server and collect raw metrics
* They write the raw data into folders under s3://foxsec-metrics 
* Post processing is applied if we need to reformat it (for Athena)
* Athena tables are applied to the relevant folders and added to the foxsec_metrics db
* The foxsec_metrics db is a data source in STMO

The foxsec_metrics data source in STMO is currently restricted to the Secops team, but the plan is to expand that to all of the Firefox security teams.
We also plan to make some STMO visualisations on this data accessible to everyone in Mozilla.

We would like a subset of the available bugzilla fields so that we don't inadvertently expose details of unfixed security bugs.
The bugs that we are interested in and the fields we would like are currently being collated in this gdoc: https://docs.google.com/document/d/1v5HZlH77PHcPSup0gDJTKUKzuCBE3ltGVwgz0d_DeU0/edit#
Note that right now this is work in progress - I'll update this bug with that info once we have finalized it.

We will need to be able to combine bugzilla data with data from the foxsec_metrics db. I'm not sure if we can combine multiple data sources in STMO.
:robotblake - can you confirm if we can combine data from multiple data sources in the same query in STMO?
Flags: needinfo?(bimsland)
Group: infrasec → bugzilla-security
:robotblake - we also would want to query both data sources in redash, so if that's a limitation, please let us know
Is this really security sensitive?
See Also: → 1506154
:dylan, problably not ;) Just internal then?
A meeting was held on Nov 14th and the above doc updated with notes.
:dylan - you were down to define the schema that BMO will use to provide access to the data. Any eta on this?
Flags: needinfo?(dylan)
I plan on hashing out a gdoc today. Sorry for the delay.
Also can we drop the groups from this bug? security bugs in this product in my inbox give me anxiety.
Flags: needinfo?(dylan)
Group: bugzilla-security → bmo-infra
Group: bmo-infra

Any progress on the schema Dylan?

Flags: needinfo?(dylan)

I hope to hop back on this soon.

Flags: needinfo?(dylan)
Assignee: nobody → dylan
Component: Administration → General
Priority: -- → P1
Attached file Framework for sending data to STMO (obsolete) —
Attached file Database model helpers (obsolete) —
Depends on: 1526371
Keywords: bmo-goal
Attached file Add DBIx::Class (obsolete) —
Attachment #9042520 - Attachment is obsolete: true

:dylan I'm afraid I dont understand the BMO schema / API definition enough to understand if the code in the above attachment includes all of fields we highlighted in the gdoc: https://docs.google.com/document/d/1v5HZlH77PHcPSup0gDJTKUKzuCBE3ltGVwgz0d_DeU0/edit#heading=h.9zk2sxea4fs
Can you confirm if all of the fields we highlighted will be included in the schema or if not which fields will be missing?
I'm not concerned about the structure of the data, we can manipulate that in STMO, its just whether the raw info will be there.

Flags: needinfo?(dylan)

we chatted about this, and the PR has a revised list.

Flags: needinfo?(dylan)

The fields we need are:

blocks
component
creation_time
creator
depends_on
dupe_of
duplicates
flags
groups
id
is_open
keywords
last_change_time
last_resolved
priority
product
resolution
severity
bug_status
target_milestone
whiteboard

Ideally we'd also like:

platform

:dylan does that help?

Flags: needinfo?(dylan)
Depends on: bmo-model
Alias: bmo-stmo
Flags: needinfo?(dylan)
Attachment #9042548 - Attachment is obsolete: true
Attachment #9042521 - Attachment is obsolete: true
Attached file GitHub Pull Request

(In reply to Simon Bennetts [:psiinon] from comment #15)

The first version omits the following fields, which will be added in a future update.

The fields we need are:
creation_time

Missing due to dates being difficult.

is_open

Missing because it's non-trivial to expose.

last_change_time

Because it's a date.

last_resolved

Because it's added by an extension

whiteboard

Because it's added by an extension.

platform

Missing because it was overlooked.

Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Blocks: 1558950
You need to log in before you can comment on or make changes to this bug.