1305742 - Add gfx backend information to crash_aggregates

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Jim Mathies [:jimm]]

We have access to various telemetry data via sql.telemetry.mozilla.org [1]. From within this interface we have a crash metric dataset we can query named crash_aggregates [2], which includes 'stats' and 'dimensions':

DESCRIBE crash_aggregates
  activity_date	varchar	true	false	
  dimensions		true	false	
  stats		true	false	
  submission_date	varchar	true	true

SELECT stats
FROM crash_aggregates
LIMIT 1
{
    "usage_hours_squared": 486.28761782407406,
    "gmplugin_crashes_squared": 0,
    "main_crashes_squared": 0,
    "usage_hours": 42.68194444444445,
    "content_crashes": 0,
    "content_crashes_squared": 0,
    "main_crashes": 0,
    "plugin_crashes": 0,
    "plugin_crashes_squared": 0,
    "ping_count": 13,
    "gmplugin_crashes": 0
}

SELECT dimensions
FROM crash_aggregates
LIMIT 1
{
    "build_id": "20151208100201",
    "os_name": "Windows_NT",
    "application": "Firefox",
    "country": "AE",
    "os_version": "5.2",
    "architecture": "x86",
    "build_version": "43.0",
    "channel": "beta",
    "e10s_enabled": "False"
}

From within dimensions data, I would like have accelerated graphics backend information (gfx.features.compositor in the environment data) to filter on.

[1] https://sql.telemetry.mozilla.org/queries/1022/source#table
[2] https://github.com/mozilla/telemetry-batch-view/blob/master/docs/CrashAggregateView.md

Comment 1

[Jim Mathies [:jimm]]

Hey Roberto, can you help to find an owner here?

Comment 2

[Mauro Doglio [:mdoglio]]

I'm taking this. A couple of questions:
1)Is that value always present/required?
2)Do I need to plan any backfill for that value?

Comment 3

[Jim Mathies [:jimm]]

Hey Mauro,

(In reply to Mauro Doglio [:mdoglio] from comment #2)
> I'm taking this. A couple of questions:
> 1)Is that value always present/required?

Yes it appears to be based on the telemetry env docs. [1]

> 2)Do I need to plan any backfill for that value?

Having backfilled data for beta 50 would be great but not required.

[1] http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/data/environment.html

Comment 4

[Jim Mathies [:jimm]]

Hey milan, can you confirm 'gfx.features.compositor' is a suitable telemetry entry [1] for filtering between accelerated and non-accelerated clients? (e.g. basic, none -> un-accelerated, everything else -> accelerated)

[1] http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/data/environment.html

Comment 5

[Mauro Doglio [:mdoglio]]

If this is not particularly urgent I'll put it in the plan for q4.

Comment 6

[Jim Mathies [:jimm]]

(In reply to Mauro Doglio [:mdoglio] from comment #5)
> If this is not particularly urgent I'll put it in the plan for q4.

This is urgent, we'll need this data for beta 50 analysis related to e10s and XP/Vista. So like, this week, or next? Is this doable Mauro?

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Jim Mathies [:jimm] from comment #4)
> Hey milan, can you confirm 'gfx.features.compositor' is a suitable telemetry
> entry [1] for filtering between accelerated and non-accelerated clients?
> (e.g. basic, none -> un-accelerated, everything else -> accelerated)
> 
> [1]
> http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/
> data/environment.html

Yes, except you can't trust the OS X values until bug 1306383 is fixed.

Comment 8

[Mauro Doglio [:mdoglio]]

Yeah it's doable, I'll work on it asap.

Comment 9

[Jim Mathies [:jimm]]

(In reply to Milan Sreckovic [:milan] from comment #7)
> (In reply to Jim Mathies [:jimm] from comment #4)
> > Hey milan, can you confirm 'gfx.features.compositor' is a suitable telemetry
> > entry [1] for filtering between accelerated and non-accelerated clients?
> > (e.g. basic, none -> un-accelerated, everything else -> accelerated)
> > 
> > [1]
> > http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/
> > data/environment.html
> 
> Yes, except you can't trust the OS X values until bug 1306383 is fixed.

Thanks, I'm currently only interested in windows so no issue here.

(In reply to Mauro Doglio [:mdoglio] from comment #8)
> Yeah it's doable, I'll work on it asap.

Thanks!

Comment 10

[Mauro Doglio [:mdoglio]]

I see sometimes the value for gfx/features/compositor is 'none'. Does it have the same semantic as a non existing value?

Comment 11

[Jim Mathies [:jimm]]

(In reply to Mauro Doglio [:mdoglio] from comment #10)
> I see sometimes the value for gfx/features/compositor is 'none'. Does it
> have the same semantic as a non existing value?

A field that's empty seems to imply none for layers system so yes, I think you could combine these.

http://searchfox.org/mozilla-central/rev/c1e745733c84630821ef53754b627f2c0b0b5202/gfx/thebes/gfxPlatform.h#100

Comment 12

[GitHub Autolander Bot]

Attachment: [telemetry-batch-view] mozilla:bug-1305742-add-gfx-compositor > mozilla:master

Comment 13

[Mauro Doglio [:mdoglio]]

It's taking more than expected to bring this work to completion because of a memory issue when running the tests on CI. I may need to rewrite a bunch of tests because of that, which will probably take another day or two. Sorry it's taking so long.

Comment 14

[Jim Mathies [:jimm]]

(In reply to Mauro Doglio [:mdoglio] from comment #13)
> It's taking more than expected to bring this work to completion because of a
> memory issue when running the tests on CI. I may need to rewrite a bunch of
> tests because of that, which will probably take another day or two. Sorry
> it's taking so long.

Is this post associated with this bug? What tests?

Comment 15

[Mauro Doglio [:mdoglio]]

I was referring to the pull request in attachment 1 [details] [diff] [review], the CI tests that were having memory issues were the ones on Travis. I found a way to circumvent the problem though. I'm now addressing the last few issues in the review. I'm confident I can start doing the backfill today.

Comment 16

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mozilla:crash-backfill-dag > mozilla:master

Comment 17

[Mauro Doglio [:mdoglio]]

The gfx backend is now available as dimensions['gfx_compositor']. The data has been backfilled starting from September 20th.

Comment 18

[Jim Mathies [:jimm]]

(In reply to Mauro Doglio [:mdoglio] from comment #17)
> The gfx backend is now available as dimensions['gfx_compositor']. The data
> has been backfilled starting from September 20th.

Thanks Mauro! Data looks good.