1021048 - Get B2G Nightly 32 and future Nightly versions into Socorro correctly

Status: RESOLVED DUPLICATE of bug 1153352

(Socorro :: Backend, task)

Description

[Robert Kaiser]

We do get crashes now where we have the correct version and channel set, see e.g. https://crash-stats.mozilla.com/search/?version=32.0a1&product=B2G&release_channel=nightly&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=release_channel&_columns=android_device

Still, we do not get Nightly 32 (or "32.0a1") listed in the versions dropdown of https://crash-stats.mozilla.com/home/products/B2G - we should make sure that this and other such versions do and will show up correctly on Socorro.

Comment 1

[Robert Helmer [:rhelmer]]

If these are builds we're publishing to FTP (like http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014-06-04-16-02-02-mozilla-central-flame/) then we should be picking them up, if not it's a big in ftpscraper

Comment 2

[Robert Kaiser]

(In reply to Robert Helmer [:rhelmer] from comment #1)
> If these are builds we're publishing to FTP (like
> http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014-06-04-16-02-02-
> mozilla-central-flame/) then we should be picking them up, if not it's a big
> in ftpscraper

That's the builds I mean, yes. That said, shouldn't there be some JSON file or so with build info in the same directory? If so, we probably need a releng fix here.

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Robert Kaiser (:[e-mail]) from comment #2)
> (In reply to Robert Helmer [:rhelmer] from comment #1)
> > If these are builds we're publishing to FTP (like
> > http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014-06-04-16-02-02-
> > mozilla-central-flame/) then we should be picking them up, if not it's a big
> > in ftpscraper
> 
> That's the builds I mean, yes. That said, shouldn't there be some JSON file
> or so with build info in the same directory? If so, we probably need a
> releng fix here.

The ftpscraper is looking for something like:
socorro_unagi-stable_2013-01-25-07.json

I don't see any JSON files at all in the above dir just a sources.xml (which I believe are there for partner's benefit, I wonder if we could use those instead?)

We should ask releng why there aren't JSON metadata files anymore.

Comment 4

[Robert Kaiser]

(In reply to Robert Helmer [:rhelmer] from comment #3)
> We should ask releng why there aren't JSON metadata files anymore.

catlee, any idea on that?

Comment 5

[[e-mail] (:bhearsum)]

We definitely generate this as part of the nightly build, looks like it's never uploaded though. We probably need to add something like "{workdir}/socorro.json" to public upload list: https://mxr.mozilla.org/mozilla-central/source/b2g/config/flame/config.json#13

Not sure if there's anything else, I haven't deal with these files before...

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #5)
> We definitely generate this as part of the nightly build, looks like it's
> never uploaded though. We probably need to add something like
> "{workdir}/socorro.json" to public upload list:
> https://mxr.mozilla.org/mozilla-central/source/b2g/config/flame/config.
> json#13
> 
> Not sure if there's anything else, I haven't deal with these files before...

Aki, do we need to do more than this?

Comment 7

[Aki Sasaki (not active)]

It looks like make-socorro-json and upload-source-manifest are their own actions:
http://hg.mozilla.org/build/mozharness/file/332b0fe18d4f/scripts/b2g_build.py#l180

This predated having any sort of [other] public upload.  These files live here (for 2.0 nightly):
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/nightly/2.0.0/latest/

The right thing to do at this point is probably kill those two actions, and make sure the source manifest + socorro json are uploaded in the public upload step.  We'll need to move the make_socorro_json generation part up in b2g_build.py at the very least.

Comment 8

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #7)
> It looks like make-socorro-json and upload-source-manifest are their own
> actions:
> http://hg.mozilla.org/build/mozharness/file/332b0fe18d4f/scripts/b2g_build.
> py#l180
> 
> This predated having any sort of [other] public upload.  These files live
> here (for 2.0 nightly):
> http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/nightly/2.0.0/latest/
> 
> The right thing to do at this point is probably kill those two actions, and
> make sure the source manifest + socorro json are uploaded in the public
> upload step.  We'll need to move the make_socorro_json generation part up in
> b2g_build.py at the very least.

a) looks like rhelmer can use b2g/manifests/ for now.
b) I filed bug 1021853 for the work in comment 7.

Comment 10

[Gregor Wagner [:gwagner]]

Any updates here?

Comment 11

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Also to note : https://metrics.services.mozilla.com/fxos-ftu-dashboard/
is getting some sort of data.  

I was just wondering, could we use it as a metrics for ADI?

Comment 12

[Robert Helmer [:rhelmer]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #11)
> Also to note : https://metrics.services.mozilla.com/fxos-ftu-dashboard/
> is getting some sort of data.  
> 
> I was just wondering, could we use it as a metrics for ADI?

Does the data need to be private? That's the main problem I could see. crash-stats graphs are public and it's trivial to determine ADI if you know the equation is (crashes / hundred_ADI)

If that is not a problem then the main question is integrating. If it *is* a problem we should talk about whether it makes sense to have a more restricted crash-stats dashboard.

It should certainly be possible to use the API at https://crash-stats.mozilla.org/api and the ADI data to make a dashboard right now, it doesn't necessarily need to be hosted on crash-stats.m.o

Comment 13

[Robert Kaiser]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #11)
> I was just wondering, could we use it as a metrics for ADI?

No. We'd need something that is a proxy to actual usage, not a count of new device activations, which FTU stuff is. In addition, as Rob mentions, that data is said to be highly confidential, and Socorro exposes the ADI data publicly.

Comment 14

[Gregor Wagner [:gwagner]]

(In reply to Robert Kaiser (:[e-mail]) from comment #13)
> (In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from
> comment #11)
> > I was just wondering, could we use it as a metrics for ADI?
> 
> No. We'd need something that is a proxy to actual usage, not a count of new
> device activations, which FTU stuff is. In addition, as Rob mentions, that
> data is said to be highly confidential, and Socorro exposes the ADI data
> publicly.

Can we just normalize the data so we can't determine any ADI number?
Only show percentage without any absolute number and only if you log in you see the actual crash count?
Normalize the crash count that the sum is always 1000000? Just throwing out ideas here.

Comment 15

[Robert Kaiser]

(In reply to Gregor Wagner [:gwagner] from comment #14)
> Can we just normalize the data so we can't determine any ADI number?

No. A count of device activations is not a count or proxy to actual daily usage. And crash rates only make sense in relation to something that reflects actual usage, not first-time activations.

Comment 16

[Gregor Wagner [:gwagner]]

(In reply to Robert Kaiser (:[e-mail]) from comment #15)
> (In reply to Gregor Wagner [:gwagner] from comment #14)
> > Can we just normalize the data so we can't determine any ADI number?
> 
> No. A count of device activations is not a count or proxy to actual daily
> usage. And crash rates only make sense in relation to something that
> reflects actual usage, not first-time activations.

So what can we do here?

Comment 17

[Robert Kaiser]

(In reply to Gregor Wagner [:gwagner] from comment #16)
> So what can we do here?

For the ADI we can do nothing at all until we have some kind of actual usage metric for B2G that we can use.

But this bug is not at all about the ADI. It's way more important to actually get FTP scraping and/or other reporting of available/valid builds into the Socorro DB work correctly.

Comment 18

[Robert Helmer [:rhelmer]]

(In reply to Gregor Wagner [:gwagner] from comment #16)
> (In reply to Robert Kaiser (:[e-mail]) from comment #15)
> > (In reply to Gregor Wagner [:gwagner] from comment #14)
> > > Can we just normalize the data so we can't determine any ADI number?
> > 
> > No. A count of device activations is not a count or proxy to actual daily
> > usage. And crash rates only make sense in relation to something that
> > reflects actual usage, not first-time activations.
> 
> So what can we do here?

Would you mind filing a bug on just this? Socorro::General is fine.

Note that I suspect the answer is going to be that you can pull crash rate from crash-stats/api and use that to make a dashboard that uses whatever internal source of ADI we have available for B2G, as I really doubt we'll be able to make that public in any meaningful way.

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Kairo, fyi, we're moving to Taskcluster and it has an API where you don't necessarily have to scrape:
http://docs.taskcluster.net/

Comment 20

[Robert Kaiser]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #19)
> Kairo, fyi, we're moving to Taskcluster and it has an API where you don't
> necessarily have to scrape:
> http://docs.taskcluster.net/

It's not me who is doing any work on this, rhelmer is the person who probably at some point will need to work on getting this done. Unfortunately, he and the Socorro team have been busy recently with getting the migration to AWS done. I don't think we'll see much progress before that is (mostly) finished.

Comment 21

[Robert Helmer [:rhelmer]]

It feels like we've drifted a bit from the original bug.

If we don't have a stable place to pull in release metadata quite yet, then we can do it manually via the crash-stats admin UI. Not having to scrape FTP sounds great to me, as far as I have heard from talking to folks, this API does not exist yet.

The question about a source for ADI is important, but I think should be handled in a separate bug.

Comment 22

[Robert Kaiser]

(In reply to Robert Helmer [:rhelmer] from comment #21)
> It feels like we've drifted a bit from the original bug.

I don't see where. This bug always has been and still is about getting the current B2G builds listed in the Socorro UI (version dropdown etc.) - so in the end about getting the versions into the database (which for other products is achieved by FTP scraper). Anything else is up to other/followup bugs.

> The question about a source for ADI is important, but I think should be
> handled in a separate bug.

Not sure why you bring that up, the last comment in this bug that even mentions ADI clearly states that they're not the scope of this bug.

Comment 23

[Robert Helmer [:rhelmer]]

Not currently working on this - getting B2G releases recorded into Socorro is a long-standing problem. 

Releng has suggested bug 1153352 as a way to make this happen, I think this is better than trying to fix B2G piecemeal or to make ftpscraping work (we'd rather disable that service)

Comment 24

[Peter Bengtsson [:peterbe]]

For what it's worth, in https://bugzilla.mozilla.org/show_bug.cgi?id=1217828 we're fixing the ftpscraper and part of that fix is that we're now correctly picking up B2G releases.
The reason it stopped working was that the scraper would only look into directories start with "1." so sub-directories like "2.5.0" would always be ignored.