785698 - Set useful release/update channels and upload symbols for B2G

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Kaiser]

For having useful crash reports for B2G, we need symbol information for the commonly used builds, and we need useful release (update) channels set.

Currently, the Mac desktop builds apparently upload symbols, but all other (nightly) builds don't. Therefore, we only get useful (symbolized) stacks for Mac builds right now.

Also, all B2G nightly builds seem to use the "default" channel, even though some of the mozconfigs would contain the --enable-update-channel=${MOZ_UPDATE_CHANNEL} line we use for Firefox, but that env var used in there isn't being set by buildbot, so we fall back to "default" to not actually have an empty channel in there.

The problem with that is that Socorro only puts crash reports into e.g. the "17.0a1" version is they also have the "nightly" channel set (to hide noise of private testing builds, etc. - which have the "default" channel), and the same for aurora and beta versions. So, as long as those builds don't come with useful channels set, we just ignore them in any crash stats.

This second issue is connected to bug 778341.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Is this bug only related to desktop builds at this time ? As I read bug 778341 it's about device builds for B2G.

Comment 2

[Robert Kaiser]

At this time, we don't have crashes for device builds, but once we do (and have people using "official" builds from our build system), the same is true for those, as Socorro doesn't care if the crash comes from a desktop or device.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Grabbing this, as I'm already sorting out build (and update) requirements with B2G devs and PMs for related bug#778341.

Comment 4

[(not currently active) Ted Mielczarek]

I'm not sure that having crash reporting for desktop B2G builds adds much value. It's not testing anywhere near the same code as a device build.

Comment 5

[Robert Kaiser]

Well, Socorro has exactly the same requirements for device builds. We can intentionally leave desktop builds on the "default" channel so that they are ignored in crash-stats, but then we should clean up the --enable-update-channel lines in their mozconfigs to make sure we don't set one.
For the device builds, we will need to have the channels set in the same way as Firefox Nightly/Aurora/Beta/Release builds do, or we'll probably require a few weeks (!) of Socorro work to get things to work somehow - and in that case, we'd need to know soon so the team can plan for that. And we'll also need symbols uploaded for the device builds.

Still, *right now* the crash reports from desktop builds are all we have, and as from the Mac builds I'm seeing a lot of IPC-related crashes there, I fear that our process separation might hit us with some severe issues across the board, and that code is the same on all platforms including devices, AFAIK. We have no way other than Mac builds to investigate any stacks right now, though.

All that said, my main concern is that we will have the right things in place once we get device crashes in, and from all I've seen, the device (ARM) builds have exactly the same issues there.

Comment 6

[Robert Kaiser]

Oh, and we need this set in the mozconfigs for devices as well:
export MOZILLA_OFFICIAL=1

Should I file a separate bug for that?

Comment 7

[Robert Kaiser]

(In reply to Robert Kaiser (:[e-mail]) from comment #6)
> Oh, and we need this set in the mozconfigs for devices as well:
> export MOZILLA_OFFICIAL=1
> 
> Should I file a separate bug for that?

Meanwhile, jsmith filed bug 789108 for this in terms of the images we're using for testing right now, so let's follow up on those right now.

For the desktop builds, I guess it might be easy to turn on the symbol creation and push for Linux and Windows as well, but we shouldn't put too much priority on the desktop builds in terms of crash reporting right now, the device builds are more important for sure.

Comment 8

[Myk Melez [:myk] [@mykmelez]]

(In reply to Ted Mielczarek [:ted] from comment #4)
> I'm not sure that having crash reporting for desktop B2G builds adds much
> value. It's not testing anywhere near the same code as a device build.

App developers are using B2G Desktop to test their apps, as phones and emulators are hard to come by and difficult to set up/configure.  And they need B2G Desktop to be stable, and crashes in it to get addressed, for that.  So there's value in getting the symbols flowing for B2G Desktop Linux and Windows builds.

Comment 9

[(not currently active) Ted Mielczarek]

I have a hard time believing Desktop B2G builds will hit a different set of crashes than Desktop Firefox builds. I'm not morally opposed to the concept, though.

Comment 10

[Robert Kaiser]

(In reply to Ted Mielczarek [:ted] from comment #9)
> I have a hard time believing Desktop B2G builds will hit a different set of
> crashes than Desktop Firefox builds.

Actually, they might, as B2G heavily uses content processes, which Firefox doesn't use the same way (even though plugin processes are similar in code).

Comment 11

[Myk Melez [:myk] [@mykmelez]]

It's possible that the two will crash mostly in the same way, but I have been testing B2G Desktop nightly builds that are crashing on startup, and the Firefox nightlies from the same days aren't crashing, so I would very much like symbols in order to determine the cause of that crash.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #11)
> It's possible that the two will crash mostly in the same way, but I have
> been testing B2G Desktop nightly builds that are crashing on startup, and
> the Firefox nightlies from the same days aren't crashing, so I would very
> much like symbols in order to determine the cause of that crash.

This ended up being Gaia bug 794662, which is triggered by JS Engine bug 794927, so the underlying cause of the crash is not B2G-specific, but it was showing up specifically in B2G builds using a Gaia profile.

Comment 13

[Myk Melez [:myk] [@mykmelez]]

And fixing bug 794662 exposed bug 795484, also a B2G Desktop startup crasher that doesn't manifest on Firefox startup.  So we are very much in a world in which B2G Desktop crashes differently from Firefox.