809949 - Need ability to gather memory reports on production phones

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect, P4)

Description

[Dave Hylands [:dhylands]]

On production phones, adb shell will not be root.

Currently, the way we gather memory reports is to use adb shell to send a signal to the b2g process. On the dog-fooding phones adb shell gives us a root shell, which is what makes this possible.

For production phones, we'll no longer have a root shell, so we need another way.

One potential solution is to make the killer program setuid root.
Another potential solution is to use a named pipe

Comment 1

[Doug Turner (:dougt)]

I'd love to have this, but I am not sure we'd block shipping v.1 for this support

Comment 2

[Justin Lebar (not reading bugmail)]

I agree we can't block release on this, but to be clear, this is probably the highest priority outstanding MemShrink bug; user reports have been invaluable to us on Firefox, and we're not OK shipping a phone without this capability.

So let's please devote resources to this as soon as it's fixable (at the moment, we don't even have release phones).

The way we've been denoting these high-priority but not-blocking MemShrink fixes is with blocking-basecamp+ and [soft-blocker], so I'm going to tag this accordingly.  (I'll be the first to admit that this is an abuse of b-b+, but we were unable to agree on a better procedure with release drivers.)

Comment 3

[Nicholas Nethercote [inactive]]

(In reply to Justin Lebar [:jlebar] from comment #2)
> I agree we can't block release on this, but to be clear, this is probably
> the highest priority outstanding MemShrink bug; user reports have been
> invaluable to us on Firefox, and we're not OK shipping a phone without this
> capability.

I agree!

Comment 4

[Ting-Yuan Huang]

Are we going to have something formal, like crash reports, to gather memory reports from users automatically on OOM, or just to make memory reports directly and easily accessible to developers on production phones?

Comment 5

[Andrew McCreight [:mccr8]]

More like the latter: memory reports should be easily accessible to users on production phones. The usual work flow for desktop is a user files a bug report like "When I use Firefox, it uses too much memory!". Then we get them to go to about:memory and copy-and-paste it into the bug, perhaps also along with their list of addons, and go from there to analyze what the problem is.

The user doesn't necessarily need to be able to see the memory report themselves (though that might be handy), but they need a way to easily report it to us.

Comment 6

[Justin Lebar (not reading bugmail)]

> Are we going to have something formal, like crash reports, to gather memory reports from 
> users automatically on OOM, or just to make memory reports directly and easily accessible 
> to developers on production phones?

Note that we can't collect a memory report on OOM; it's too late, we're too far gone by that point.

Comment 7

[Andrew Overholt [:overholt]]

We're now marking soft blockers as P4/blocking-basecamp=- so I'm changing this bug to be consistent.

Comment 8

[Justin Lebar (not reading bugmail)]

Attachment: Patch, v1

I'm really sorry to ask for expedited reviews here, but I'm afraid that if we
don't land this within the next few days, we will have a touch time getting
this in for B2G v1.  (I.e., my magic approval powers may run out soon.)

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 700349 [details] [diff] [review]
Patch, v1

Review of attachment 700349 [details] [diff] [review]:
-----------------------------------------------------------------

I won't pretend I know much of the MessageLoopForIO API, but this sounds good to me.

Comment 10

[Justin Lebar (not reading bugmail)]

I left in the signal handler because

 a) My tools currently send signals, so I'd need to change them before removing the signal handler, and

 b) I thought it might be useful to be able to send a signal to a specific process, particularly if the main thread is horked.  But maybe we can get around this by creating multiple fifos, one per process.

Anyway, I'd like to push this to a follow-up, if that's OK, since we're so close here.

Comment 11

[Nicholas Nethercote [inactive]]

Comment on attachment 700349 [details] [diff] [review]
Patch, v1

Review of attachment 700349 [details] [diff] [review]:
-----------------------------------------------------------------

r=me, but I admit to adding very little value with my review.

Comment 12

[Justin Lebar (not reading bugmail)]

With our powers combined...

https://hg.mozilla.org/integration/mozilla-inbound/rev/8461f75fd62b

Comment 13

[Ed Morley [:emorley]]

Backed out for breaking the build:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=8461f75fd62b

https://hg.mozilla.org/integration/mozilla-inbound/rev/199a1cc814bf

Comment 14

[Justin Lebar (not reading bugmail)]

Well, that was fun.

Comment 15

[Justin Lebar (not reading bugmail)]

https://tbpl.mozilla.org/?tree=Try&rev=7d48de6b9419

Comment 16

[Justin Lebar (not reading bugmail)]

I'm hitting a crash in xpcshell when shutting down the IO thread.  It appears to be a double-free, maybe?  I'm spinning a valgrind build now.

Comment 17

[Justin Lebar (not reading bugmail)]

Hmm.  Here's a bit of deja vu from bug 817946; I'm seeing errors in Chromium's MessagePump / libevent.

This really looks like a bug not in my code.  :(

> Invalid write of size 8
>    at 0x6CA2797: event_queue_remove (event.c:932)
>    by 0x6CA2ED8: event_del (event.c:804)
>    by 0x6CBD106: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:141)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5ad8 is 8 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 8
>    at 0x6CA2F73: event_base_free (event.c:226)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5ad0 is 0 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 1
>    at 0x6CA2F76: event_base_free (event.c:227)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b4c is 124 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 8
>    at 0x6CA2E96: event_del (event.c:782)
>    by 0x6CA2F80: event_base_free (event.c:228)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b08 is 56 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 2
>    at 0x6CA2EAB: event_del (event.c:792)
>    by 0x6CA2F80: event_base_free (event.c:228)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b16 is 70 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 1
>    at 0x6CA2EB2: event_del (event.c:797)
>    by 0x6CA2F80: event_base_free (event.c:228)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b4c is 124 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 1
>    at 0x6CA2EB8: event_del (event.c:800)
>    by 0x6CA2F80: event_base_free (event.c:228)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b4c is 124 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)
> 
> Invalid read of size 1
>    at 0x6CA2EC3: event_del (event.c:803)
>    by 0x6CA2F80: event_base_free (event.c:228)
>    by 0x6CBD130: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:147)
>    by 0x6CBD14E: base::MessagePumpLibevent::~MessagePumpLibevent() (message_pump_libevent.cc:148)
>    by 0x6CACA7C: MessageLoop::~MessageLoop() (ref_counted.h:107)
>    by 0x6CB1D42: base::Thread::ThreadMain() (thread.cc:166)
>    by 0x6CBD57B: ThreadFunc(void*) (platform_thread_posix.cc:39)
>    by 0x4E3BE99: start_thread (pthread_create.c:308)
>    by 0x8FCE4BC: clone (clone.S:112)
>  Address 0x112a5b4c is 124 bytes inside a block of size 128 free'd
>    at 0x4C2B6F9: free (vg_replace_malloc.c:446)
>    by 0x5256F72: moz_free (mozalloc.cpp:48)
>    by 0x6CBD1AD: base::MessagePumpLibevent::FileDescriptorWatcher::StopWatchingFileDescriptor() (mozalloc.h:224)
>    by 0x6C901EE: (anonymous namespace)::SignalPipeWatcher::StopWatching() (nsMemoryInfoDumper.cpp:238)
>    by 0x6C8F955: RunnableMethod<(anonymous namespace)::FdWatcher, void ((anonymous namespace)::FdWatcher::*)(), Tuple0>::Run() (tuple.h:383)
>    by 0x6CAB995: MessageLoop::RunTask(Task*) (message_loop.cc:333)
>    by 0x6CAD81C: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) (message_loop.cc:341)
>    by 0x6CADA51: MessageLoop::DoWork() (message_loop.cc:441)
>    by 0x6CBCF8B: base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) (message_pump_libevent.cc:311)
>    by 0x6CAB8CB: MessageLoop::RunInternal() (message_loop.cc:215)
>    by 0x6CAB8DA: MessageLoop::RunHandler() (message_loop.cc:208)
>    by 0x6CABB5C: MessageLoop::Run() (message_loop.cc:182)

Comment 18

[Justin Lebar (not reading bugmail)]

In a debug build, I get an assertion in libevent:

> xpcshell: ../../../src/ipc/chromium/src/third_party/libevent/event.c:274: void 
> event_base_free(struct event_base *): Assertion `((&base->eventqueue)->tqh_first == ((void*)0))' 
> failed.

Comment 19

[Justin Lebar (not reading bugmail)]

Oh.  Apparently libevent doesn't have any assertions to keep you from adding the same file watcher twice.  That's what was going on here, and I bet what we were encountering in bug 817946.

Comment 20

[Justin Lebar (not reading bugmail)]

Okay, I filed bug 830123 on the Chromium MessagePump / libevent issue.  The fix to this patch is trivial (remove the duplicate WatchFileDescriptor call), so we should be ready to go with a minor change to the Preferences memory reporter, which I'll post in a moment.

Comment 21

[Justin Lebar (not reading bugmail)]

Attachment: Patch, v2

This should fix the issues I was seeing on tryserver.

Comment 22

[Justin Lebar (not reading bugmail)]

https://tbpl.mozilla.org/?tree=Try&rev=e55c27b17aa7

Comment 23

[Justin Lebar (not reading bugmail)]

Comment on attachment 701589 [details] [diff] [review]
Patch, v2

[Approval Request Comment]
Bug caused by (feature/regressing bug #): n/a
User impact if declined: Can't get memory reports from production devices, which means we have zero insight into memory usage on production devices.
Testing completed: locally
Risk to taking this patch (and alternatives if risky): This is the least-risky way I could think of to let us gather memory reports from user phones.  It's not a tiny though.
String or UUID changes made by this patch: none

Comment 24

[Justin Lebar (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1132ced73377

Comment 25

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/1132ced73377

Comment 26

[Justin Lebar (not reading bugmail)]

This doesn't need tef approval if we can land it today, right?  Can we just do that?

Comment 27

[Lucas Adamski [:ladamski]]

Not blocking but driver approved per triage.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/c2da6d34be4c

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Backed out for bustage.
https://hg.mozilla.org/releases/mozilla-b2g18/rev/2793599d2a95

Comment 30

[Justin Lebar (not reading bugmail)]

(In reply to Ryan VanderMeulen from comment #29)
> Backed out for bustage.
> https://hg.mozilla.org/releases/mozilla-b2g18/rev/2793599d2a95

Yeah, this needs bug 830125 to get around those assertions.  Thanks for trying to land it, though!  That was an unpleasant merge, at least by my standards.  :)

Comment 31

[Justin Lebar (not reading bugmail)]

Let's try this again, shall we?  https://hg.mozilla.org/releases/mozilla-b2g18/rev/59d0b27ec7aa