Intermittent TEST-UNEXPECTED-FAIL | leakcheck | plugin process: missing output line for total leaks!

NEW
Unassigned

Status

()

P3
normal
3 years ago
20 hours ago

People

(Reporter: nigelb, Unassigned)

Tracking

(Depends on: 1 bug, {intermittent-failure})

Trunk
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox46 affected)

Details

(URL)

(Reporter)

Description

3 years ago
https://treeherder.mozilla.org/logviewer.html#?job_id=19725226&repo=mozilla-inbound


 01:01:20     INFO -  --DOMWINDOW == 1 (0x7fcd1477c800) [pid = 5218] [serial = 201] [outer = (nil)] [url = chrome://browser/content/browser.xul]
 01:01:20     INFO -  --DOMWINDOW == 0 (0x7fcd31c49400) [pid = 5218] [serial = 5] [outer = (nil)] [url = resource://gre-resources/hiddenWindow.html]
 01:01:21     INFO -  nsStringStats
 01:01:21     INFO -   => mAllocCount:         845393
 01:01:21     INFO -   => mReallocCount:       102250
 01:01:21     INFO -   => mFreeCount:          845393
 01:01:21     INFO -   => mShareCount:        1123197
 01:01:21     INFO -   => mAdoptCount:          26878
 01:01:21     INFO -   => mAdoptFreeCount:      26878
 01:01:21     INFO -   => Process ID: 5218, Thread ID: 140519937627968
 01:01:21     INFO -  TEST-INFO | Main app process: exit 0
 01:01:21     INFO -  runtests.py | Application ran for: 0:09:57.302575
 01:01:21     INFO -  zombiecheck | Reading PID log: /tmp/tmpcKwv82pidlog
 01:01:21     INFO -  ==> process 5218 launched child process 5275
 01:01:21     INFO -  ==> process 5218 launched child process 5371
 01:01:21     INFO -  ==> process 5218 launched child process 5380
 01:01:21     INFO -  ==> process 5218 launched child process 5389
 01:01:21     INFO -  zombiecheck | Checking for orphan process with PID: 5275
 01:01:21     INFO -  zombiecheck | Checking for orphan process with PID: 5371
 01:01:21     INFO -  zombiecheck | Checking for orphan process with PID: 5380
 01:01:21     INFO -  zombiecheck | Checking for orphan process with PID: 5389
 01:01:21     INFO -  Stopping web server
 01:01:21     INFO -  Stopping web socket server
 01:01:21     INFO -  Stopping ssltunnel
 01:01:21     INFO -  TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
 01:01:21     INFO -  TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
 01:01:21     INFO -  TEST-INFO | leakcheck | tab process: leak threshold set at 10000 bytes
 01:01:21     INFO -  TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
 01:01:21     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, plugin process 5380
 01:01:21     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
 01:01:21     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|
 01:01:21     INFO -     0 |TOTAL                                 |       47        0|     848        0|
 01:01:21     INFO -  nsTraceRefcnt::DumpStatistics: 28 entries
 01:01:21     INFO -  TEST-PASS | leakcheck | plugin process: no leaks detected!
 01:01:21     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 5218
 01:01:21     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
 01:01:21     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|
 01:01:21     INFO -     0 |TOTAL                                 |       22        0|46519274        0|
 01:01:21     INFO -  nsTraceRefcnt::DumpStatistics: 1465 entries
 01:01:21     INFO -  TEST-PASS | leakcheck | default process: no leaks detected!
 01:01:21  WARNING -  TEST-UNEXPECTED-FAIL | leakcheck | plugin process: missing output line for total leaks!
 01:01:21     INFO -  TEST-INFO | leakcheck | missing output line from log file /tmp/tmpnS2WoB.mozrunner/runtests_leaks_plugin_pid5371.log
 01:01:21     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 5275
 01:01:21     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
 01:01:21     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|
 01:01:21     INFO -     0 |TOTAL                                 |       22     4648|26517147       34|
 01:01:21     INFO -    16 |AsyncTransactionTrackersHolder        |       72       72|     331        1|
 01:01:21     INFO -    84 |CompositorChild                       |      880      880|       1        1|
 01:01:21     INFO -    86 |CondVar                               |       40      120|     880        3|
 01:01:21     INFO -   262 |IPC::Channel                          |       16       32|      18        2|
 01:01:21     INFO -   329 |MessagePump                           |       16       16|      50        1|
 01:01:21     INFO -   339 |Mutex                                 |       32       96|    8520        3|
 01:01:21     INFO -   363 |PCompositorChild                      |      776      776|       1        1|
 01:01:21     INFO -   371 |PImageBridgeChild                     |      920      920|       1        1|
 01:01:21     INFO -   445 |RefCountedMonitor                     |       80      160|      13        2|
 01:01:21     INFO -   447 |RefCountedTask                        |       16       64|      26        4|
 01:01:21     INFO -   507 |StoreRef                              |       16       32|      21        2|
 01:01:21     INFO -   572 |WaitableEventKernel                   |       72       72|      53        1|
 01:01:21     INFO -   578 |WeakReference<MessageListener>        |       16       32|    6705        2|
 01:01:21     INFO -   630 |base::Thread                          |       48       48|       3        1|
 01:01:21     INFO -   663 |ipc::MessageChannel                   |      512     1024|      13        2|
 01:01:21     INFO -  1136 |nsTArray_base                         |        8       48| 8771838        6|
 01:01:21     INFO -  1146 |nsThread                              |      256      256|      49        1|
 01:01:21     INFO -  nsTraceRefcnt::DumpStatistics: 1255 entries
 01:01:21     INFO -  TEST-INFO | leakcheck | tab process: leaked 1 AsyncTransactionTrackersHolder
 01:01:21     INFO -  TEST-INFO | leakcheck | tab process: leaked 1 CompositorChild
(Reporter)

Comment 1

3 years ago
Andrew, Can you take a look at this one? It's kind of weird.
Flags: needinfo?(continuation)
What this assertion means is that the plugin process crashed before it was able to output a leak log. It is odd that the actual crash doesn't show up in the summary. That's probably a separate bug.

I looked at the two logs for this that are in orange factor. If you look at the line after the TEST-UNEXPECTED-FAIL line, it has the file name for the leak log, which contains the PID for the process that crashed: runtests_leaks_plugin_pid5371.log. You can then look back in the log to see where that process was created, and how it crashed.

The crash looks like this:

[NPAPI 4969] ###!!! ASSERTION: plug removed: 'glib assertion', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/toolkit/xre/nsSigHandlers.cpp, line 140
#01: libglib-2.0.so.0 + 0x4ef61
#02: libglib-2.0.so.0 + 0x4f172
#03: libnptest.so + 0x53d4
#04: libgtk-x11-2.0.so.0 + 0x136dd8
#05: libgobject-2.0.so.0 + 0xfca2
[...]
#17: libglib-2.0.so.0 + 0x48124
#18: base::MessagePumpForUI::RunWithDispatcher(base::MessagePump::Delegate*, base::MessagePumpForUI::Dispatcher*) [ipc/chromium/src/base/message_pump_glib.cc:192]
#19: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:235]
#20: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:520]
#21: XRE_InitChildProcess [toolkit/xre/nsEmbedFunctions.cpp:629]
#22: content_process_main(int, char**) [ipc/contentproc/plugin-container.cpp:240]
#23: libc.so.6 + 0x2176d
#24: _start
** (plugin-container:4969): ERROR **: plug removed
[Parent 4811] WARNING: [PluginModuleParent::ActorDestroy] abnormal shutdown without minidump!: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/plugins/ipc/PluginModuleParent.cpp, line 1592

In both logs, the crash is happening in test_fullscreen-api.html, which was just enabled for e10s in bug 1191597. I don't know if this is a problem with the full screen API or with some odd plugin thing.

Xidorn, do you know what is happening here? Thanks.
Blocks: 1191597
Component: General → Plug-ins
Flags: needinfo?(continuation) → needinfo?(quanxunzhen)
Product: Firefox → Core
It looks like bug 1237853 was filed on this issue already, but I guess it wasn't expected to be intermittently happening still?
I filed bug 1239343 for the test harness not noticing the fatal plugin assertion.
(In reply to Andrew McCreight [:mccr8] from comment #2)
> What this assertion means is that the plugin process crashed before it was
> able to output a leak log. It is odd that the actual crash doesn't show up
> in the summary. That's probably a separate bug.
> 
> I looked at the two logs for this that are in orange factor. If you look at
> the line after the TEST-UNEXPECTED-FAIL line, it has the file name for the
> leak log, which contains the PID for the process that crashed:
> runtests_leaks_plugin_pid5371.log. You can then look back in the log to see
> where that process was created, and how it crashed.
> 
> The crash looks like this:
> 
> [NPAPI 4969] ###!!! ASSERTION: plug removed: 'glib assertion', file
> /builds/slave/m-in-l64-d-0000000000000000000/build/src/toolkit/xre/
> nsSigHandlers.cpp, line 140
> #01: libglib-2.0.so.0 + 0x4ef61
> #02: libglib-2.0.so.0 + 0x4f172
> #03: libnptest.so + 0x53d4

FWIW, this is (per running addr2line on libnptest.so):
DeleteWidget
/builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/plugins/test/testplugin/nptest_gtk2.cpp:264

which is explicitly calling g_error.
(In reply to Andrew McCreight [:mccr8] from comment #3)
> It looks like bug 1237853 was filed on this issue already, but I guess it
> wasn't expected to be intermittently happening still?

Before I add the workaround, it was a permafail. I guess the workaround was not reliable enough, and there might be some race condition here.

To totaly resolve this, that bug need to be fixed. But if we do not get much progress there while this intermittent gets more annoying, we probably can make the workaround a bit more reliable via allowing more time before closing the window.

(BTW, why there is no red circle next to "Bugzilla@Mozilla" today given there is a ni? for me... probably a bug need to be filed...)
Flags: needinfo?(quanxunzhen)
Depends on: 1237853
Duplicate of this bug: 1239256
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 3
* fx-team: 3
* mozilla-central: 1

Platform breakdown:
* linux64: 5
* osx-10-6: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-03-07&endday=2016-03-13&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 3
* mozilla-inbound: 2
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-03-14&endday=2016-03-20&tree=all
8 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* linux64: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-03-21&endday=2016-03-27&tree=all
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* linux64: 3
* windows7-32: 1
* osx-10-6: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-03-28&endday=2016-04-03&tree=all
Ed, is there anything we can do to make this failure starrable in Treeherder?
Flags: needinfo?(emorley)
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 6
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-04-04&endday=2016-04-10&tree=all
14 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* fx-team: 6
* try: 1
* ash: 1

Platform breakdown:
* linux64: 7
* windows7-32: 3
* linux32: 3
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-04-11&endday=2016-04-17&tree=all
(In reply to Ryan VanderMeulen [:RyanVM] from comment #12)
> Ed, is there anything we can do to make this failure starrable in Treeherder?

It's already starrable, the bug summary is just incomplete.

The algorithm (https://github.com/mozilla/treeherder/blob/cda4b33cef44192a73546be51c593ba0ad838e05/treeherder/model/error_summary.py#L97-L121) is roughly as follows:

1) Try leak-specific search if line is of form "leakcheck | .*\d+ bytes leaked (Object-1, Object-2, Object-3, ...)"
2) Else if in pipe-delimited format && middle token not blacklisted/too short a string, then search using the middle token
3) Otherwise try the full line

In this case, we reach #3, since the middle token is "leakcheck", which is in the blacklist:
https://github.com/mozilla/treeherder/blob/cda4b33cef44192a73546be51c593ba0ad838e05/treeherder/model/error_summary.py#L171
Flags: needinfo?(emorley)
Summary: Intermittent leakcheck | plugin process: missing output line for total leaks! → Intermittent TEST-UNEXPECTED-FAIL | leakcheck | plugin process: missing output line for total leaks!
26 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* fx-team: 5
* mozilla-aurora: 3
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 16
* linux32: 7
* windows7-32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-04-18&endday=2016-04-24&tree=all
43 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 27
* fx-team: 7
* mozilla-central: 4
* try: 3
* mozilla-aurora: 1
* ash: 1

Platform breakdown:
* linux64: 21
* windows7-32: 15
* linux32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-04-25&endday=2016-05-01&tree=all
35 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 19
* fx-team: 9
* try: 3
* mozilla-aurora: 2
* mozilla-central: 1
* ash: 1

Platform breakdown:
* linux64: 21
* windows7-32: 13
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-05-10&endday=2016-05-16&tree=all
30 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 6
* mozilla-central: 3
* mozilla-beta: 2
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 24
* windows7-32: 5
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-05-16&endday=2016-05-22&tree=all
36 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 18
* mozilla-inbound: 10
* fx-team: 4
* mozilla-central: 1
* mozilla-beta: 1
* mozilla-aurora: 1
* ash: 1

Platform breakdown:
* windows7-32: 20
* linux64: 13
* linux32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-05-23&endday=2016-05-29&tree=all
24 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 15
* fx-team: 6
* try: 3

Platform breakdown:
* linux64: 13
* windows7-32: 6
* linux32: 3
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-05-30&endday=2016-06-05&tree=all
27 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 18
* try: 4
* fx-team: 3
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 21
* windows7-32: 5
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-06-06&endday=2016-06-12&tree=all
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* mozilla-central: 1

Platform breakdown:
* linux64: 4
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-06-13&endday=2016-06-19&tree=all
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 3
* try: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 10
* linux32: 3
* windows7-32: 2
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-06-20&endday=2016-06-26&tree=all
20 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* autoland: 5
* try: 3
* fx-team: 3
* mozilla-central: 1

Platform breakdown:
* linux64: 13
* linux32: 3
* windows7-32-vm: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-06-27&endday=2016-07-03&tree=all
23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* fx-team: 5
* autoland: 5
* mozilla-aurora: 3
* try: 2

Platform breakdown:
* linux64: 13
* linux32: 6
* windows7-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-07-04&endday=2016-07-10&tree=all
22 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* fx-team: 5
* try: 4
* autoland: 4
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 11
* windows7-32: 8
* windows7-32-vm: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-07-11&endday=2016-07-17&tree=all
19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* autoland: 4
* try: 2
* mozilla-central: 2
* ash: 2
* fx-team: 1

Platform breakdown:
* linux64: 12
* linux32: 6
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-07-18&endday=2016-07-24&tree=all
22 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 4
* try: 3
* autoland: 2
* mozilla-central: 1
* mozilla-beta: 1
* ash: 1

Platform breakdown:
* linux64: 14
* linux32: 4
* windows7-32: 3
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-07-25&endday=2016-07-31&tree=all
34 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 18
* autoland: 10
* mozilla-central: 2
* fx-team: 2
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 21
* windows7-32: 6
* linux32: 5
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-08-01&endday=2016-08-07&tree=all
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 11
* fx-team: 2
* try: 1
* mozilla-inbound: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* linux32: 2
* windows7-32: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-08-08&endday=2016-08-14&tree=all
18 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* autoland: 7
* fx-team: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 13
* windows7-32: 3
* windows7-32-vm: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-08-15&endday=2016-08-21&tree=all
31 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 13
* mozilla-inbound: 12
* try: 2
* mozilla-central: 2
* mozilla-aurora: 2

Platform breakdown:
* linux64: 22
* linux32: 5
* windows7-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-08-22&endday=2016-08-28&tree=all

Comment 34

2 years ago
Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.
Priority: -- → P3
29 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* try: 6
* autoland: 5
* mozilla-central: 3
* mozilla-beta: 2
* mozilla-aurora: 2
* fx-team: 1

Platform breakdown:
* linux64: 20
* windows7-32: 5
* linux32: 2
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-08-29&endday=2016-09-04&tree=all
11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 6
* mozilla-inbound: 3
* try: 1
* fx-team: 1

Platform breakdown:
* linux64: 9
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-09-05&endday=2016-09-11&tree=all
I looked at one log, and it looked like there was an assertion.
https://public-artifacts.taskcluster.net/JYpgqo41R-OxQaoEn2ibnQ/0/public/logs/live_backing.log

I wonder why the log parser is missing this. Maybe the weird character.

09:02:04     INFO -  [NPAPI 3029] ###!!! ASSERTION: plug removed: 'glib assertion', file /home/worker/workspace/build/src/toolkit/xre/nsSigHandlers.cpp, line 140
Oh I already noticed that in comment 2...
19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* autoland: 5
* fx-team: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 14
* linux32: 3
* windows7-32-vm: 1
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-09-12&endday=2016-09-18&tree=all
20 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* autoland: 6
* mozilla-beta: 2
* mozilla-aurora: 2
* fx-team: 2
* try: 1

Platform breakdown:
* linux64: 13
* windows7-32: 5
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-09-19&endday=2016-09-25&tree=all
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 4
* try: 1
* mozilla-beta: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* windows7-32: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-09-26&endday=2016-10-02&tree=all
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* try: 4
* autoland: 2
* mozilla-central: 1
* mozilla-aurora: 1
* fx-team: 1

Platform breakdown:
* linux64: 9
* linux32: 5
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-10-03&endday=2016-10-09&tree=all
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 3
* mozilla-beta: 2
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 4
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-10-10&endday=2016-10-16&tree=all
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 5
* try: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2016-10-17&endday=2016-10-23&tree=all
Bug 1239362 is now fixed, but this error is still occurring sometimes, and the "plug removed" assertion is not involved.
Depends on: 1239362
The volume did seem to go down quite a bit after that bug landed.

I looked at the two failures in the last week. If you look at the line after the UNEXPECTED-FAIL line, it gives the PID of the process that failed to create a leak log:
TEST-INFO | leakcheck | missing output line from log file /tmp/tmpDERMDD.mozrunner/runtests_leaks_plugin_pid3384.log

Then you can search for the pid in the log.

For the Linux one, I see something about "received an X Window System error". Maybe that is related? That log is odd because it looks like shutdown gets far enough to print out nsStringStats. It looks like that's happening during dom/html/test/test_fullscreen-api.html.

For the Windows one, I'm not sure if there are any errors in the log around when the plugin process is created, but it looks like it is created during dom/plugins/test/mochitest/test_hangui.xul so maybe something went wrong with that.
(In reply to Andrew McCreight [:mccr8] from comment #46)
> For the Linux one, I see something about "received an X Window System
> error". Maybe that is related? [...] It looks like that's
> happening during dom/html/test/test_fullscreen-api.html.

BadWindow with request_code 3, which is X_GetWindowAttributes, does sound
related, thanks, and likely still due to bug 1237853.

GDK2 (used in the plugin process) is calling exit(1), without using g_error(), and so this is missing our X/GDK error handler and not triggering a crash report.  I suspect it would be possible to InstallX11ErrorHandler() (designed for GDK2) for the plugin process to get slightly better error messages and trigger the crash reporter.
That actually sounds similar to bug 1309420 comment 14: when we assert, we are okay, but when we hit some X windows error, we leak (probably because we exit without shutting things down properly). I just worked around that by disabling the test, which wasn't running in debug builds either.
See Also: → bug 1309420
Depends on: 1331320
5 failures in 812 pushes (0.006 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-inbound: 3
* try: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 3
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-02-20&endday=2017-02-26&tree=all
15 failures in 867 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 4
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 8
* windows8-64: 3
* windows7-32-vm: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-04-03&endday=2017-04-09&tree=all
9 failures in 894 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* mozilla-central: 2
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 4
* windows7-32-vm: 2
* windows7-32: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-04-10&endday=2017-04-16&tree=all
1 failures in 814 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-esr52: 1

Platform breakdown:
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-06-12&endday=2017-06-18&tree=all
6 failures in 892 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 3
* mozilla-inbound: 2
* mozilla-beta: 1

Platform breakdown:
* linux64-ccov: 3
* windows7-32-vm: 1
* windows7-32: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-06-19&endday=2017-06-25&tree=all
2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32-vm: 1
* linux64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-06-26&endday=2017-07-02&tree=all
1 failures in 656 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-07-03&endday=2017-07-09&tree=all
2 failures in 822 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* windows8-64: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-07-17&endday=2017-07-23&tree=all
3 failures in 1008 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2
* mozilla-beta: 1

Platform breakdown:
* windows8-64: 1
* windows7-32-vm: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-07-24&endday=2017-07-30&tree=all
1 failures in 901 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-release: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-08-07&endday=2017-08-13&tree=all
12 failures in 924 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 6
* mozilla-inbound: 4
* try: 1
* autoland: 1

Platform breakdown:
* linux64-ccov: 5
* linux64: 4
* linux64-stylo-disabled: 1
* linux64-stylo: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-09-04&endday=2017-09-10&tree=all
14 failures in 1032 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 13
* autoland: 1

Platform breakdown:
* linux64-ccov: 13
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-09-11&endday=2017-09-17&tree=all
4 failures in 943 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 3
* try: 1

Platform breakdown:
* linux64-ccov: 2
* windows7-32: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-09-18&endday=2017-09-24&tree=all
10 failures in 885 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 7
* try: 1
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows7-32: 3
* linux64: 3
* windows7-32-stylo-disabled: 2
* linux64-ccov: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-09-25&endday=2017-10-01&tree=all
1 failures in 824 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-10-02&endday=2017-10-08&tree=all
3 failures in 947 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2
* mozilla-inbound: 1

Platform breakdown:
* linux64-ccov: 2
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-10-09&endday=2017-10-15&tree=all
1 failures in 912 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-10-23&endday=2017-10-29&tree=all
2 failures in 849 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1
* mozilla-central: 1

Platform breakdown:
* macosx64-stylo-disabled: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-11-06&endday=2017-11-12&tree=all
1 failures in 762 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* macosx64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-11-13&endday=2017-11-19&tree=all
2 failures in 744 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2

Platform breakdown:
* linux64: 1
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-11-20&endday=2017-11-26&tree=all
1 failures in 792 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-11-27&endday=2017-12-03&tree=all
1 failures in 423 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-12-11&endday=2017-12-17&tree=all
2 failures in 147 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr52: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2017-12-25&endday=2017-12-31&tree=all
15 failures in 788 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 15

Platform breakdown:
* linux64-stylo-disabled: 4
* linux32: 4
* osx-10-10: 3
* linux64: 3
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2018-01-08&endday=2018-01-14&tree=all
1 failures in 701 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2018-01-22&endday=2018-01-28&tree=all
2 failures in 702 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2018-02-05&endday=2018-02-11&tree=all
3 failures in 685 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 2
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2018-02-12&endday=2018-02-18&tree=all
2 failures in 814 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1239258&startday=2018-03-12&endday=2018-03-18&tree=all
1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-03-26&endday=2018-04-01&tree=trunk
2 failures in 792 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2

Platform breakdown:
* windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-04-09&endday=2018-04-15&tree=trunk
1 failures in 551 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-04-30&endday=2018-05-06&tree=trunk
4 failures in 660 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 3
* mozilla-central: 1

Platform breakdown:
* linux32: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-05-07&endday=2018-05-13&tree=trunk
1 failures in 639 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr60: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-05-14&endday=2018-05-20&tree=trunk
3 failures in 632 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux32: 2
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-05-21&endday=2018-05-27&tree=trunk
2 failures in 669 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* macosx64-ccov: 1
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-06-04&endday=2018-06-10&tree=trunk
1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-06-25&endday=2018-07-01&tree=trunk
1 failures in 653 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-07-16&endday=2018-07-22&tree=trunk
1 failures in 2561 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-08-05&endday=2018-08-11&tree=all
1 failures in 2579 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-08-06&endday=2018-08-12&tree=all
2 failures in 2568 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows10-64: 1
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-08-20&endday=2018-08-26&tree=all
Recent occurrences here are actually Bug 1497029 as all failure logs have 
22:39:17     INFO -    File "Z:\task_1538864482\build\tests\bin\fix_stack_using_bpsyms.py", line 76, in __init__
22:39:17     INFO -      (rva, size, line, filenum) = line.split(None)

before the errors, so please disregard the next Orange factor comment regarding the failures on the 6th of October.
10 failures in 2708 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 10

Platform breakdown:
* windows7-32: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-10-01&endday=2018-10-07&tree=all
2 failures in 2759 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1239258&startday=2018-10-15&endday=2018-10-21&tree=all
You need to log in before you can comment on or make changes to this bug.