932781 - Trunk trees closed due to OOMs from leaking hundreds of DOMWindows

Reporter

Description

•

11 years ago

Still trying to get context as to the full story here piecing together from the IRC logs: 03:22 philor khuey: remember when you fixed The OOM from storage threads? could you do that again, for whatever it is now, please? 03:23 khuey philor: oh boy 03:23 khuey philor: that was a long time ago ... don't really remember that 03:23 khuey got a bug #? 03:24 romaxa glandium: which revision did you use for https://bugzilla.mozilla.org/attachment.cgi?id=823804 ? 03:25 philor khuey: it was 18/19, yeah, and I don't have any reason to believe it's storage again, but the current inbound closure, bug 929359, and bug 932159 where I hid asan browser-chrome are all about the way we're constantly on the edge of OOM now 03:25 khuey oh, hmm 03:25 khuey is this the thing were we weren't shutting down the threads properly? 03:26 philor it was, yeah, we were keeping 300 threads alive through the life of the test 03:26 khuey yeah 03:27 khuey 802239 03:27 khuey I believe 03:27 khuey firebot: bug 802239 03:27 firebot Bug https://bugzilla.mozilla.org/show_bug.cgi?id=802239 maj, --, mozilla19, khuey, RESO FIXED, mozStorage leaks one thread per connection until shutdown 03:29 philor "we're done with looking at OOM-destroyed runs" - so brave, so foolish, so unaware of his future 03:31 khuey philor: and I guess ASAN isn't kind enough to print out thread stacks when it decides to kill the browser? 03:33 khuey philor: there's also the question of why ASAN is trying to allocate 51 GB of memory ... 03:33 khuey er, sorry 03:33 khuey 32 03:33 khuey MB 03:33 khuey ok that's more reasonable 03:34 khuey although still a lot ... 03:35 philor alas, no, asan doesn't want to print anything, because it doesn't know anything's wrong, all it sees is a mozapps/extensions/ test start, and then timeout, and then another start, and then timeout, etc. 03:37 khuey philor: fun 03:37 khuey philor: do we have any idea what's going on? 03:37 Hughman khuey: that i can do 03:38 khuey njn: is 932159 related to the OOM stuff you were looking at? 03:39 njn khuey: ASAN? doesn't sound related 03:39 khuey njn: well it's an OOM 03:39 njn khuey: bug 929359 is the one I've looked at, it's windows debug mochi-2 failures 03:40 khuey right 03:40 njn khuey: and AFAICT, we've been slightly OOMing there for some time, and a minor change just pushed us over the edge... so it seems unlikely to affect elsewhere(?) 03:40 philor I'd like to call the asan b-c OOM the leading indicator for b-c OOMs to come, but, the tree is closed for b-c OOM so I guess it's not really "to come" 03:41 * philor looks at what we backed out for it, double-takes 03:42 philor every single bit of that is Qt 03:42 philor we backed out a pure-Qt patch for Windows browser-chrome OOM? 03:42 njn woo 03:42 * philor lights a candle, sacrifices a chicken, and looks for a goat 03:43 KWierso philor: yeah 03:43 philor people-waiting-for-inbound-to-open: go to bed or go get drunk, it ain't gonna happen 03:44 glandium philor: it might in 22 minutes 03:45 glandium romaxa: you can get 68c4d885d6dd from try 03:45 froydnj I feel like that belongs on a meme somewhere 03:46 philor if backing out that patch causes Win7 debug browser-chrome to not OOM, I can solve all of our testing capacity problems 03:46 philor stop running tests, buy magic 8-balls 03:46 philor !8ball did I pass tests? 03:46 firebot philor: Of course. 03:46 philor sweet, ship me 03:47 khuey lol 03:47 glandium philor: wait, it was ooming? 03:48 khuey lol 03:50 philor glandium: which was? the inbound Win7 debug b-c closure? (NS_ERROR_FAILURE) [nsIZipReader.open], one of our common OOM markers, yeah 03:50 philor asan b-c? yeah 03:50 khuey it's kind of impressively how thoroughly we die on oom 03:51 khuey assertions, crashes, js exceptions 03:51 khuey everything 03:51 philor win7's m2 leak? yeah, it's OOM from the cycle collector 03:51 khuey hmm 03:51 khuey we do have 700 DOM windows open 03:51 khuey that is not good 03:52 bz That bug sounds like for some reason the popup is using the parent's refresh driver or something???? 03:52 glandium philor: asan b-c is hidden 03:52 philor in fact, right after zipreader fails, "ASSERTION: Ran out of memory while building cycle collector graph" 03:52 bz Oh, are these animated gifs? 03:52 bz OK 03:52 khuey we are clearly GCing 03:52 philor glandium: yeah, I hid it because it got a case of the permaorange, decoder tried to fix it by making it use less memory, it's picked up another incomprehensible failure during the day today, I haven't looked at the new one 03:53 khuey 18:05:02 INFO - --DOMWINDOW == 686 (3DC59B38) [serial = 18590] [outer = 5D391770] [url = about:blank] 03:53 khuey so wtf 03:53 khuey how did we accumulate 700 windows 03:53 njn khuey: it goes over 1000 sometimes 03:53 khuey philor: is this bc thing windows 8 only? 03:53 bz in a test run? 03:53 khuey bz: yeah 03:53 philor khuey: win7 only 03:53 bz tests end up with tons of windows.... 03:53 * bz is not quite sure how 03:53 bz "not enough gc" 03:53 khuey er 03:53 khuey yeah 03:53 philor as is the mochitest-2 thing 03:53 khuey win7 only 03:54 njn khuey: ~800 is the threshold for the win7 debug mochi-2 failure 03:54 roc could some test be leaving a window open 03:54 njn khuey: below that, we're usually ok, above that we fail 03:54 dolske I will laugh if test are slow because they're swapping. :) 03:55 philor yeah, maybe that's why Mac debug is surviving, but taking twice as long as it did on 18 03:55 philor though it does, come to think of it, have a very frequent mochitest-2 shutdown timeout... 03:56 khuey bz: 700 DOM windows that we don't optimize out of the CC graph seems like the kind of thing that would cause a CC oom 03:57 njn philor: my ideas so far haven't worked, so now I'm trying two of them in tandem 03:57 njn philor: oh, and I'm facing East this time 03:57 roc Looks like on Mac we hit 638 DOMWindows 03:57 roc in b-c 03:57 roc or thereabouts 03:57 njn roc: it probably varies significantly 03:57 philor njn: pour a glass of rum for Jambo 03:57 njn roc: on mochi-2, I've seen it range from ~600 to ~1050 03:57 roc njn: regardless, this number is way way way too large 03:58 philor the Mac m2 shutdown timeout I'm looking at happens just after we finish GCing 704 domwindows 03:58 njn roc: indeed; I'd love it if someone fixed that instead of me trying to tweak my pldhash tweaks 03:58 dolske http://i.imgur.com/cjzOhdO.jpg 03:58 roc either there are a lot of windows actually open, in which case tests need to be fixed, or we have a leak, in which case code needs to be fixed 04:00 khuey looks like the number skyrockets during the pb tests 04:01 roc Linux 64 debug has 661 windows 04:01 roc let's see if I can reproduce this 04:02 khuey hmm, or maybe not 04:03 khuey maybe it is just a slow increase 04:03 roc actually Linux 64 debug peaks at over 900 windows in the log I looked at 04:05 khuey roc: which one are you looking at? 04:05 khuey oh 04:05 khuey bc and oth are not the same thing anymore 04:05 khuey I'm looking at the wrong type of log 04:06 njn in mochi-2, test_Range-deleteContents.html seems to open lots of windows 04:06 roc I looked through the mochitest and reftest logs for Linux64 debug. The window counts look reasonable except for mochitest-2 and browser-chrome 04:06 khuey so almost all of these late windows are chrome://global/content/mozilla.xhtml 04:06 * khuey wonders wtf that is 04:07 khuey is that really the about:mozilla page? 04:07 khuey wtf 04:07 njn test_Range-extractContents.html, too 04:07 khuey aha 04:07 njn khuey: you mentioned test_Range previously, IIRC 04:07 khuey and a ton of devtools stuff 04:08 khuey https://khuey.pastebin.mozilla.org/3378934 04:08 khuey that's a successful bc run 04:08 khuey notice all the devtools stuff not getting GCd until the end 04:08 njn is the devtools pane being opened and then not closed? 04:09 khuey well the number of windows seems to suggest no 04:09 khuey since that would be 50 or more devtools pages 04:09 khuey *panes 04:09 philor and we're still orange on the tip 04:09 glandium khuey: as well as about:mozilla and about:blank 04:09 khuey glandium: yeah ... 04:09 philor so the world isn't quite completely insane, that's nice 04:10 njn in mochi-2 shutdown, iframe-between-tests.html shows up a lot 04:10 khuey njn: the good news is if I can reproduce locally I can use my "dump the CC graph and then conditionally breakpoint on release" trick 04:11 glandium heh, it *is* intermittent 04:11 njn khuey: tons of Range-test-iframe.html#[blah blah blah] 04:11 glandium 2 greens out of 5 on m-i tip 04:13 njn yah, those range tests look like they're killing mochi-2 04:13 philor so we just disable Ms2ger/* and devtools/*, and we're good? 04:14 glandium disable all the tests 04:14 philor oh, yeah 04:14 philor since we actually *did* disable devtools/* for the storage threads thing, and just wound up OOM elsewhere 04:16 njn glandium: just disable the range tests, for mochi-2 04:20 roc in b-c, some of the leaked windows are coming from browser/base/content/test/social/browser_social_activation.js 04:20 roc I suspect via http://lxr.mozilla.org/mozilla-ce...ents/social/FrameWorker.jsm#195 04:22 glandium roc: haha, social 04:22 * philor should have blamed markh from the start 04:23 khuey njn: if I run the devtools tests locally I get assertions about extra shutdown CCs 04:23 * markh ducks 04:23 khuey anyways, lunch time 04:23 roc well 04:23 roc devtools leaks a lot too 04:37 * njn finishes the shortest MemShrink report yet 04:51 njn I like how somebody retriggered win7-mochi-2 twice in the most recent m-i push to get it to fail 04:52 njn "oh, you think you can pass, hey? Take that! Er... take that! Yeah, thought so." 04:52 nigelb njn: heh 04:53 nigelb Isn't that normal? I mean, I've seen sheriffs do that. 05:06 philor I like big retriggers 05:07 philor thus the multiple tens I've got spread around right now 05:10 khuey njn: so after running the inspector tests I have 96 surviving nsGlobalWindows 05:11 njn khuey: is that higher or lower than expected? 05:11 khuey njn: well if we weren't leaking we'd expect ~12 05:12 njn khuey: I see 05:12 njn khuey: is this relevant for the bc OOMs? 05:12 njn khuey: because the mochi-2 OOMs seem to be range-related 05:12 khuey njn: right 05:37 nigelb philor: what's going on with inbound now? are we waiting for the tests to pass after the backout or something? 05:38 philor nigelb: no, the backout did nothing, we're waiting to see whether any of my deeper and wider retriggers show anything 05:38 nigelb Ah, ok 06:01 glandium njn: do you know much about the gdbserver feature of valgrind? 06:01 njn glandium: very little, unfortunately 06:01 njn why do you ask? 06:02 glandium njn: because sewardj is not around and because valgrind doesn't allow me to attach when it hits invalid reads 06:03 dholbert KWierso, not that you should still be working, but on the off chance that you are: any idea about the prospects for inbound reopening? :) 06:03 njn doesn't allow? how 06:03 njn ? 06:03 KWierso dholbert: 302 philor :) 06:03 philor dholbert: see above, where I recommended either going to bed or getting drunk, your choice 06:04 dholbert philor, I suspected as much. :) sounds like a plan 06:04 philor funzors, our first OOM in browser_dragdrop.js was October 1st 06:04 KWierso I suggest both options 06:05 philor this will make it more difficult to find a truly all-green push only on inbound to revert to 06:05 njn philor: good thing we branched yesterday 06:05 * philor backspaces repeatedly 06:06 philor yeah, that would rather be the story of our branching lives 06:06 Mook may I suggest 9b2a99adc05e? 06:07 philor heh 06:07 glandium njn: it doesn't leave me time to do it 06:08 philor bwc has absolutely no idea where I live, right? 06:08 philor let's keep it that way 06:08 philor because I'm starting to like the idea of backing him out for the second day in a row 06:09 glandium njn: forget it, i'm dumb 06:14 philor actually, the right parallel probably isn't the storage threads, it's the Win PGO linker memory 06:15 philor somebody's going to have to fix this, it won't be quick, and in the meantime we can probably land approved patches that cause us to use less memory, and nothing else 06:16 philor sweet, got one leak on m-c tip that's probably from a CC OOM 06:16 glandium philor: so, can i land build system changes? :) 06:16 philor glandium: no, you may never land another thing 06:16 glandium boohoo 06:16 philor no, on second thought, you can land build system changes that cause us to not build things 06:17 glandium philor: i can land build system changes that cause us to not test things ;) 06:18 philor leaked mozapps/extensions/ stuff, strike one 06:18 philor ASSERTION: Extra shutdown CC 06:19 philor and the stuff that appears to be killing asan, wonder what decoder's try parent actually was 06:21 philor but a piddly 658 domwindows 06:23 philor so, did the first thing on inbound above the last merge to central push us over the cliff? 06:23 philor or did bwc? 06:24 philor or are we already over the edge on central and running on air without realizing we're about to fall? 06:40 capella the sky is falling? 06:43 njn philor: I've got some ACME brand rockets in a wooden box over here 06:43 njn just strap one of those to your back 06:43 philor njn: coincidence, I've got some ACME roller skates! 06:49 philor does anyone have a persuasive reason why m-c and fx-team should stay open? 06:49 khuey no 06:50 bz leprechauns? 06:50 khuey bah no roc 06:50 philor guess I'll file something with a crappy comment 0, to have something to point at in the closure message 06:51 bz "leprechauns" sounds like a good comment 0 06:51 khuey bz: are you in CA? 06:51 khuey or just awake very late? :-P 06:51 bz Latter 06:51 * bz was debugging some matlab code and rocking a baby to sleep 06:51 bz She's finally asleep 06:51 bz now to emulate her.... ;) 06:52 bz The good news is my patches are green on try and have reviews 06:52 bz The bad news.... 06:52 glandium bz: yeah, i have 10 build system patches waiting to land too 06:53 glandium that are green on try 06:53 khuey glandium: philor probably hasn't remembered to close b-s 06:54 bz aurora is open, I bet! 06:54 philor don't think it's even in treestatus 06:54 glandium khuey: i don't want to have to merge b-s to m-c or m-i afterwards. I don't think it's in a state to be merged gracefully. 06:55 philor yeah, aurora's oddly green, despite having every reason to have permaorange asan b-c, and nearly permaorange win7 debug m2 and b-c 06:55 bz All these worries about whether the code will compile after merging.... 06:55 khuey glandium: have we even used b-s in the last 6 months? 06:55 bz_sleep It's clearly bedtime. ;) 06:55 glandium khuey: it was used last month 06:55 khuey ok 06:56 glandium khuey: i'd rather not pollute m-c with its history 06:56 khuey oh what did we do to it? 06:57 glandium khuey: probably nothing shocking, but i hate to merge separate branches that add nothing just for fun 06:58 glandium that being said, i really don't see why build system only changes couldn't get approval to land on m-i 06:58 glandium philor: ^ 06:59 philor glandium: if you want to take over, just say the word 06:59 glandium philor: take over what? 06:59 philor it's midnight, and I'm typing up a bug that I totally don't understand because nobody else will 07:00 philor if you want to sheriff the trees, and retrigger enough to verify that you haven't fucked a fucked tree even worse than it's already fucked, because you just have to get something landed and out of your tree, take over 07:00 philor all of it 07:00 philor good night, fuck this 07:01 khuey so I fixed one path where the devtools tests are leaking 07:01 khuey but it didn't help and I can't find any more :-( ... 07:35 roc khuey: did you get anywhere with the leak? 07:35 khuey roc: well I fixed a couple things 07:35 khuey still dealing with more ... 07:35 roc cool 07:37 roc maybe tests should fail if we hit 500 DOMWindows or something like that 07:38 khuey roc: we really need to hueyfix chrome->chrome stuff 07:43 Tomcat|sheriffduty so the leaks are stopping us from reopen inbound right 07:44 khuey yes 07:44 Tomcat|sheriffduty like bug 929359 and friends 07:44 Tomcat|sheriffduty thx khuey was trying to get a picture what happened over hte night here 07:45 khuey bad things :-) 07:45 * khuey is poking at some leaks now ... 10:21 Tomcat|sheriffduty wow leaked 273 DOMWINDOW(s) on the last run from m-c before the closure 10:22 khuey well that's cause the cycle collector started working 10:22 khuey Tomcat|sheriffduty: ok, so your first mission is to disable the testRange tests from m2 again

Ed Morley [:emorley]

Reporter