Closed Bug 637279 Opened 14 years ago Closed 14 years ago

OOM termination due to adverticum ads on Hungarian sites, hasznaltauto.hu, vatera.hu, menetrendek.hu [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ]

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED DUPLICATE of bug 654707

People

(Reporter: scoobidiver, Unassigned)

References

Details

(Keywords: crash, regression)

Crash Data

Attachments

(1 file)

unversioned .dlls for this sig on 20110327
9.08 KB, text/plain
Details
It is a new crash signature in the trunk. It is currently #37 top crasher in 4.0b12. Signature mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) UUID 834065b8-7001-486e-81c2-b5be42110228 Time 2011-02-28 01:58:51.636339 Uptime 100 Install Age 8368 seconds (2.3 hours) since version was first installed. Product Firefox Version 4.0b12 Build ID 20110222210221 Branch 2.0 OS Windows NT OS Version 6.1.7600 CPU x86 CPU Info GenuineIntel family 6 model 15 stepping 2 Crash Reason EXCEPTION_BREAKPOINT Crash Address 0x69681a39 App Notes AdapterVendorID: 10de, AdapterDeviceID: 10c3, AdapterDriverVersion: 8.17.12.6099 Frame Module Signature [Expand] Source 0 mozalloc.dll mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:77 1 mozalloc.dll mozalloc_handle_oom memory/mozalloc/mozalloc_oom.cpp:54 2 xul.dll nsHtml5Tokenizer::stateLoop parser/html/nsHtml5Tokenizer.cpp:734 3 xul.dll nsHtml5Tokenizer::tokenizeBuffer parser/html/nsHtml5Tokenizer.cpp:391 More reports at: https://crash-stats.mozilla.com/report/list?range_value=4&range_unit=weeks&signature=mozalloc_abort%28char%20const*%20const%29%20|%20mozalloc_handle_oom%28%29%20|%20nsHtml5Tokenizer%3A%3AstateLoop%28int%2C%20unsigned%20short%2C%20int%2C%20unsigned%20short*%2C%20int%2C%20int%2C%20int%29
This is an OOM crash under here: 734 addAttributeWithValue(); But that function is presumably inlined?
It is currently #20 top crasher in 4.0. Chris, can we get URLs. Hungarian sites as in bug 633473?
stats 379 total crashes for mozalloc_abort.char.const..const....mozalloc_handle_oom.....nsHtml5Tokenizer::stateLoop.int,.unsigned.short,.int,.unsigned.short.,.int,.int,.int. on 20110317-crashdata.csv 7 startup crashes inside 30 sec. 170 startup crashes inside 3 min. 149 repeated crashes inside 3 min. of last crash most are hitting between 4am and 2pm pacific time correlation to time of day -- hourly frequency of reports 4 2011031700 5 2011031701 14 2011031702 13 2011031703 19 2011031704 21 2011031705 14 2011031706 19 2011031707 23 2011031708 36 2011031709 31 2011031710 36 2011031711 42 2011031712 33 2011031713 25 2011031714 8 2011031715 2 2011031716 8 2011031717 4 2011031718 2 2011031719 2 2011031721 8 2011031722 10 2011031723 pct. all 372195 379 0.00101828 4.0 94950 332 0.00349658 4.0b12 12432 45 0.00361969 4.0b13pre1303 1 0.00076746 4.0b11 5746 1 0.000174034 domains of sites 144 http://www.hasznaltauto.hu 1 http://www.hasznaltauto.hu/auto/renault/megane/renault_megane_1.4_rl-4587752 1 http://www.hasznaltauto.hu/auto/peugeot/206/peugeot_206_sw_1.4_presence_szep_megkimelt_allapotban-4553766 1 http://www.hasznaltauto.hu/auto/opel/astra_h/opel_astra_h_caravan_1.7_cdti_enjoy_25000km-4585996 1 http://www.hasznaltauto.hu/auto/opel/astra_g/opel_astra_g_1.7_td-4573093 1 http://www.hasznaltauto.hu/auto/opel/astra_f/opel_astra_f_caravan_gl_1.4-16v-4466560 1 http://www.hasznaltauto.hu/auto/opel/astra_f/opel_astra_f_1.6_si_cd-4592735 1 http://www.hasznaltauto.hu/auto/opel/astra/opel_astra_1.8_gt-4590506 1 http://www.hasznaltauto.hu/auto/moszkvics/aleko/moszkvics_aleko_21412-4593687 1 http://www.hasznaltauto.hu/auto/mitsubishi/lancer/mitsubishi_lancer_2.0_evo_viii_mr-4540701 1 http://www.hasznaltauto.hu/auto/mitsubishi/asx/mitsubishi_asx_1.6_mivec_invite-4211512 1 http://www.hasznaltauto.hu/auto/mitsubishi/3000_gt/mitsubishi_3000_gt_tura_sport-4592171 1 http://www.hasznaltauto.hu/auto/mercedes-benz/e_270/mercedes-benz_e_270_cdi_avantgarde_automata-4527015 1 http://www.hasznaltauto.hu/auto/mercedes-benz/e_200/mercedes-benz_e_200_cdi_classic_automata-4068112 1 http://www.hasznaltauto.hu/auto/mercedes-benz/c_180/mercedes-benz_c_180_classic-4408871 1 http://www.hasznaltauto.hu/auto/land_rover/range_rover/land_rover_range_rover_3.6_tdv8_vogue_automata-4448277 1 http://www.hasznaltauto.hu/auto/lada/4x4/lada_4x4-4435397 1 http://www.hasznaltauto.hu/auto/jeep/grand_cherokee/jeep_grand_cherokee_4.0_limited_automata_beleszeretsz-4595863 1 http://www.hasznaltauto.hu/auto/honda/civic/honda_civic_1.5i_vtec_ls_abssrsklima-4548550 1 http://www.hasznaltauto.hu/auto/ford/mondeo/ford_mondeo_tdci_trend_akcio-4598926 1 http://www.hasznaltauto.hu/auto/ford/galaxy/ford_galaxy_2.0_tdi_ghia_40000_km-4231827 1 http://www.hasznaltauto.hu/auto/ford/fiesta/ford_fiesta-4570773 1 http://www.hasznaltauto.hu/auto/citroen/c3_pluriel/citroen_c3_pluriel_1.6_sensodrive_csere_beszamitas-4567958 1 http://www.hasznaltauto.hu/auto/citroen/berlingo/citroen_berlingo_1.8_sx_klima_ketoldali_toloajto-4574314 1 http://www.hasznaltauto.hu/auto/chevrolet/camaro/chevrolet_camaro_rally_sport-3850850 1 http://www.hasznaltauto.hu/auto/bmw/520/bmw_520_520i_24v-4483658 1 http://www.hasznaltauto.hu/auto/bmw/318/bmw_318_318d-4563057 1 http://www.hasznaltauto.hu/auto/audi/tt/audi_tt_coupe_1.8_t_quattro_s-line_225le-4402779 1 http://www.hasznaltauto.hu/auto/audi/s4/audi_s4_avant-4592297 1 http://www.hasznaltauto.hu/4544128 1 http://www.hasznaltauto.hu/4540984 39 http://www.vatera.hu searches for nike, lcd monitor, galaxy, nokia+n96, and many other products with urls like http://www.vatera.hu/listings/index.php?q=nike ... and other sites. 26 http://divat-ruha.vatera.hu 19 http://www.menetrendek.hu 16 http://elektronika.vatera.hu 13 http://szamitastechnika.vatera.hu 10 http://auto-motor.vatera.hu 9 http://epitkezes-otthon.vatera.hu 7 http://jatek.vatera.hu 7 http://gyujtemeny.vatera.hu 7 \N// 6 http://sport.vatera.hu 6 http://mobil-telefon.vatera.hu 6 http://allateledel.vatera.hu 5 http://ekszer-ora.vatera.hu 3 http://zene.vatera.hu 3 http://www.youtube.com 3 http://www.facebook.com 3 http://menetrendek.hu 3 http://antik.vatera.hu 2 https://www8.receita.fazenda.gov.br 2 http://www8.receita.fazenda.gov.br 2 http://www.szoftverbazis.hu 2 http://www.freebase.com 2 http://utazas-szabadido.vatera.hu 2 http://letoltes.prim.hu 2 http://haztartasigep.vatera.hu 2 http://fenykepezo-kamera.vatera.hu 2 http://fanfic.hu 1 https://www.facebook.com 1 http://www.veteran.hu 1 http://www.uni-miskolc.hu 1 http://www.serebii.net
Depends on: 633473
Summary: Crash [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ] → Crash [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ] mainly at hasznaltauto.hu and menetrendek.hu
A few user complained about this crash (sorry only Hungarian references). I was not able to reproduce under Linux (Ubuntu 10.10/32 bit). One of our users created a video: http://www.youtube.com/watch?v=3Xgk_9M0gKE It demonstrates Firefox crash when you select a link then go back quickly causes crash. I don't know if it is related the original issue. Please increase prio and fix the bug ASAP because vatera.hu and hasznaltauto.hu are popular websites.
So can anybody reproduce using comment 4 to help?
Kami, some actual steps to reproduce would be good.... Specific ones: which exact url to load, which exact link to click. Though I think it's quite odd that hitting back would trigger and out-of-memory crash! That said, vatera.hu does show up in the list in comment 3... but you did double-check that the crash reports that user submitted showed this crash?
When users figure out how to hit this, it looks like they hit it consistently. checking --- mozalloc_abort.char.const..const....mozalloc_handle_oom.....nsHtml5Tokenizer::stateLoop.int,.unsigned.short,.int,.unsigned.short...int,.int,.int. 20110326-crashdata.csv found in: 4.0 4.0b12 4.0b10 os breakdown mozalloc_abort.char.const..const....mozalloc_handle_oom.....nsHtml5Tokenizer::stateLoop.int,.unsigned.short,.int,.unsigned.short...int,.int,.int.Total 1232 Win5.1 0.58 Win6.0 0.05 Win6.1 0.37 Correlation to startup or time of session 1232 total crashes for mozalloc_abort.char.const..const....mozalloc_handle_oom.....nsHtml5Tokenizer::stateLoop.int,.unsigned.short,.int,.unsigned.short...int,.int,.int. on 20110326-crashdata.csv 31 startup crashes inside 30 sec. 558 startup crashes inside 3 min. 470 repeated crashes inside 3 min. of last crash I did a google translate on bunch of comments for the last few days and didn't see any matches for users commenting on hitting the back button, but several users commented on hitting the crashes there repeatedly. the concentration around .hu sites continues but there is a very long tail of specfic urls. These are the only urls with more than 3 repeats out of 1200+ we collected yesterday. 37 http://www.menetrendek.hu/cgi-bin/menetrend/html.cgi 29 \N 25 http://www.hasznaltauto.hu/kereso 18 http://www.hasznaltauto.hu/ 11 http://imgs.adverticum.net/external/1710535.html?goa3&v&externalhost=http%3A%2F%2Fimgs.adverticum.net%2Fexternal&statichost=http%3A%2F%2Fimgs.adverticum.net%2Fbanners&location=http%3A%2F%2Fimgs.adverticum.net%2Fbanners%2F1710532%2F&target =_blank&timestamp 8 http://menetrendek.hu/cgi-bin/menetrend/html.cgi 7 http://www.hasznaltauto.hu/ujkereses/auto 5 http://www.szoftverbazis.hu/ 5 http://www.hasznaltauto.hu/szukites 4 http://www.vatera.hu/ 3 http://www.kapu.hu/ 3 http://www.hasznaltauto.hu/kereso/motor 3 http://www.hasznaltauto.hu/kereso/kishaszonjarmu 18% have avast AV 18% (163/900) vs. 11% (42647/388604) snxhk.dll 21% have RadioWMPCore.dll 21% (188/900) vs. 14% (54658/388604) RadioWMPCore.dll which appears like it might have been distributed by current Conduit tool bars, or might have come from 34 conduit tool bars that might have been removed off of AMO. RadioWMPCore.dll gets dropped in places like Directory: %APPDATA%\Mozilla\Firefox\Profiles\5s6uj0hw.default\extensions\{66f2e20d-0da8-4c11-a9c8-dd8477b88acd}\components\RadioWMPCore.dll by about addons found in the list here: https://mxr.mozilla.org/addons/search?string=RadioWMPCore.dll some examples /207645/META-INF/manifest.mf line 51 -- Name: components/RadioWMPCore.dll https://addons.mozilla.org/en-US/firefox/search/?q=Swiss-Banks.org+Finance+Toolbar&cat=all&x=0&y=0 (no longer available on amo) /229912/META-INF/manifest.mf line 51 -- Name: components/RadioWMPCore.dll https://addons.mozilla.org/es-Es/firefox/addon/tugatech-toolbar/ (no longer available on amo) /58198/META-INF/manifest.mf line 51 -- Name: components/RadioWMPCore.dll this one is still around... https://addons.mozilla.org/en-us/firefox/addon/tv-fox-watch-tv-online/ 38% (338/900) vs. 28% (109173/388604) explorerframe.dll these might be playing around with windows theme'ing. hard to tell if that might have also been going on in the video referenced in comment 4.
jorge, what is the current policy on conduit addons on amo? we are seeing RadioWMPCore.dll hanging around in many firefox 4.0 top crashes and it looks like this comes along with many (all?) conduit addons. do we block some of these conduit addons that might have been removed?
Conduit add-ons are allowed, and none are blocklisted as far as I know. If the crashes are happening to add-ons that have already been disabled, I think it's worth considering blocklisting them. If there's a problem with current versions of the DLL, we should try to work with Conduit to get it fixed.
> we should try to work with Conduit to get it fixed? do you have a contact? we don't have many details to start the conversation, but we do see a bunch of out of memory conditions that cause crashes with signatures like this one and others with "mozalloc_abort(char const* const) | mozalloc_handle_oom()" where RadioWMPCore.dll is on the module list. If they can give us more details about what that .dll does we might be able to work with the crash URL list to construct some test cases.
I forgot, but a sample of Adware.Ezula that we found in the wild also seems to be distributing conduit addons and RadioWMPCore.dll see https://bugzilla.mozilla.org/show_bug.cgi?id=633445#c35 and below for other things underway to try and put AV blocks in place. So its possible the frequently crashing systems with this signature might also be affected by a regional malware outbreak in Hungary. That would explain the ease with which some are able to reproduce and the difficulty that others are having. At first glance I don't see randomly named .dlls in the module list like we saw with bug 633445. more digging required to example a larger sample of stacks and module lists.
(In reply to comment #6) > Kami, some actual steps to reproduce would be good.... Specific ones: which > exact url to load, which exact link to click. > > Though I think it's quite odd that hitting back would trigger and out-of-memory > crash! That said, vatera.hu does show up in the list in comment 3... but you > did double-check that the crash reports that user submitted showed this crash? As a Linux user I was not able to reproduce it. Users from the itcafe.hu said the crashes caused the advertisements of the page on Windows boxes. With AdBlock they were able to avoid the creshes. I am not sure about the two (many of crashreports and click on link and quick back) are same problem.
I asked Hungarian users of the ITCafe forum to search for RadioWMPCore.dll. I hope they will answer soon. Can I anything do for lighting fast elimination of this bug? It seems this the only negative tone of Firefox 4 launch.
(In reply to comment #14) > Bug 633445 comment 33 looks related. I ran the same report for a sample of these crashes. we don't see the randomly named unversioned .dll's so much here, but we do see frequent instances of some of the other unversioned .dlls that are hooking into the process. RadioWMPCore.dll UnlockerHook.dll frozen.dll RocketDock.dll UberIcon.dll achook.dll BTKeyInd.dll DockShellHook.dll dadkeyb.dll googletoolbar-ff4.dll BTKeyInd.dll AudioService_ff4.dll rpchromebrowserrecordhelper.dll Nv3DVStreaming.dll
(In reply to comment #10) > If they can give us more details about what that .dll does we might be able to > work with the crash URL list to construct some test cases. According to the developers, the DLL is just a wrapper for Windows Media Player in order to play radio. If there are any specific calls or code I should be looking for, you can let me know and I'll figure it out with the developers. They are copied on the bug now.
if there is a test case that exercises the RadioWMPCore.dll in use to play radio, or some live sites that they know of that use the .dll that might be a start. some instructions on which addons or packages that we can install to get things set up to run the test cases or example sites would be good as well.
They gave me this link: http://newyorkradio.myradiotoolbar.com/xpi. It's a direct link to an XPI that includes the radio feature (haven't tested it myself). The radio feature can be tested just by hitting "play" on the toolbar.
Have you got any ideas what can I tell to our Hungarian users?
> Have you got any ideas what can I tell to our Hungarian users? * http://support.mozilla.com/en-US/kb/Is%20my%20Firefox%20problem%20a%20result%20of%20malware * Use another browser for these few faulty sites.
I'd still really appreciate an answer to my question from comment 6...
(In reply to comment #20) > * Use another browser for these few faulty sites. Do we actually have any evidence that the sites are "faulty" here? (As opposed to a malware outbreak in Hungary manifesting on Hungarian sites, because users in Hungary are more likely to browse Hungarian sites.) "Use another browser" is rather bad advice if the problem is malware on the users' computers.
Removing any Conduit Toolbars from your system is probably the best advice. The are implicated in this crash and many others that we see in the top crash list.
Ok, It seems I found a machine that crash on hasznaltauto.hu site. What information, tool, debuig version need to track down the problems?
I assume it crashes with this stack, right? Does it crash in safe mode? Past that, I think you should ask Henri what would be most useful to look for.
Hi! I did few test, and the finding are worse than I expected (www.hasznaltauto.hu. Firefox crashes aftwer few click. It is easy to reproduce just select type of car (Gyártmány) for example Alfa Romeo. the hit Search (Keresés) button. After that click on few linkg, go back, click on few links, do search again with Keresés. I am sure it will crash whitin a minute ot two. I tried on Windows XP - normal mode - normal mode (with adblock) - safe mode - safe mode without flash and all cases produced crash. I filled the crash reporter as kami911 with text "is it from the top ten crasher?" I hope it will help. KAMI
(In reply to comment #24) > Ok, It seems I found a machine that crash on hasznaltauto.hu site. What > information, tool, debuig version need to track down the problems? Getting a crash in a debugger in a debug build would be useful in order to confirm that code inlining in release builds isn't causing the crash stats to implicate the wrong line in the HTML tokenizer. Do you have an environment set up for running a debug build in a debugger? Debug builds are available from ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/ (In reply to comment #25) > I assume it crashes with this stack, right? Does it crash in safe mode? > > Past that, I think you should ask Henri what would be most useful to look for. If the stacks visible today are correct and the crash happens in inlined attributeNameComplete() (different from the function implicated in comment 1), I'd look for an attribute name that is not one of the standard HTML attribute names and that is very long.
OK. I can reproduce the crash on clean (no extensions, no malware) Windows 7 and Windows XP systems using the instructions in comment 26.
Hi Henri, It is great news. Can you tell me an expected time for the fix? I read several complains about it on Hungarian forums. Is this will fix also the vatera.hu site related problems?
(In reply to comment #29) > It is great news. Can you tell me an expected time for the fix? So far, none of the crashes I've seen with the steps to reproduce have had nsHtml5Tokenizer::stateLoop on the stack. Instead, they seem to be OOM crashes wherever (style system, layout, networking). So it looks like this isn't necessarily an HTML parser problem at all. It's so far unclear to me, what makes www.hasznaltauto.hu exhaust memory, so I can't give any estimate about time to fix. I wonder if all the affected sites share a third-party advertising script. Otherwise, it doesn't make sense that this happens particularly on Hungarian sites.
FWIW, I'm not seeing runaway memory consumption on Linux with the steps to reproduce.
Crashes with the steps to reproduce: http://crash-stats.mozilla.com/report/index/bp-417ba07f-0935-4a96-9c60-cff1b2110405 http://crash-stats.mozilla.com/report/index/bp-7fe85ec9-2245-4b51-8768-637852110405 http://crash-stats.mozilla.com/report/index/bp-94f00c9a-2b14-4229-8819-9a5682110405 http://crash-stats.mozilla.com/report/index/bp-be463394-6473-4e3a-aa96-ee69c2110405 http://crash-stats.mozilla.com/report/index/bp-e49721ae-517a-4386-81a0-838362110405 The memory numbers that Resource Monitor shows when the process dies aren't particularly elevated and the process dies without exhausting system memory. That is, the process consistently experiences OOM as judged by mozalloc, but viewed from the outside, it shouldn't be OOMing. CCing cjones, because I'm ignorant about the expected and designed behavior of mozalloc on Windows.
The comments are complaining about the crash. I think you shouldn't translate them directly because some of them contain bad words ;oS Generally it is about mainly hasznaltauto.hu and vatera.hu. One of two mentioning after use of back button and plug-ins. They also reported it is not happened with 3.6.x. After upgrading to 4.0 it happens 3-8 times a day. Can we write to users as soon as the fix is available? Maybe we need official news about this problem and that we are working on it.
(In reply to comment #33) > Maybe we need official > news about this problem and that we are working on it. Let's not over-announce the "working on it" part, since we don't yet have firm enough an idea of what needs fixing. So far, it is is known that there's a problem of memory exhaustion from the point of view of the memory allocator, but it isn't yet known what's consuming memory and why the allocator fails to allocate more memory when there still seems to be more memory available on the system. FWIW, the crashes linked to in comment 32 were all on a virtual machine that had 512 MB of RAM.
Except for http://www.uni-miskolc.hu/ , the Hungarian sites in comment 3 all include goa3.js in order to use show ads by http://adverticum.com/ The sites that crash the most also both use "median" as an analytics provider.
Using Adblock Plus to block goa3.js (both copies on www.hasznaltauto.hu!) makes the crash go away.
uni miskolc is a homepage of a University. I can confirm there is a company called Medián and they have webaudit system. I am not sure it is a advertisement/banner program. AFAIR it is auditing only system. It is popular in Hungary. http://www.webaudit.hu/ However goa3.js itself an another advertisement company's script called Adverticum. I found references in their site how to enable Adverticum monitoring for websites.
(In reply to comment #19) > Have you got any ideas what can I tell to our Hungarian users? For now, using Adblock Plus to block requests to http://forrest.adverticum.net/ seems to be the best way to avoid this crash.
Component: HTML: Parser → General
QA Contact: parser → general
Summary: Crash [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ] mainly at hasznaltauto.hu and menetrendek.hu → OOM termination due to adverticum ads on Hungarian sites, mainly at hasznaltauto.hu, vatera.hu and menetrendek.hu [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ]
This crash still happens if I use the old parser instead of the new one.
I sent notify to Adverticum via e-mail.
Henri, which memory numbers were you looking at? Is it possible that we're running out of address space (due to mmapped stuff) but not ram?
I'm the main developer of the mentioned advertisement code 'goa3.js'. This code runs on multiple hungarian websites without any problem, but as Kami reported we can reproduce the OOM problem on the mentioned websites. We would love to help you to solve this issue and discover the source of this bad situation in these circumstances. If you need any further information please let me know!
The crazy thing is if you block the follow css in AdBlock the bug is not reproducible and the site is usable and the advertisements are visible. http://hasznaltauto.medija.hu/static/css/hasznaltauto_201103071600.css
Oops it was a little bit early it crashed but much later (after around 10 search and back)
With the new summary without [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ], this bug is no longer displayed in crash stats and in about:crashes.
(In reply to comment #42) > Henri, which memory numbers were you looking at? I was looking at the Memory tab in Resource Monitor on Windows 7. > Is it possible that we're > running out of address space (due to mmapped stuff) but not ram? What's the best way to find out? (In reply to comment #43) > I'm the main developer of the mentioned advertisement code 'goa3.js'. > > This code runs on multiple hungarian websites without any problem, but as Kami > reported we can reproduce the OOM problem on the mentioned websites. > > We would love to help you to solve this issue and discover the source of this > bad situation in these circumstances. If you need any further information > please let me know! Is there a third type of ad besides a JPEG ad and a Flash ad that goa3.js may put on the affected sites? (After loads that *didn't* crash, I saw both JPEG ads and Flash ads. I didn't have Flash installed, though.) That is, should the expectation be that there's a third type of ad that crashes or should the assumption be that one of these two types is displayed when the crash occurs? Also, it would be useful to find the smallest page that uses goa3.js and triggers the out-of-memory situation. (In reply to comment #46) > With the new summary without [@ mozalloc_abort(char const* const) | > mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, > unsigned short*, int, int, int) ], this bug is no longer displayed in crash > stats and in about:crashes. nsHtml5Tokenizer::stateLoop isn't really part of the cause of the crash here. There are various stacks for this OOM crash. The likely reason why nsHtml5Tokenizer::stateLoop may look more common than the other stacks is that especially with inlining, nsHtml5Tokenizer::stateLoop is probably one of the largest methods on the common page load path, so if the browser is going to run out of memory, the largest method is the most likely to be the one to make the last allocation that exhausts the already low memory. Anyway, I've restored the signature.
Summary: OOM termination due to adverticum ads on Hungarian sites, mainly at hasznaltauto.hu, vatera.hu and menetrendek.hu [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] → OOM termination due to adverticum ads on Hungarian sites, hasznaltauto.hu, vatera.hu, menetrendek.hu [@ mozalloc_abort(char const* const) | > mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, > unsigned short*, int, int int) ]
(In reply to comment #47) > Is there a third type of ad besides a JPEG ad and a Flash ad that goa3.js may > put on the affected sites? (After loads that *didn't* crash, I saw both JPEG > ads and Flash ads. I didn't have Flash installed, though.) There are HTML/XHTML ads served by the goa3 tag on these sites. These kind of ads contain mostly (~90%) javascript tags coming from other third-party adservers that are serving Flash contents usually. > That is, should the > expectation be that there's a third type of ad that crashes or should the > assumption be that one of these two types is displayed when the crash occurs? > So there is a third type, but I do not know whether it causes the crashes or there's no connection at all. Trying my best to find out.
(In reply to comment #42) > Is it possible that we're > running out of address space (due to mmapped stuff) but not ram? Probably not, because this also happens with a 64-bit build.
Component: General → HTML: Parser
QA Contact: general → parser
Sorry. I didn't mean to change the component. I believe form history accidentally changed it for me.
Component: HTML: Parser → General
QA Contact: parser → general
(In reply to comment #48) > > That is, should the > > expectation be that there's a third type of ad that crashes or should the > > assumption be that one of these two types is displayed when the crash occurs? > > > > So there is a third type, but I do not know whether it causes the crashes or > there's no connection at all. Trying my best to find out. According to our tests it doesn't matter which ad type is served at the time of the crash.
This looks like a dup of bug 637166 or vice versa. Bug 637166 comment 3 and Bug 637166 comment 5 have a bit of analysis. The symptoms might have changed in the meantime, but I couldn't make any further progress on bug 637166 without a heap profile. I'm still not set up to grab an xperf profile, but if anyone else could get one, that would help immensely.
While debugging the problem KARASZI István (the main developer of goa3.js) has found out that the bug is somehow connected with the GlobalStorage function of Firefox4. If we disable the corresponding part in the invocation script, then Firefox4 stops crashing. If we turn it on again it keeps crashing on hasznaltauto.hu. We've installed a new version which is not using the GlobalStorage, but we stored a copy of the previous version in the following location to help you to find the real problem: http://imgs.adverticum.net/scripts/goa3.ff4/goa3.js I really hope it will help and the problem can be solved soon on the sites with and without goa3.js as well.
(In reply to comment #52) > This looks like a dup of bug 637166 or vice versa. Bug 637166 comment 3 and > Bug 637166 comment 5 have a bit of analysis. Bug 637166 comment 3 talks about updating Flash at a high rate. *This* OOM happens even without Flash installed. (In reply to comment #53) > While debugging the problem KARASZI István (the main developer of goa3.js) has > found out that the bug is somehow connected with the GlobalStorage function of > Firefox4. If we disable the corresponding part in the invocation script, then > Firefox4 stops crashing. If we turn it on again it keeps crashing on > hasznaltauto.hu. > We've installed a new version which is not using the GlobalStorage, but we > stored a copy of the previous version in the following location to help you to > find the real problem: > > http://imgs.adverticum.net/scripts/goa3.ff4/goa3.js Thank you! CCing the developers who have changes globalStorage backing code in the Firefox 4 development cycle.
Component: General → DOM
QA Contact: general → general
(In reply to comment #54) > (In reply to comment #52) > > This looks like a dup of bug 637166 or vice versa. Bug 637166 comment 3 and > > Bug 637166 comment 5 have a bit of analysis. > > Bug 637166 comment 3 talks about updating Flash at a high rate. *This* OOM > happens even without Flash installed. Flash may be a red herring. Or these may be different bugs.
(In reply to comment #55) > > Flash may be a red herring. Or these may be different bugs. Refering to (In reply to comment #26) > > Firefox crashes aftwer few click. It is easy to reproduce just select type of > car (Gyártmány) for example Alfa Romeo. the hit Search (Keresés) button. After > that click on few linkg, go back, click on few links, do search again with > Keresés. I am sure it will crash whitin a minute ot two. > > I tried on Windows XP > > - normal mode > - normal mode (with adblock) > - safe mode > - safe mode without flash > > and all cases produced crash. >
Someone reported that http://www.menetrendek.hu/cgi-bin/menetrend/html.cgi is still causes crashes. Search for cities (Honnan, Hová) - just enter for example: Budapest, Pécs - then back button causes crash. Tested on Windows (XP, Win7 32,64bit) and clean builds under VMware Virtual Machine. He also noted www.hasznaltauto.hu was wrong but now it is working OK. (original post: http://mozilla.fsf.hu/2011/04/firefox-4-instabilitas-bizonyos-weboldalakon/#comment-460)
With the last test-case (menetrendek.hu) I could not reproduce the crash on MacOS, not even with multiple search and back.
(In reply to comment #58) > With the last test-case (menetrendek.hu) I could not reproduce the crash on > MacOS, not even with multiple search and back. Have you seen any variant of this crash on Mac? I believe this is a Windows-only problem.
Yes, we've seen this crash on Mac and Linux systems as well.
It has dropped 25 places for the last 3 days and is now #69 top crasher in 4.0.
Summary: OOM termination due to adverticum ads on Hungarian sites, hasznaltauto.hu, vatera.hu, menetrendek.hu [@ mozalloc_abort(char const* const) | > mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, > unsigned short*, int, int int) ] → OOM termination due to adverticum ads on Hungarian sites, hasznaltauto.hu, vatera.hu, menetrendek.hu [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ]
(In reply to comment #61) > It has dropped 25 places for the last 3 days and is now #69 top crasher in 4.0. cause the workaround we made in our code, but the bug is still there
Who is responsible for this area? Should we assing this bug to him?
honza and josh, can you have a look?
Honza, this is probably going to have to be you. I don't have a windows dev environment set up, and I am going to be travelling for the next six weeks.
I'll take a look at this.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
It is now #14 top crasher in 4.0.1 over the last 3 days.
Bug 654707 - a crash reproducible quit easily on http://www.hasznaltauto.hu/, *might* be related. Adding just for reference, but could be a red herring.
This is very likely the #1 topcrasher for Hungarian users and is probably making Firefox worse for that population segment than for most others using Firefox.
This (or likely dup bug 637166) would be a great opportunity to show off one of our fancy new heap analyzers.
I read Hungarian if anyone needs help with that.
For now I'm giving up on this. If anyone has time to look at this, feel free to take. I'll return to this later (a week or so).
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW
So what is next? It is critical problem for us. It is cruel and it does not help to maintain the high popularity of Firefox in Hungary.
blocking2.0: --- → ?
No longer depends on: 633473
honza, is there anyone you can suggest that can look at this until you get a chance to return to it?
The repro steps in comment #57 don't seem to be an actual out-of-memory condition. When the process expires from those steps it's only using ~100MB of actual memory and ~500MB of address space. The crash that takes out the process when I hit the back button is an access violation in PL_DHashTableFinish while processing delayed DOM events. I think #57 is a different bug. > xul.dll!PL_DHashTableFinish(PLDHashTable * table=0x14d80d48) Line 386 + 0x24 bytes C xul.dll!nsTHashtable<nsBaseHashtableET<nsStringHashKey,int> >::~nsTHashtable<nsBaseHashtableET<nsStringHashKey,int> >() Line 318 + 0x9 bytes C++ xul.dll!nsBaseHashtable<nsStringHashKey,int,int>::~nsBaseHashtable<nsStringHashKey,int,int>() + 0xf bytes C++ xul.dll!nsDataHashtable<nsStringHashKey,int>::~nsDataHashtable<nsStringHashKey,int>() + 0xf bytes C++ xul.dll!nsDataHashtable<nsStringHashKey,int>::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!nsAutoPtr<nsDataHashtable<nsStringHashKey,int> >::assign(nsDataHashtable<nsStringHashKey,int> * newPtr=0x00000000) Line 71 + 0x1c bytes C++ xul.dll!nsAutoPtr<nsDataHashtable<nsStringHashKey,int> >::operator=(nsDataHashtable<nsStringHashKey,int> * rhs=0x00000000) Line 135 C++ xul.dll!nsGlobalWindow::FireDelayedDOMEvents() Line 8455 C++ xul.dll!nsGlobalWindow::FireDelayedDOMEvents() Line 8441 + 0x55 bytes C++ xul.dll!nsDocShell::RestoreFromHistory() Line 7329 + 0x24 bytes C++ xul.dll!nsDocShell::RestorePresentationEvent::Run() Line 6714 + 0x21 bytes C++ xul.dll!nsThread::ProcessNextEvent(int mayWait=0, int * result=0x0045d514) Line 618 + 0x19 bytes C++ xul.dll!NS_ProcessNextEvent_P(nsIThread * thread=0x07766a90, int mayWait=0) Line 250 + 0x16 bytes C++ xul.dll!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate * aDelegate=0x00e6f160) Line 110 + 0xe bytes C++ xul.dll!MessageLoop::RunInternal() Line 219 C++ xul.dll!MessageLoop::RunHandler() Line 203 C++ xul.dll!MessageLoop::Run() Line 177 C++ xul.dll!nsBaseAppShell::Run() Line 191 C++ xul.dll!nsAppShell::Run() Line 248 + 0x9 bytes C++ xul.dll!nsAppStartup::Run() Line 224 + 0x1c bytes C++ xul.dll!XRE_main(int argc=4, char * * argv=0x07754a98, const nsXREAppData * aAppData=0x07755300) Line 3682 + 0x25 bytes C++ firefox.exe!NS_internal_main(int argc=4, char * * argv=0x07754a98) Line 158 + 0x12 bytes C++ firefox.exe!wmain(int argc=4, wchar_t * * argv=0x077508e0) Line 106 + 0xd bytes C++ firefox.exe!__tmainCRTStartup() Line 552 + 0x19 bytes C firefox.exe!wmainCRTStartup() Line 371 C kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes
Yeah, please get a separate bug filed on comment 57.
I bet this is bug 654707. That is a double-free bug in local/globalStorage that could easily be leading to confusion in the memory allocator. At least we should land the fix in that bug and see if that fixes this.
(Also, steps to reproduce here involve using the back button and bug 654707 is code that runs when we restore a document from bfcache.)
Depends on: 654707
Philor just landed bug 654707 on mozilla-central so people should retest with the next nightly build.
I will test when builds of 2011-05-16 will be available.
It seems English build is available. I am going to test it.
Kami, can you tell some exact sites and STRs to test it? Somehow i hit this bug just for a couple times.
(In reply to comment #82) > Kami, can you tell some exact sites and STRs to test it? Somehow i hit this > bug just for a couple times. This is how I crash Linux: 1. Visit http://www.hasznaltauto.hu/ 2. Left click on one of the car ads under "SMS KIEMELT HIRDETÉSEK" 3. Right click and select Back or click the Back button in Firefox Navigation bar That gives[@ libc-2.13.so@0x323d5 ] crashes with Linux every time, but I suppose it's the same problems as described in this bug. Reproduced: Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110515 Firefox/6.0a1 Works for me: Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 Crash IDs: bp-9bd97f82-95d6-4d6c-98e8-d0b0a2110516 bp-8f7131da-6d76-4f21-a2d3-815772110515 bp-4d8119e1-cb44-4ab8-b0b6-e17032110515 bp-8efc93be-280e-4a84-ac20-548762110515
Using this build: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 The STR is WFM, i cannot reproduce the crash.
Hi! I got a good news: I tested the latest Minefield build (2011-05-16) fgrom here: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ And it seems the promblem has solved. I will ask more friends and the Hungarian community to test this version to be sure this was the problem. Is the localized 6a1 builds still contain this fix? If we tested can we backport to 4.x and 5.x series too?
(In reply to comment #82) > Kami, can you tell some exact sites and STRs to test it? Somehow i hit this > bug just for a couple times. "One of our users created a video: http://www.youtube.com/watch?v=3Xgk_9M0gKE It demonstrates Firefox crash when you select a link then go back quickly causes crash. I don't know if it is related the original issue." another description: I did few test, and the finding are worse than I expected (www.hasznaltauto.hu. "Firefox crashes aftwer few click. It is easy to reproduce just select type of car (Gyártmány) for example Alfa Romeo. the hit Search (Keresés) button. After that click on few linkg, go back, click on few links, do search again with Keresés. I am sure it will crash whitin a minute ot two."
(In reply to comment #83) > (In reply to comment #82) > This is how I crash Linux: > > 1. Visit http://www.hasznaltauto.hu/ > 2. Left click on one of the car ads under "SMS KIEMELT HIRDETÉSEK" > 3. Right click and select Back or click the Back button in Firefox > Navigation bar > > That gives[@ libc-2.13.so@0x323d5 ] crashes with Linux every time, but I > suppose it's the same problems as described in this bug. > > Reproduced: > Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110515 Firefox/6.0a1 > > Works for me: > Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 > It is interesting because I haven't able to reporoduce this on Linux any time. However Windows version always crashed before 2011-05-16.
(In reply to comment #85) > Is the localized 6a1 builds still contain this fix? Yes, the latest l10n builds contain this patch. I tested with the hu l10n build.
(In reply to comment #84) > Using this build: > Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 > > The STR is WFM, i cannot reproduce the crash. And that's maybe because you use the very latest build. As stated in comment 79 the fix was checked in recently, so you have to use a version older than 2011-05-16 if you want to crash. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/
I know that, i tested whether the patch fixed the problem or not. ;)
According to comment 83, this seems to be fixed by bug 654707. Can anyone please confirm this for sure, ones again, so that we can close this bug?
I call for testers in this Hungarian article: http://mozilla.fsf.hu/2011/05/firefox-4-instabilitas-bizonyos-weboldalakon-lehetseges-megoldas/ As far as any new details are available I will post it here.
I tested the STR in comment 83 on native Windows XP SP3 and found that sometimes I have to repeat step 2 and 3 a few times to make it crash, but it's still very easy to crash though. With a x86_64 Firefox on Linux it seems to always crash when I click Back the very first time, but i686 Firefox on x86_64 Linux behaves like Windows XP. At least that's what I've experience with a limited number of trials. Reproduced. Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 bp-499429fa-5341-407a-b032-1ab6b2110516 bp-b6c98305-8f6c-4c80-ba1e-ce19e2110516 Reproduced. Mozilla/5.0 (Windows NT 5.1; rv:5.0a2) Gecko/20110516 Firefox/5.0a2 bp-4c04e3b7-c788-49e5-a779-80bea2110516 Reproduced. Mozilla/5.0 (Windows NT 5.1; rv:6.0a1) Gecko/20110514 Firefox/6.0a1 bp-39ad7884-d3a6-41b1-b40c-49cec2110516 bp-a6cc96b3-9d8b-4cfa-af95-3d2752110516 Works for me / VERIFIED. Mozilla/5.0 (Windows NT 5.1; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 I also tested running Firefox in the WINE emulator, but so far I have not managed to get it to crash with a non-fixed Firefox. Maybe it has another kind of heap or something like that. Mozilla/5.0 (Windows NT 6.0; rv:6.0a1) Gecko/20110515 Firefox/6.0a1
So are you confirming that it is fixed in version 6 dated 20110516 or later or saying that you can still make it crash in such a build?
(In reply to comment #93) > Works for me / VERIFIED. > Mozilla/5.0 (Windows NT 5.1; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 I think Thomas verifies the bag has been fixed. Close this as FIXED/VERIFIED or DUPLICATE of bug 654707?
I can not make the following two browsers dated 20110516 crash, but I can easily make older ones crash using the STR in comment 83 and if necessary iterating step 2 and 3 a few times. Mozilla/5.0 (Windows NT 5.1; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110516 Firefox/6.0a1 When I following the STR in comment 83 I do not get exactly the same Crash IDs as in the bug summary, so maybe someone should look at the crashes in comment 83 and 93 and see if they are equivalent before declaring the bug verified. The Linux Crash IDs are completely different to the Windows ones.
(In reply to comment #96) > When I following the STR in comment 83 I do not get exactly the same Crash > IDs as in the bug summary, so maybe someone should look at the crashes in > comment 83 and 93 and see if they are equivalent before declaring the bug > verified. The Linux Crash IDs are completely different to the Windows ones. That's expected. See comment 47, second to last paragraph.
It seems the fix is okay so far.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → DUPLICATE
blocking2.0: ? → ---
Crash Signature: [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() | nsHtml5Tokenizer::stateLoop(int, unsigned short, int, unsigned short*, int, int, int) ]
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: