Closed Bug 1395598 Opened 7 years ago Closed 7 years ago

Intermittent AddressSanitizer: stack-buffer-overflow on address 0x7fd63cf0da90 at pc 0x7fd650221486 bp 0x7fd63e932b40 sp 0x7fd63e932b38

Categories

(Core :: DOM: Core & HTML, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla57
Tracking Status
firefox-esr52 56+ fixed
firefox55 --- wontfix
firefox56 + fixed
firefox57 + fixed

People

(Reporter: aryx, Assigned: asuth)

References

(Blocks 1 open bug)

Details

(Keywords: csectype-bounds, intermittent-failure, sec-critical, Whiteboard: [adv-main56+][adv-esr52.4+][post-critsmash-triage])

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=127422045&repo=mozilla-inbound

[task 2017-08-31T14:42:00.134065Z] 14:42:00     INFO - TEST-START | dom/tests/browser/browser_largeAllocation_non_win32.js
[task 2017-08-31T14:42:11.741369Z] 14:42:11     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:11.754085Z] 14:42:11     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:11.754400Z] 14:42:11     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:11.754654Z] 14:42:11     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:11.764104Z] 14:42:11     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:11.766766Z] 14:42:11     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:11.767168Z] 14:42:11     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:11.767461Z] 14:42:11     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:11.769529Z] 14:42:11     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:21.056422Z] 14:42:21     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:21.059240Z] 14:42:21     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:21.061322Z] 14:42:21     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:21.064618Z] 14:42:21     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:21.066942Z] 14:42:21     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:21.069884Z] 14:42:21     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:21.071859Z] 14:42:21     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:21.073896Z] 14:42:21     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:21.075863Z] 14:42:21     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:27.536100Z] 14:42:27     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:27.538527Z] 14:42:27     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:27.541944Z] 14:42:27     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:27.545146Z] 14:42:27     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:27.548115Z] 14:42:27     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:27.552357Z] 14:42:27     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:27.554776Z] 14:42:27     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:27.558506Z] 14:42:27     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:27.561086Z] 14:42:27     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:34.052101Z] 14:42:34     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:34.054339Z] 14:42:34     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:34.059117Z] 14:42:34     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:34.062953Z] 14:42:34     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:34.068418Z] 14:42:34     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:34.070653Z] 14:42:34     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:34.072926Z] 14:42:34     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:34.075008Z] 14:42:34     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:34.077074Z] 14:42:34     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:40.188029Z] 14:42:40     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:40.191047Z] 14:42:40     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:40.194164Z] 14:42:40     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:40.197389Z] 14:42:40     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:40.200104Z] 14:42:40     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:40.204587Z] 14:42:40     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:40.206694Z] 14:42:40     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:40.208742Z] 14:42:40     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:40.211389Z] 14:42:40     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:46.320100Z] 14:42:46     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:46.320536Z] 14:42:46     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:46.320819Z] 14:42:46     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:46.321077Z] 14:42:46     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:46.324756Z] 14:42:46     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:46.329018Z] 14:42:46     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:46.331217Z] 14:42:46     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:46.333430Z] 14:42:46     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:46.335802Z] 14:42:46     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:52.312106Z] 14:42:52     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:42:52.312534Z] 14:42:52     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:42:52.323616Z] 14:42:52     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:42:52.323941Z] 14:42:52     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:42:52.324270Z] 14:42:52     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:42:52.328386Z] 14:42:52     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:42:52.330418Z] 14:42:52     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:42:52.335606Z] 14:42:52     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:42:52.337678Z] 14:42:52     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:00.393685Z] 14:43:00     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:00.399369Z] 14:43:00     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:00.399690Z] 14:43:00     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:00.408055Z] 14:43:00     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:00.411208Z] 14:43:00     INFO - GECKO(1104) |       4        640 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:00.414445Z] 14:43:00     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:00.416558Z] 14:43:00     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:00.421128Z] 14:43:00     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:00.424227Z] 14:43:00     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:08.575311Z] 14:43:08     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:08.578072Z] 14:43:08     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:08.581297Z] 14:43:08     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:08.583409Z] 14:43:08     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:08.587620Z] 14:43:08     INFO - GECKO(1104) |       4        640 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:08.589682Z] 14:43:08     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:08.591694Z] 14:43:08     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:08.595524Z] 14:43:08     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:08.597620Z] 14:43:08     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:18.021254Z] 14:43:18     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:18.021666Z] 14:43:18     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:18.024270Z] 14:43:18     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:18.026563Z] 14:43:18     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:18.030040Z] 14:43:18     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:18.032315Z] 14:43:18     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:18.035207Z] 14:43:18     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:18.039436Z] 14:43:18     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:18.041453Z] 14:43:18     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:26.868100Z] 14:43:26     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:26.868524Z] 14:43:26     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:26.868805Z] 14:43:26     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:26.869256Z] 14:43:26     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:26.875628Z] 14:43:26     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:26.878793Z] 14:43:26     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:26.880985Z] 14:43:26     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:26.884546Z] 14:43:26     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:26.888544Z] 14:43:26     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:38.934264Z] 14:43:38     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:38.937281Z] 14:43:38     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:38.938662Z] 14:43:38     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:38.941675Z] 14:43:38     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:38.943816Z] 14:43:38     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:38.946143Z] 14:43:38     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:38.948998Z] 14:43:38     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:38.951933Z] 14:43:38     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:38.954694Z] 14:43:38     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:44.931050Z] 14:43:44     INFO - GECKO(1104) | MEMORY STAT | vsize 20974102MB | residentFast 671MB
[task 2017-08-31T14:43:44.933190Z] 14:43:44     INFO - TEST-OK | dom/tests/browser/browser_largeAllocation_non_win32.js | took 104803ms
[task 2017-08-31T14:43:45.169331Z] 14:43:45     INFO - checking window state
[task 2017-08-31T14:43:45.408388Z] 14:43:45     INFO - TEST-START | dom/tests/browser/browser_localStorage_e10s.js
[task 2017-08-31T14:43:46.825732Z] 14:43:46     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:43:46.826161Z] 14:43:46     INFO - GECKO(1104) | Suppressions used:
[task 2017-08-31T14:43:46.826444Z] 14:43:46     INFO - GECKO(1104) |   count      bytes template
[task 2017-08-31T14:43:46.830276Z] 14:43:46     INFO - GECKO(1104) |     692      22080 nsComponentManagerImpl
[task 2017-08-31T14:43:46.832370Z] 14:43:46     INFO - GECKO(1104) |       3        480 mozJSComponentLoader::LoadModule
[task 2017-08-31T14:43:46.834587Z] 14:43:46     INFO - GECKO(1104) |     304       8933 libfontconfig.so
[task 2017-08-31T14:43:46.837551Z] 14:43:46     INFO - GECKO(1104) |       1         72 nss_ClearErrorStack
[task 2017-08-31T14:43:46.840371Z] 14:43:46     INFO - GECKO(1104) |      16       2316 libglib-2.0.so
[task 2017-08-31T14:43:46.848279Z] 14:43:46     INFO - GECKO(1104) | -----------------------------------------------------
[task 2017-08-31T14:44:02.127122Z] 14:44:02     INFO - GECKO(1104) | =================================================================
[task 2017-08-31T14:44:02.131285Z] 14:44:02    ERROR - GECKO(1104) | ==1104==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fd63cf0da90 at pc 0x7fd650221486 bp 0x7fd63e932b40 sp 0x7fd63e932b38
[task 2017-08-31T14:44:02.137800Z] 14:44:02     INFO - GECKO(1104) | WRITE of size 4 at 0x7fd63cf0da90 thread T44 (localStorage DB)
[task 2017-08-31T14:44:04.634037Z] 14:44:04     INFO - GECKO(1104) |     #0 0x7fd650221485 in mozilla::dom::(anonymous namespace)::SyncLoadCacheHelper::LoadDone(nsresult) /builds/worker/workspace/build/src/dom/storage/StorageIPC.cpp:687:10
[task 2017-08-31T14:44:04.636550Z] 14:44:04     INFO - GECKO(1104) |     #1 0x7fd6501fe3ac in Finalize /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:1319:13
[task 2017-08-31T14:44:04.637994Z] 14:44:04     INFO - GECKO(1104) |     #2 0x7fd6501fe3ac in PerformAndFinalize /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:1034
[task 2017-08-31T14:44:04.641567Z] 14:44:04     INFO - GECKO(1104) |     #3 0x7fd6501fe3ac in mozilla::dom::StorageDBThread::ThreadFunc() /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:553
[task 2017-08-31T14:44:04.643147Z] 14:44:04     INFO - GECKO(1104) |     #4 0x7fd6501f8c4e in mozilla::dom::StorageDBThread::ThreadFunc(void*) /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:496:11
[task 2017-08-31T14:44:04.658203Z] 14:44:04     INFO - GECKO(1104) |     #5 0x7fd6659d74d3 in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:216:5
[task 2017-08-31T14:44:04.676137Z] 14:44:04     INFO - GECKO(1104) |     #6 0x7fd669d306b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
[task 2017-08-31T14:44:04.793293Z] 14:44:04     INFO - GECKO(1104) |     #7 0x7fd668db93dc in clone /build/glibc-bfm8X4/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
[task 2017-08-31T14:44:04.806647Z] 14:44:04     INFO - GECKO(1104) | Address 0x7fd63cf0da90 is located in stack of thread T21 (IPDL Background)
[task 2017-08-31T14:44:04.808368Z] 14:44:04     INFO - GECKO(1104) | SUMMARY: AddressSanitizer: stack-buffer-overflow /builds/worker/workspace/build/src/dom/storage/StorageIPC.cpp:687:10 in mozilla::dom::(anonymous namespace)::SyncLoadCacheHelper::LoadDone(nsresult)
[task 2017-08-31T14:44:04.808691Z] 14:44:04     INFO - GECKO(1104) | Shadow bytes around the buggy address:
[task 2017-08-31T14:44:04.809967Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.810384Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.812739Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.819754Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.821849Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.827309Z] 14:44:04     INFO - GECKO(1104) | =>0x0ffb479d9b50: 00 00[00]00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.830640Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.832804Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.835158Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.843402Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.845408Z] 14:44:04     INFO - GECKO(1104) |   0x0ffb479d9ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2017-08-31T14:44:04.847423Z] 14:44:04     INFO - GECKO(1104) | Shadow byte legend (one shadow byte represents 8 application bytes):
[task 2017-08-31T14:44:04.852043Z] 14:44:04     INFO - GECKO(1104) |   Addressable:           00
[task 2017-08-31T14:44:04.855113Z] 14:44:04     INFO - GECKO(1104) |   Partially addressable: 01 02 03 04 05 06 07
[task 2017-08-31T14:44:04.857411Z] 14:44:04     INFO - GECKO(1104) |   Heap left redzone:       fa
[task 2017-08-31T14:44:04.861497Z] 14:44:04     INFO - GECKO(1104) |   Heap right redzone:      fb
[task 2017-08-31T14:44:04.863615Z] 14:44:04     INFO - GECKO(1104) |   Freed heap region:       fd
[task 2017-08-31T14:44:04.865835Z] 14:44:04     INFO - GECKO(1104) |   Stack left redzone:      f1
[task 2017-08-31T14:44:04.870341Z] 14:44:04     INFO - GECKO(1104) |   Stack mid redzone:       f2
[task 2017-08-31T14:44:04.873188Z] 14:44:04     INFO - GECKO(1104) |   Stack right redzone:     f3
[task 2017-08-31T14:44:04.875318Z] 14:44:04     INFO - GECKO(1104) |   Stack partial redzone:   f4
[task 2017-08-31T14:44:04.878314Z] 14:44:04     INFO - GECKO(1104) |   Stack after return:      f5
[task 2017-08-31T14:44:04.883240Z] 14:44:04     INFO - GECKO(1104) |   Stack use after scope:   f8
[task 2017-08-31T14:44:04.885335Z] 14:44:04     INFO - GECKO(1104) |   Global redzone:          f9
[task 2017-08-31T14:44:04.887330Z] 14:44:04     INFO - GECKO(1104) |   Global init order:       f6
[task 2017-08-31T14:44:04.889770Z] 14:44:04     INFO - GECKO(1104) |   Poisoned by user:        f7
[task 2017-08-31T14:44:04.892320Z] 14:44:04     INFO - GECKO(1104) |   Container overflow:      fc
[task 2017-08-31T14:44:04.894383Z] 14:44:04     INFO - GECKO(1104) |   Array cookie:            ac
[task 2017-08-31T14:44:04.897130Z] 14:44:04     INFO - GECKO(1104) |   Intra object redzone:    bb
[task 2017-08-31T14:44:04.903692Z] 14:44:04     INFO - GECKO(1104) |   ASan internal:           fe
[task 2017-08-31T14:44:04.905783Z] 14:44:04     INFO - GECKO(1104) |   Left alloca redzone:     ca
[task 2017-08-31T14:44:04.907696Z] 14:44:04     INFO - GECKO(1104) |   Right alloca redzone:    cb
[task 2017-08-31T14:44:04.910709Z] 14:44:04     INFO - GECKO(1104) | Thread T44 (localStorage DB) created by T21 (IPDL Background) here:
[task 2017-08-31T14:44:04.912967Z] 14:44:04     INFO - GECKO(1104) |     #0 0x4a3df6 in __interceptor_pthread_create /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:245:3
[task 2017-08-31T14:44:04.915319Z] 14:44:04     INFO - GECKO(1104) |     #1 0x7fd6659d4279 in _PR_CreateThread /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:457:14
[task 2017-08-31T14:44:04.917502Z] 14:44:04     INFO - GECKO(1104) |     #2 0x7fd6659d3e8e in PR_CreateThread /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:548:12
[task 2017-08-31T14:44:04.923085Z] 14:44:04     INFO - GECKO(1104) |     #3 0x7fd6501f7e3f in mozilla::dom::StorageDBThread::Init(nsString const&) /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:291:13
[task 2017-08-31T14:44:04.926606Z] 14:44:04     INFO - GECKO(1104) |     #4 0x7fd65021186c in GetOrCreate /builds/worker/workspace/build/src/dom/storage/StorageDBThread.cpp:213:32
[task 2017-08-31T14:44:04.928780Z] 14:44:04     INFO - GECKO(1104) |     #5 0x7fd65021186c in mozilla::dom::StorageDBParent::RecvStartup() /builds/worker/workspace/build/src/dom/storage/StorageIPC.cpp:832
[task 2017-08-31T14:44:04.931001Z] 14:44:04     INFO - GECKO(1104) |     #6 0x7fd64bd1e0c7 in mozilla::dom::PBackgroundStorageParent::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/PBackgroundStorageParent.cpp:562:20
[task 2017-08-31T14:44:04.942101Z] 14:44:04     INFO - GECKO(1104) |     #7 0x7fd64bcf8de8 in mozilla::ipc::PBackgroundParent::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/PBackgroundParent.cpp:1076:28
[task 2017-08-31T14:44:04.964545Z] 14:44:04     INFO - GECKO(1104) |     #8 0x7fd64b795369 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2110:25
[task 2017-08-31T14:44:04.967059Z] 14:44:04     INFO - GECKO(1104) |     #9 0x7fd64b792144 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:2036:17
[task 2017-08-31T14:44:04.969522Z] 14:44:04     INFO - GECKO(1104) |     #10 0x7fd64b793954 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:1882:5
[task 2017-08-31T14:44:04.971877Z] 14:44:04     INFO - GECKO(1104) |     #11 0x7fd64b793fa8 in mozilla::ipc::MessageChannel::MessageTask::Run() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:1915:15
[task 2017-08-31T14:44:05.013120Z] 14:44:05     INFO - GECKO(1104) |     #12 0x7fd64a9fcbbd in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1039:14
[task 2017-08-31T14:44:05.016737Z] 14:44:05     INFO - GECKO(1104) |     #13 0x7fd64aa021f8 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:521:10
[task 2017-08-31T14:44:05.019747Z] 14:44:05     INFO - GECKO(1104) |     #14 0x7fd64b79e2f4 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/workspace/build/src/ipc/glue/MessagePump.cpp:368:5
[task 2017-08-31T14:44:05.036055Z] 14:44:05     INFO - GECKO(1104) |     #15 0x7fd64b6fd64b in RunInternal /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:326:10
[task 2017-08-31T14:44:05.037832Z] 14:44:05     INFO - GECKO(1104) |     #16 0x7fd64b6fd64b in RunHandler /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:319
[task 2017-08-31T14:44:05.039265Z] 14:44:05     INFO - GECKO(1104) |     #17 0x7fd64b6fd64b in MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:299
[task 2017-08-31T14:44:05.041129Z] 14:44:05     INFO - GECKO(1104) |     #18 0x7fd64a9f756f in nsThread::ThreadFunc(void*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:427:11
[task 2017-08-31T14:44:05.042751Z] 14:44:05     INFO - GECKO(1104) |     #19 0x7fd6659d74d3 in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:216:5
[task 2017-08-31T14:44:05.044187Z] 14:44:05     INFO - GECKO(1104) |     #20 0x7fd669d306b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Group: core-security → dom-core-security
Andrew, mccr8 thinks this may just be a shutdown issue but please take a look.
Flags: needinfo?(bugmail)
## Bug Analysis
This is a legit bug.  We're calling SyncLoadCacheHelper::LoadDone() twice:
- StorageDBThread.cpp:1095, StorageDBThread::DBOperation::Perform(aThread), mType==opPreloadUrgent
- StorageDBThread.cpp:1319 (crash point), StorageDBThread::DBOperation::Finalize(aRv), mType==opPreloadUrgent

Each call to LoadDone looks like:
  virtual void LoadDone(nsresult aRv)
  {
    // Called on the aCache background thread
    MonitorAutoLock monitor(mMonitor);
    mLoaded = true;
    *mRv = aRv;
    monitor.Notify();
  }

Where mRv is:
  nsresult* mRv;
And happens to be part of the PBackground thread's stack.

I haven't checked out the version control history, but this seems like the likely reason for the bug:
- Perform() can return early thanks to NS_ENSURE_SUCCESS() checks throughout without getting to mCache->LoadDone(NS_OK) at the bottom.  (Although the clever loop invariant check screws up; if ExecuteStep() fails the error will never be reported.)  This makes Finalize(aRv) a reasonable person to call the LoadDone(aRv).  Maybe this was added later?

I think it's likely existed for a long while.  Why haven't we seen this before (other than maybe a bug not being filed)?  Because:
- We do issue async preloads.  If the async preload completes before the JS code attempts to read/mutate LocalStorage state, we will not go down this sync preload path.  Getting fast makes it more possible to hit the sync preload case.
- For sync preloads, we normally do synchronous SQLite IO on the current thread, directly running the operation rather than waiting for the DB thread to run it.  This case is exceptional because we had a pending update for the origin scheduled from a previous test step.  Things can pend for up to 5 seconds until the flush occurs and the way they are scheduled means phase is a factor.
- ASAN noticing our reckless behavior requires the other thread to wake-up and get off the stack in a very short number of instructions.

I'm thinking I'll:
- Add a MOZ_ASSERT_IF that checks that LoadDone() is never called when mLoaded is already true.
- Make LoadDone null out mRv to avoid the dangling pointer, plus guard the assignment to require we have a non-null value.
- Remove the redundant success LoadDone(NS_OK) call.
- Fix the while loop's step failure check.

## Threat Analysis
This is how to most reliably make the bug happen:
- Get a page loaded in any process.  Use an origin that does not currently have LocalStorage data so that pages loaded in other processes won't have preload occur, allowing a sync preload to be triggered on demand.
- Get a page in the same origin loaded in a different content process.  I think using our large-allocation header thing is a reliable way to make this happen.  Maybe using the no-opener stuff is another way?  Unsure if that's a thing yet.
- Using a BroadcastChannel to communicate between processes, have page A issue a LocalStorage write and then immediately broadcast to page B that it should do a read or write.
- Page A needs to spam doing "clever bad thing", where "clever bad thing" needs to be:
  - Something serviced by PBackground.
  - Something that will benefit from nsresult NS_OK (0x0) written into the very precise stack location.  Technically a different value could be written, but the only errors that make sense that the attacker might be able to influence are NS_ERROR_FILE_NO_DEVICE_SPACE and NS_ERROR_OUT_OF_MEMORY.  And we expect the former to be impossible in this case where we're doing reads, not writes, and we generally expect SQLite's OOM path to coincide with Firefox crashing, especially given that we're talking about the parent process here.
- Page B immediately does the read or write when it gets the message.  Because there was no async preload, this guarantees a sync preload.  As long as the StorageDBThread didn't happen to perform a flush since A issued its write, this triggers the bad path, but the following dice roll has to work out in the attacker's favor:
  - The PBackground thread wakes up before the dangerous write happens and gets the "clever bad thing" function on the stack.

Using NS_OK at a very specific stack address to do evil things seems hard, but I'm certainly not going to dare anyone about it being impossible.
Flags: needinfo?(bugmail)
Also note: Bug 1350637 moved LocalStorage to PBackground and landed on beta 56 and nightly 57.  So on release 55 the attacker gets to do their clobbering on the main thread of the parent process rather than the PBackground background thread.
Thanks for the thorough analysis!
Assignee: nobody → bugmail
Priority: -- → P1
I ran `xvfb-run ./mach mochitest --repeat 50 dom/tests/browser/browser_localStorage_e10s.js` with and without the removal of the extra LoadDone() call (but with the asserts).  The without-fix version of course asserted immediately.  The with-fix version did not assert.  (The extra runs are arguably excessive, but I added a LoadItem assertion as well.  Its guard made it seem like complicated races are possible.  (They aren't, but why not be safe.)

The NS_ENSURE_SUCCESS change manages to make this look a little bit less suspicious than it is and I tried to be verrrrry misleading in the commit message, but given that it's trivial to figure out the bug number is associated with a security bug, I'm almost certain ne'er-do-wells can figure out what's going on from the patch.
Status: NEW → ASSIGNED
Comment on attachment 8903597 [details] [diff] [review]
Correct LocalStorage mozStorage misuse

Review of attachment 8903597 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/storage/StorageDBThread.cpp
@@ +946,5 @@
>      }
> +    // The loop condition's call to ExecuteStep() may have terminated because
> +    // !NS_SUCCEEDED(), we need an early return to cover that case.  This also
> +    // covers success cases as well, but that's inductively safe.
> +    NS_ENSURE_SUCCESS(rv, rv);

Yeah, this type of loop is dangerous, one can easily overlook the need to check rv after the loop.

::: dom/storage/StorageIPC.cpp
@@ +467,1 @@
>      if (mLoaded) {

This check is now useless ?

Note for future reference: The method doesn't have to return a value anymore ?

@@ +482,2 @@
>      mLoaded = true;
> +    if (mRv) {

Is this |if| needed, the assertion already guards for non-null mRv ?
Attachment #8903597 - Flags: review?(jvarga) → review+
(In reply to Jan Varga [:janv] from comment #7)
> ::: dom/storage/StorageIPC.cpp
> @@ +467,1 @@
> >      if (mLoaded) {
> 
> This check is now useless ?

The check was already useless.  SyncLoadCacheHelper is freshly created in StorageDBParent::RecvPreload and StorageDBThread::SyncPreload either hands it off to the DB thread for LoadItem/LoadDone calls, or keeps it on the current thread.

I think the check was duplicated logic from LocalStorageCache::LoadItem which is a case where it was and still is relevant.  It makes sense there because the LocalStorageCache can see async preload results stream in one-by-one from the database or via IPC.  In those cases, it can go [async 1/N LoadItem, async 2/N LoadItem, SyncPreload gets all N, async 3/N LoadItem perceived, ...] and it's desirable to ignore the excess LoadItem calls.

I can remove the check.  I left it in mainly because:
- This code is still a little complicated, and it seems possible for me to have missed something.  In the event I missed something, that mLoaded check is actually essential, because we have the `InfallibleTArray<nsString>* aKeys` and `InfallibleTArray<nsString>* aValues` arrays that are also potential manipulations of dead stack space in the event of misuse.
- Maybe it will confuse ne'er-do-wells and they will believe all I cared about was the SQLite return value and am being pedantic about stuff.  That's also the rationale behind using mRv as the guard.  Really, mLoaded is the appropriate paranaoia guard and we should null out all 3 of mRv, mKeys, and mValues when LoadDone() is called.  I can spin a cleaner version of the patch that works that way (and with or without the paranoia guard).
 
> Note for future reference: The method doesn't have to return a value anymore
> ?

I'm not sure what you're saying here.

This is the prototype of LocalStorageCacheBridge:
  virtual bool LoadItem(const nsAString& aKey, const nsString& aValue) = 0;
This is the prototype of SyncLoadCacheHelper::LoadItem:
  virtual bool LoadItem(const nsAString& aKey, const nsString& aValue)
and it either returns false or true.

> @@ +482,2 @@
> >      mLoaded = true;
> > +    if (mRv) {
> 
> Is this |if| needed, the assertion already guards for non-null mRv ?

Related to the above, this again is an issue of playing it safe and an act of misdirection.  This can be made more straight-forward to use mLoaded, but I've seen enough crashes in dom/cache due to release builds making it past MOZ_ASSERT lines that it seems reasonable to have such guards.  In this case we do know enough about what's going on that maybe it's not really necessary.  Also, if there's a different idiom or macro you favor for these situations, please let me know.
Flags: needinfo?(jvarga)
Ok, makes sense, keep the patch as it is.

(In reply to Andrew Sutherland [:asuth] from comment #8)
> > Note for future reference: The method doesn't have to return a value anymore
> > ?
> 
> I'm not sure what you're saying here.

The method, after the changes and removed check, only returns true (always returns true) ...

But never mind, as you explained in your last comment, keep it safe.
Flags: needinfo?(jvarga)
(In reply to Andrew Sutherland [:asuth] from comment #8)
> I can remove the check.  I left it in mainly because:
> - This code is still a little complicated, and it seems possible for me to
> have missed something.  In the event I missed something, that mLoaded check
> is actually essential, because we have the `InfallibleTArray<nsString>*
> aKeys` and `InfallibleTArray<nsString>* aValues` arrays that are also
> potential manipulations of dead stack space in the event of misuse.

I actually came to the same conclusion last time I touched this code :)

Anyway, thanks for fixing this bug.
Comment on attachment 8903597 [details] [diff] [review]
Correct LocalStorage mozStorage misuse

[Security approval request comment]
# How easily could an exploit be constructed based on the patch?

I think an exploit for this is hard in general.  See comment 2 for more details, but in general, there's a probabilistic chance to successfully write an NS_OK at a compiler-determined stack location.  The attacker has little ability to impact the stack location because the scenario is the result of the IPC message dispatch process and e10s is a precondition, so they are unable to directly trigger code on the affected threads (main thread on release, PBackground background thread on beta and nightly.)

If they have fancy analysis tools, they might be able to find some existing vulnerable IPC requests they can trigger, but they probably also could have easily found this problem too before us if their tools are that good.

# Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?

Yeah.  I try some misdirection, but there's not a lot going on here that obfuscates the problem.  Understanding how to replicate the problem would be harder if you haven't been keeping up with the non-trivial set of localStorage changes going on.

# Which older supported branches are affected by this flaw?

All branches!

# If not all supported branches, which bug introduced the flaw?

n/a

# Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be?

This patch grafts onto beta and release successfully.  esr52 needs minor help because refactoring resulted in changes to filenames and I can easily provide that.  (Class names changed too, but I don't see any in the hunk contexts.)  Specifically, current StorageDBThread.cpp was previously DOMStorageDBThread.cpp and current StorageIPC.cpp was previously DOMStorageIPC.cpp.

# How likely is this patch to cause regressions; how much testing does it need?

No regressions.  This patch adds assertions that cause existing tests to explode if I hadn't fixed it right.  As indicated in comment 6, I explicitly ran the tests without the fix to ensure this was the case.
Attachment #8903597 - Flags: sec-approval?
Sec-approval+ for trunk.
We'll want Beta and ESR52 patches made and nominated to land after this goes on Trunk.
Attachment #8903597 - Flags: sec-approval? → sec-approval+
https://hg.mozilla.org/integration/mozilla-inbound/rev/298699310b26131dbb5f8c8e341bace5a995278c

This grafts cleanly to Beta/ESR52 as-is, so it just needs approval requests :)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #13)
> This grafts cleanly to Beta/ESR52 as-is, so it just needs approval requests
> :)

Oh right, mercurial tracks renames!  So fancy!  (Maybe git would be able to figure it out too?  I guess I haven't done uplifts with git.)
Comment on attachment 8903597 [details] [diff] [review]
Correct LocalStorage mozStorage misuse

BETA Approval Request Comment
[Feature/Bug causing the regression]:
Longstanding e10s-related bug, feasible to happen in the wild under multi-e10s which 56 is.

[User impact if declined]:
Ignoring attacker: If unlucky, minor stack corruption.  However, minor stack corruption can snowball into crashes and data corruption.
If clever attackers: Friendly clever attackers may win prizes, nefarious clever attackers may pwn the user's Firefox instance.

[Is this code covered by automated tests?]:
Yes, it has an assert in there that's triggered by tests if things go wrong.

[Has the fix been verified in Nightly?]:
Per comment 6, I added asserts and verified I could replicate the ASAN triggered crash circumstances without the fix, then applied the fix.

[Needs manual test from QE? If yes, steps to reproduce]: 
No.

[List of other uplifts needed for the feature/fix]:
None.

[Is the change risky?]:
No.

[Why is the change risky/not risky?]:
This was a very straightforward bug that now has assertions that ensure the problem can't happen, plus we added some defensive paranoia stuff.

[String changes made/needed]:
No string changes.
Attachment #8903597 - Flags: approval-mozilla-beta?
Comment on attachment 8903597 [details] [diff] [review]
Correct LocalStorage mozStorage misuse

ESR Approval Request Comment
[User impact if declined:]
The impact is the same as for beta, but it's worth noting that this is much harder to accidentally or even intentionally trigger on esr52 because:
- There's only a single content process by default, I think.  We did e10s-multi in 54+, right?
- LocalStorageCache instances pre-Fx54 could independently be kept alive by a timer caching thing.  For proper multi-e10s support in 54 we had to drop that, making it easier for attackers to fashion control over stuff.  Between this and the single content process, it makes it hard to trigger the bug.

Having said that, it's a safe fix to land, so it seems like a reasonable ride-along.

[Fix Landed on Version:]
It just landed on inbound (thanks RyanVM!) but I'll forget about the bug if I don't do the approval requests now, so here we are.

[Risk to taking this patch (and alternatives if risky):]
No risk.

[String or UUID changes made by this patch:]
No changes.
Attachment #8903597 - Flags: approval-mozilla-esr52?
Attachment #8903597 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
https://hg.mozilla.org/mozilla-central/rev/298699310b26
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Comment on attachment 8903597 [details] [diff] [review]
Correct LocalStorage mozStorage misuse

sec fix for esr52.4
Attachment #8903597 - Flags: approval-mozilla-esr52? → approval-mozilla-esr52+
Group: dom-core-security → core-security-release
Whiteboard: [adv-main56+][adv-esr52.4+]
Flags: qe-verify-
Whiteboard: [adv-main56+][adv-esr52.4+] → [adv-main56+][adv-esr52.4+][post-critsmash-triage]
Group: core-security-release
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.