1293828 - Intermittent Linux ASAN/static builds failing with 'mach build' did not run successfully. Please check log for errors.

Status: RESOLVED DUPLICATE of bug 1293717

(NSS :: Libraries, defect)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=33580916&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/FLfl3WOYTXOw57OJgVI6ug/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Also affects static builds, apparently: https://treeherder.mozilla.org/logviewer.html#?job_id=33589765&repo=mozilla-inbound

Comment 2

[Mike Hommey [:glandium]]

18:23:37     INFO -  ==30822==ERROR: AddressSanitizer: use-after-poison on address 0x6150003dbe68 at pc 0x0000004a08a5 bp 0x7ffc5488ac80 sp 0x7ffc5488a440
18:23:37     INFO -  WRITE of size 29 at 0x6150003dbe68 thread T0
18:23:37     INFO -      #0 0x4a08a4  (/home/worker/workspace/build/src/obj-firefox/dist/bin/xpcshell+0x4a08a4)
18:23:37     INFO -      #1 0x7fe5fbe461f1  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnssutil3.so+0x151f1)
18:23:37     INFO -      #2 0x7fe5fc504f8d  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnss3.so+0x42f8d)
18:23:37     INFO -      #3 0x7fe5fc5063f0  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnss3.so+0x443f0)
18:23:37     INFO -      #4 0x7fe5fc4dda21  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnss3.so+0x1ba21)
18:23:37     INFO -      #5 0x7fe5fc4de270  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnss3.so+0x1c270)
18:23:37     INFO -      #6 0x7fe6098da4c5  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xa2b24c5)
18:23:37     INFO -      #7 0x7fe6098dbe21  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xa2b3e21)
18:23:37     INFO -      #8 0x7fe609909503  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xa2e1503)
18:23:37     INFO -      #9 0x7fe6014e20f1  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1eba0f1)
18:23:37     INFO -      #10 0x7fe6014d9502  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1eb1502)
18:23:37     INFO -      #11 0x7fe601579681  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1f51681)
18:23:37     INFO -      #12 0x7fe60156f994  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1f47994)
18:23:37     INFO -      #13 0x7fe6098d39e8  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xa2ab9e8)
18:23:37     INFO -      #14 0x7fe60990a36f  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xa2e236f)
18:23:37     INFO -      #15 0x7fe6014e1c9a  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1eb9c9a)
18:23:37     INFO -      #16 0x7fe6014d8255  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1eb0255)
18:23:37     INFO -      #17 0x7fe602e42b3b  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x381ab3b)
18:23:37     INFO -      #18 0x7fe601536b66  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1f0eb66)
18:23:37     INFO -      #19 0x7fe602ed9e82  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x38b1e82)
18:23:37     INFO -      #20 0x7fe602ee0fec  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x38b8fec)
18:23:37     INFO -      #21 0x7fe60c06b203  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xca43203)
18:23:37     INFO -      #22 0x7fe60c04b6db  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xca236db)
18:23:37     INFO -      #23 0x7fe60c031385  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xca09385)
18:23:37     INFO -      #24 0x7fe60c06de77  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xca45e77)
18:23:37     INFO -      #25 0x7fe60c06e5ae  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xca465ae)
18:23:37     INFO -      #26 0x7fe60bbb01fd  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0xc5881fd)
18:23:37     INFO -      #27 0x7fe602dcd779  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x37a5779)
18:23:37     INFO -      #28 0x7fe602dd48bb  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x37ac8bb)
18:23:37     INFO -      #29 0x7fe602dd19da  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x37a99da)
18:23:37     INFO -      #30 0x7fe602e003f1  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x37d83f1)
18:23:37     INFO -      #31 0x7fe601536b66  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x1f0eb66)
18:23:37     INFO -      #32 0x7fe602ed9e82  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x38b1e82)
18:23:37     INFO -      #33 0x7fe602ee0fec  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libxul.so+0x38b8fec)
18:23:37     INFO -      #34 0x7fe5e4b872a9  (<unknown module>)
18:23:37     INFO -  0x6150003dbe68 is located 232 bytes inside of 512-byte region [0x6150003dbd80,0x6150003dbf80)
18:23:37     INFO -  allocated by thread T0 here:
18:23:37     INFO -      #0 0x4b667b  (/home/worker/workspace/build/src/obj-firefox/dist/bin/xpcshell+0x4b667b)
18:23:37     INFO -      #1 0x7fe60fb17e26  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libplds4.so+0x1e26)
18:23:37     INFO -      #2 0x7fe5fbe45dbf  (/home/worker/workspace/build/src/obj-firefox/dist/bin/libnssutil3.so+0x14dbf)
18:23:37     INFO -  SUMMARY: AddressSanitizer: use-after-poison (/home/worker/workspace/build/src/obj-firefox/dist/bin/xpcshell+0x4a08a4)
18:23:37     INFO -  Shadow bytes around the buggy address:
18:23:37     INFO -    0x0c2a80073770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18:23:37     INFO -    0x0c2a80073780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18:23:37     INFO -    0x0c2a80073790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18:23:37     INFO -    0x0c2a800737a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18:23:37     INFO -    0x0c2a800737b0: 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
18:23:37     INFO -  =>0x0c2a800737c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7[f7]f7 f7
18:23:37     INFO -    0x0c2a800737d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
18:23:37     INFO -    0x0c2a800737e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
18:23:37     INFO -    0x0c2a800737f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18:23:37     INFO -    0x0c2a80073800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18:23:37     INFO -    0x0c2a80073810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18:23:37     INFO -  Shadow byte legend (one shadow byte represents 8 application bytes):
18:23:37     INFO -    Addressable:           00
18:23:37     INFO -    Partially addressable: 01 02 03 04 05 06 07
18:23:37     INFO -    Heap left redzone:       fa
18:23:37     INFO -    Heap right redzone:      fb
18:23:37     INFO -    Freed heap region:       fd
18:23:37     INFO -    Stack left redzone:      f1
18:23:37     INFO -    Stack mid redzone:       f2
18:23:37     INFO -    Stack right redzone:     f3
18:23:37     INFO -    Stack partial redzone:   f4
18:23:37     INFO -    Stack after return:      f5
18:23:37     INFO -    Stack use after scope:   f8
18:23:37     INFO -    Global redzone:          f9
18:23:37     INFO -    Global init order:       f6
18:23:37     INFO -    Poisoned by user:        f7
18:23:37     INFO -    Container overflow:      fc
18:23:37     INFO -    Array cookie:            ac
18:23:37     INFO -    Intra object redzone:    bb
18:23:37     INFO -    ASan internal:           fe
18:23:37     INFO -    Left alloca redzone:     ca
18:23:37     INFO -    Right alloca redzone:    cb

Sadly missing symbols... but at least this points to NSS.

Comment 3

[Mike Hommey [:glandium]]

(In reply to Wes Kocher (:KWierso) from comment #1)
> Also affects static builds, apparently:
> https://treeherder.mozilla.org/logviewer.html#?job_id=33589765&repo=mozilla-
> inbound

That's a completely different error.

Comment 4

[Gregory Szorc [:gps]]

Should be fixed by 82eba2e77c35, which is on the autoland repo.

Comment 5

[Intermittent Failures Robot]

15 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 13
* autoland: 2

Platform breakdown:
* linux64: 12
* android-4-0-armv7-api15: 2
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293828&startday=2016-08-08&endday=2016-08-14&tree=all

Comment 6

[Intermittent Failures Robot]

7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* try: 2
* fx-team: 1

Platform breakdown:
* android-4-0-armv7-api15: 4
* linux64: 2
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293828&startday=2016-08-15&endday=2016-08-21&tree=all

Comment 7

[Intermittent Failures Robot]

5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* mozilla-central: 1

Platform breakdown:
* windows8-64: 2
* windowsxp: 1
* osx-10-7: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293828&startday=2016-09-12&endday=2016-09-18&tree=all

Comment 8

[Intermittent Failures Robot]

9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 5
* mozilla-inbound: 2
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 5
* osx-10-7: 3
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293828&startday=2016-09-19&endday=2016-09-25&tree=all