1296819 - Intermittent valgrind-test | Invalid write of size 4 at mozilla:: / _pt_root / start_thread / clone

Status: VERIFIED FIXED

(Core :: General, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=34240771&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/SN9U47DhQIavbjuMIxpVJg/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

First occurrence seems to be https://treeherder.mozilla.org/logviewer.html#?job_id=33703148&repo=mozilla-inbound

Comment 2

[Henry Chang [:hchang]]

Attachment: Bug 1296819 - Skip applying TableUpdateV4 to avoid premature update codes being run.

Review commit: https://reviewboard.mozilla.org/r/73236/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/73236/

Comment 3

[BMO Automation]

Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.

Comment 4

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 3
* try: 1
* fx-team: 1
* autoland: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2016-09-19&endday=2016-09-25&tree=all

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Am I reading the stack right that this looks like an issue with the hang monitor?

Comment 6

[Julian Seward [:jseward]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)
> Am I reading the stack right that this looks like an issue with the hang
> monitor?

Well, I don't know if it is an issue with the hang monitor per se.
The invalid write to 0x0 is this, at the end of "void RunWatchdog(void* arg)"
(the watchdog thread main function):

    // Shutdown is apparently dead. Crash the process.
    MOZ_CRASH("Shutdown too long, probably frozen, causing a crash.");

So the question that comes to mind is: why is this timing out now when it
didn't previously?

Comment 7

[Intermittent Failures Robot]

5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 3
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2016-10-03&endday=2016-10-09&tree=all

Comment 8

[Intermittent Failures Robot]

9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 4
* mozilla-inbound: 3
* mozilla-aurora: 2

Platform breakdown:
* linux64: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2016-10-17&endday=2016-10-23&tree=all

Comment 9

[Intermittent Failures Robot]

16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 7
* mozilla-inbound: 6
* try: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2016-10-24&endday=2016-10-30&tree=all

Comment 10

[Intermittent Failures Robot]

11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* autoland: 5

Platform breakdown:
* linux64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2016-10-31&endday=2016-11-06&tree=all

Comment 11

[Intermittent Failures Robot]

6 failures in 749 pushes (0.008 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 5
* mozilla-aurora: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2017-01-23&endday=2017-01-29&tree=all

Comment 12

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.

Comment 13

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Hello, I'm asking your help with an experiment with making decisions on bugs. You've been needinfo'ed on this bug. I'd like you to take one action to help this bug make progress toward a decision. The things you can do include:

* If you know or have a good guess of which product and component this bug belongs to, change the product and component of the bug
* If you know of the right person to ask about this bug, redirect the needinfo to them
* If you cannot reproduce the bug, close it

All we need you to do is one thing that will help us make a decision on the bug or resolve it.

Thank you for your help with this. If you have questions, please contact [e-mail].

Comment 14

[Henry Chang [:hchang]]

Not sure why I am ni'ed but I am trying to give feedback as far as I know:

1) There seems to be a wrong review commit (from me) pushed to this bug but it shouldn't
   be relevant to this bug.

2) I didn't find the stack but that reminds me a safebrowsing "long hang crash" which
   has been resolved for a while.

Comment 15

[Valentin Gosu [:valentin] (he/him)]

Thanks for that. It also seemed to me that this hasn't been reproduced in a while, but I thought I would ask someone who looked at it previously. Seems I should have looked more carefully.

Closing as WORKSFORME

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=124407009&repo=mozilla-inbound

Comment 17

[Intermittent Failures Robot]

1 failures in 949 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 824 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2017-10-02&endday=2017-10-08&tree=all

Comment 20

[Intermittent Failures Robot]

25 failures in 685 pushes (0.036 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 13
* autoland: 10
* mozilla-central: 2

Platform breakdown:
* linux64: 25

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-12&endday=2018-02-18&tree=all

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This has become a frequent failure. First failure after a long break was on 2018-02-14.

Julian, can you take a look at this or redirect to an appropriate contact, please?

Comment 22

[Intermittent Failures Robot]

15 failures in 151 pushes (0.099 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 13
* mozilla-inbound: 2

Platform breakdown:
* linux64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-22&endday=2018-02-22&tree=all

Comment 23

[Intermittent Failures Robot]

18 failures in 135 pushes (0.133 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6
* mozilla-central: 2
* try: 1

Platform breakdown:
* linux64: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-23&endday=2018-02-23&tree=all

Comment 24

[Intermittent Failures Robot]

74 failures in 675 pushes (0.11 failures/push) were associated with this bug in the last 7 days. 

This is the #8 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 46
* mozilla-inbound: 20
* mozilla-central: 6
* try: 2

Platform breakdown:
* linux64: 73
* linux64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-19&endday=2018-02-25&tree=all

Comment 25

[Julian Seward [:jseward]]

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #21)
> Julian, can you take a look at this or redirect to an appropriate contact,
> please?

Looking.  At first glance this looks like a shutdown hang which 
results in a MOZ_CRASH, causing Valgrind to report an invalid
write at address zero.  I wonder if we need to increase some
timeout value for the running-on-valgrind case.

Comment 26

[Intermittent Failures Robot]

25 failures in 169 pushes (0.148 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 14
* mozilla-inbound: 9
* mozilla-central: 2

Platform breakdown:
* linux64: 24
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-28&endday=2018-02-28&tree=all

Comment 27

[Intermittent Failures Robot]

15 failures in 179 pushes (0.084 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6

Platform breakdown:
* linux64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-03-01&endday=2018-03-01&tree=all

Comment 28

[Intermittent Failures Robot]

19 failures in 150 pushes (0.127 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 10
* autoland: 7
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 18
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-03-02&endday=2018-03-02&tree=all

Comment 29

[Intermittent Failures Robot]

97 failures in 831 pushes (0.117 failures/push) were associated with this bug in the last 7 days. 

This is the #10 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 55
* mozilla-inbound: 35
* mozilla-central: 5
* try: 2

Platform breakdown:
* linux64: 93
* windows2012-32: 1
* linux32-stylo-disabled: 1
* linux32: 1
* android-4-3-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-02-26&endday=2018-03-04&tree=all

Comment 30

[Tiberius Oros[:tiberius_oros]]

There have been 63 failures in the last 7 days.
Almost all the failures occur on Linux x64 /opt.

Here is a relevant log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=166912132

[task 2018-03-09T03:14:39.963Z] 03:14:39     INFO -  14:09.27 --31681-- WARNING: Serious error when reading debug info
[task 2018-03-09T03:14:39.963Z] 03:14:39     INFO -  14:09.27 --31681-- When reading debug info from /dev/shm/org.chromium.GwrzjR (deleted):
[task 2018-03-09T03:14:39.963Z] 03:14:39     INFO -  14:09.27 --31681-- failed to stat64/stat this file
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681== Thread 63 Shutdow~minator:
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 TEST-UNEXPECTED-FAIL | valgrind-test | Invalid write of size 4 at mozilla:: / _pt_root / start_thread / clone
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681== Invalid write of size 4
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681==    at 0x114EC2CD: mozilla::(anonymous namespace)::RunWatchdog(void*) (nsTerminator.cpp:206)
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681==    by 0x642CB80: _pt_root (ptthread.c:201)
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681==    by 0x4E3AB4F: start_thread (pthread_create.c:304)
[task 2018-03-09T03:15:56.146Z] 03:15:56     INFO -  15:25.45 ==31681==    by 0x5D51FBC: clone (clone.S:112)
[task 2018-03-09T03:15:56.147Z] 03:15:56     INFO -  15:25.45 ==31681==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

:jseward any updates on this?

Comment 31

[Julian Seward [:jseward]]

(In reply to Tiberius Oros[:tiberius_oros] from comment #30)
> :jseward any updates on this?

I have been looking at this in detail this week (sorry, I should
have mentioned this earlier).  It's not easy to figure out the root
cause.  I currently have 2 bugs in progress: bug 1442603 and bug 1443499
which I believe should improve the situation.  I hope to have them
landed by early next week at the latest.

Comment 32

[Intermittent Failures Robot]

48 failures in 747 pushes (0.064 failures/push) were associated with this bug in the last 7 days. 

This is the #20 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 28
* mozilla-inbound: 13
* mozilla-central: 6
* try: 1

Platform breakdown:
* linux64: 48

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-03-05&endday=2018-03-11&tree=all

Comment 33

[Julian Seward [:jseward]]

(In reply to Julian Seward [:jseward] from comment #31)
> (In reply to Tiberius Oros[:tiberius_oros] from comment #30)
> > :jseward any updates on this?
> [..]
> I currently have 2 bugs in progress: bug 1442603 and bug 1443499

These have both now landed.  They reduce the worst-case BHR resource
usage, which I suspect may be a factor here.  Let's see if that
makes a difference.  If not, I will look into increasing the shutdown
hang timeouts when running on valgrind.

Comment 34

[Intermittent Failures Robot]

11 failures in 814 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 2

Platform breakdown:
* linux64: 10
* windows2012-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296819&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 35

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 36

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This bug is back on mozilla-release and mozilla-esr60, failing on recent pushes 40% of all runs (?).

Log https://treeherder.mozilla.org/logviewer.html#?job_id=176808362&repo=mozilla-release

[task 2018-05-03T18:57:12.162Z] 18:57:12     INFO -  14:49.54 TEST-UNEXPECTED-FAIL | valgrind-test | Invalid write of size 4 at mozilla:: / _pt_root / start_thread / clone
[task 2018-05-03T18:57:12.162Z] 18:57:12     INFO -  14:49.54 ==30231== Invalid write of size 4
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==    at 0x11111EED: mozilla::(anonymous namespace)::RunWatchdog(void*) (nsTerminator.cpp:206)
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==    by 0x642B60C: _pt_root (ptthread.c:201)
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==    by 0x4E3AB4F: start_thread (pthread_create.c:304)
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==    by 0x5D51FBC: clone (clone.S:112)
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 ==30231==
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54 {
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54    <insert_a_suppression_name_here>
[task 2018-05-03T18:57:12.163Z] 18:57:12     INFO -  14:49.54    Memcheck:Addr4
[task 2018-05-03T18:57:12.164Z] 18:57:12     INFO -  14:49.54    fun:_ZN7mozilla12_GLOBAL__N_111RunWatchdogEPv
[task 2018-05-03T18:57:12.164Z] 18:57:12     INFO -  14:49.54    fun:_pt_root
[task 2018-05-03T18:57:12.164Z] 18:57:12     INFO -  14:49.54    fun:start_thread
[task 2018-05-03T18:57:12.164Z] 18:57:12     INFO -  14:49.54    fun:clone
[task 2018-05-03T18:57:12.164Z] 18:57:12     INFO -  14:49.54 }

This sounds similar to comment 33. Julian, can you take a look at this, please?

Comment 37

[Intermittent Failures Robot]

10 failures in 551 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 6
* mozilla-esr60: 3
* try: 1

Platform breakdown:
* linux64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 38

[Intermittent Failures Robot]

1 failures in 660 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr60: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 39

[Intermittent Failures Robot]

2 failures in 639 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 1
* mozilla-esr60: 1

Platform breakdown:
* linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 40

[Intermittent Failures Robot]

7 failures in 632 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr60: 5
* try: 2

Platform breakdown:
* linux64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 41

[Julian Seward [:jseward]]

Attachment: bug1296819-increase-shutdown-timeout-1.diff

Increases the shutdown timeout by a factor of 3 when built --enable-valgrind
*and* actually running on Valgrind.  Otherwise no effect.

Comment 42

[Andrea Marchesini [:baku]]

Comment on attachment 8981448 [details] [diff] [review]
bug1296819-increase-shutdown-timeout-1.diff

Review of attachment 8981448 [details] [diff] [review]:
-----------------------------------------------------------------

looks good to me. Thanks.

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/17ad383212f4
Intermittent valgrind-test | Invalid write of size 4 at mozilla:: / _pt_root / start_thread / clone.  r=[e-mail].

Comment 44

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/17ad383212f4

Comment 45

[Intermittent Failures Robot]

5 failures in 655 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr60: 5

Platform breakdown:
* linux64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 46

[Julian Seward [:jseward]]

Comment on attachment 8981448 [details] [diff] [review]
bug1296819-increase-shutdown-timeout-1.diff

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:

 For automation runs on Valgrind, Valgrind slows down progress so much
 that the Shutdown Hang Reporter sometimes kills the process unnecessarily,
 leading the test to fail.  This patch increases the SHR timeouts
 only for builds which are configured --enable-valgind and which are actually
 running on Valgrind.

User impact if declined: 

 Zero, because it requires (1) config with --enable-valgind and (2) actually
 running on Valgrind.

Fix Landed on Version:

 62

Risk to taking this patch (and alternatives if risky): 

  Very low risk (no end-user effect)

String or UUID changes made by this patch: 

 None.

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.

Comment 47

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8981448 [details] [diff] [review]
bug1296819-increase-shutdown-timeout-1.diff

Per bug 1443499 comment 9, taking for ESR60.

Comment 48

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/00155725c3ea

Comment 49

[Intermittent Failures Robot]

5 failures in 669 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr60: 4
* mozilla-release: 1

Platform breakdown:
* linux64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1296819&startday=2018-06-04&endday=2018-06-10&tree=trunk