1414593 - Permaorange Ccov tp5o_webext | application crashed [@ libc-2.23.so + 0x35428]

Status: NEW

(Firefox :: New Tab Page, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=142192726&repo=mozilla-central

https://queue.taskcluster.net/v1/task/Pp_0yRMIQpGMQAOL8FwymA/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

9 failures in 857 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 9

Platform breakdown:
* linux64-ccov: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 2

[Tiberius Oros[:tiberius_oros]]

- filed 4 days ago
- 33 failures in the last 7 days
- occurs only on linux64-ccov platform and opt buid type
- recent log file: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=143018864&lineNumber=17201

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the root cause here is from an activity stream update in bug 1410541:
https://hg.mozilla.org/mozilla-central/rev/d1f0c44b2d79

some related parts of the crash:
[task 2017-11-07T20:09:24.851Z] 20:09:24     INFO -  PROCESS-CRASH | tp5o_webext | application crashed [@ libc-2.23.so + 0x35428]
[task 2017-11-07T20:09:24.852Z] 20:09:24     INFO -  Crash dump filename: /tmp/tmpEOfFTQ/profile/minidumps/3804c9ff-92fb-de08-5dd8-326d2f3cc90d.dmp
[task 2017-11-07T20:09:24.853Z] 20:09:24     INFO -  Operating system: Linux
[task 2017-11-07T20:09:24.853Z] 20:09:24     INFO -                    0.0.0 Linux 3.13.0-112-generic #159-Ubuntu SMP Fri Mar 3 15:26:07 UTC 2017 x86_64
[task 2017-11-07T20:09:24.854Z] 20:09:24     INFO -  CPU: amd64
[task 2017-11-07T20:09:24.854Z] 20:09:24     INFO -       family 6 model 62 stepping 4
[task 2017-11-07T20:09:24.855Z] 20:09:24     INFO -       4 CPUs
[task 2017-11-07T20:09:24.856Z] 20:09:24     INFO -  GPU: UNKNOWN
[task 2017-11-07T20:09:24.857Z] 20:09:24     INFO -  Crash reason:  SIGABRT
[task 2017-11-07T20:09:24.857Z] 20:09:24     INFO -  Crash address: 0x3e800000543
[task 2017-11-07T20:09:24.858Z] 20:09:24     INFO -  Process uptime: not available
[task 2017-11-07T20:09:24.858Z] 20:09:24     INFO -  Thread 0 (crashed)
[task 2017-11-07T20:09:24.859Z] 20:09:24     INFO -   0  libc-2.23.so + 0x35428
[task 2017-11-07T20:09:24.860Z] 20:09:24     INFO -      rax = 0x0000000000000000   rdx = 0x0000000000000006
[task 2017-11-07T20:09:24.860Z] 20:09:24     INFO -      rcx = 0xffffffffffffffff   rbx = 0x00007fc81b76e000
[task 2017-11-07T20:09:24.861Z] 20:09:24     INFO -      rsi = 0x0000000000000543   rdi = 0x0000000000000543
[task 2017-11-07T20:09:24.861Z] 20:09:24     INFO -      rbp = 0x00007fc815c701dc   rsp = 0x00007ffcafb9d2d8
[task 2017-11-07T20:09:24.861Z] 20:09:24     INFO -       r8 = 0x0000000002b0eda0    r9 = 0xfeff0011feff0900
[task 2017-11-07T20:09:24.862Z] 20:09:24     INFO -      r10 = 0x0000000000000008   r11 = 0x0000000000000202
[task 2017-11-07T20:09:24.862Z] 20:09:24     INFO -      r12 = 0x0000000000000216   r13 = 0x00007fc815c70390
[task 2017-11-07T20:09:24.866Z] 20:09:24     INFO -      r14 = 0x0000000000002c8e   r15 = 0x0000000002a94cc0
[task 2017-11-07T20:09:24.867Z] 20:09:24     INFO -      rip = 0x00007fc81a2ed428
[task 2017-11-07T20:09:24.867Z] 20:09:24     INFO -      Found by: given as instruction pointer in context
[task 2017-11-07T20:09:24.868Z] 20:09:24     INFO -   1  libc-2.23.so + 0x3702a
[task 2017-11-07T20:09:24.868Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d2e0   rip = 0x00007fc81a2ef02a
[task 2017-11-07T20:09:24.869Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.869Z] 20:09:24     INFO -   2  libfontconfig.so.1.9.0 + 0x261dc
[task 2017-11-07T20:09:24.870Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d398   rip = 0x00007fc815c701dc
[task 2017-11-07T20:09:24.870Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.871Z] 20:09:24     INFO -   3  libfontconfig.so.1.9.0 + 0x26390
[task 2017-11-07T20:09:24.871Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d3a8   rip = 0x00007fc815c70390
[task 2017-11-07T20:09:24.872Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.872Z] 20:09:24     INFO -   4  libc-2.23.so + 0x8453c
[task 2017-11-07T20:09:24.873Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d3c0   rip = 0x00007fc81a33c53c
[task 2017-11-07T20:09:24.873Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.874Z] 20:09:24     INFO -   5  libc-2.23.so + 0x190210
[task 2017-11-07T20:09:24.874Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d3c8   rip = 0x00007fc81a448210
[task 2017-11-07T20:09:24.875Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.875Z] 20:09:24     INFO -   6  libc-2.23.so + 0x193760
[task 2017-11-07T20:09:24.876Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d3d0   rip = 0x00007fc81a44b760
[task 2017-11-07T20:09:24.876Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.877Z] 20:09:24     INFO -   7  libc-2.23.so + 0x190210
[task 2017-11-07T20:09:24.877Z] 20:09:24     INFO -      rsp = 0x00007ffcafb9d3e0   rip = 0x00007fc81a448210
[task 2017-11-07T20:09:24.878Z] 20:09:24     INFO -      Found by: stack scanning
[task 2017-11-07T20:09:24.878Z] 20:09:24     INFO -   8  libfontconfig.so.1.9.0 + 0x261dc


:Mardak, I see you authored bug 1410541, can you help figure out why when using web extensions in talos we crash during code coverage collection?

Comment 4

[Ed Lee :Mardak]

How did you bisect / test? Looks like `try: -b o -p linux64-ccov -t g5-e10s --artifact` ? I'll try to bisect within the activity stream changes.

https://github.com/mozilla/activity-stream/compare/6245c446e61bb13fed4bd8f7c600de44d9fed1ef...95b4c35393b7192d680d1291b6960200be5e7570

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I did a bunch of try pushes with that syntax, but oddly the jobs are not added, when the builds are done I had to 'add new jobs' for the g5 job.

If you have ways to bisect the activity stream data, I am happy to do another large round of try pushes and baby sit them- likewise point me at the try pushes you have and I will add the jobs :)

Comment 6

[Ed Lee :Mardak]

(In reply to Ed Lee :Mardak from comment #4)
> https://github.com/mozilla/activity-stream/compare/6245c446e61bb13fed4bd8f7c600de44d9fed1ef...95b4c35393b7192d680d1291b6960200be5e7570
Given the long latency of testing, I just ended up pushing all commits in this range:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=0534a2c0d4502267f81357322891c18aee8a694a&filter-searchStr=ccov%20g5&tochange=62b0395210633288d697f6a69eb9e07f954a2bf5

https://github.com/mozilla/activity-stream/pull/3695

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks for identifying the root cause here- hopefully this is an easy fix, if not it can wait a few weeks to resolve for when it makes the most sense.

Comment 8

[Intermittent Failures Robot]

62 failures in 849 pushes (0.073 failures/push) were associated with this bug in the last 7 days. 

This is the #26 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 62

Platform breakdown:
* linux64-ccov: 62

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 9

[Tiberius Oros[:tiberius_oros]]

Last week's summary:
- 46 total failures
- platform: linux64-ccov
- build: opt

Here is a recent log:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=145858061&lineNumber=17409

And snippet from it:

[task 2017-11-18T21:58:40.953Z] 21:58:40     INFO -  TEST-INFO | 1337: exit 6
17405
[task 2017-11-18T21:58:40.969Z] 21:58:40     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/ZvMxtzqHRbykZKZOFvT99A/artifacts/public/build/target.crashreporter-symbols.zip
17406
[task 2017-11-18T21:58:48.681Z] 21:58:48     INFO -  mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpL6RM1R/profile/minidumps/2d5ec068-bd42-e86b-0206-533542a20a6f.dmp /tmp/tmpkyRSAz
17407
[task 2017-11-18T21:58:56.789Z] 21:58:56     INFO -  mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/2d5ec068-bd42-e86b-0206-533542a20a6f.dmp
17408
[task 2017-11-18T21:58:56.804Z] 21:58:56     INFO -  mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/2d5ec068-bd42-e86b-0206-533542a20a6f.extra
17409
[task 2017-11-18T21:58:56.805Z] 21:58:56     INFO -  PROCESS-CRASH | tp5o_webext | application crashed [@ libc-2.23.so + 0x35428]
17410
[task 2017-11-18T21:58:56.805Z] 21:58:56     INFO -  Crash dump filename: /tmp/tmpL6RM1R/profile/minidumps/2d5ec068-bd42-e86b-0206-533542a20a6f.dmp
17411
[task 2017-11-18T21:58:56.806Z] 21:58:56     INFO -  Operating system: Linux
17412
[task 2017-11-18T21:58:56.806Z] 21:58:56     INFO -                    0.0.0 Linux 3.13.0-112-generic #159-Ubuntu SMP Fri Mar 3 15:26:07 UTC 2017 x86_64
17413
[task 2017-11-18T21:58:56.806Z] 21:58:56     INFO -  CPU: amd64
17414
[task 2017-11-18T21:58:56.807Z] 21:58:56     INFO -       family 6 model 62 stepping 4
17415
[task 2017-11-18T21:58:56.807Z] 21:58:56     INFO -       4 CPUs
17416
[task 2017-11-18T21:58:56.808Z] 21:58:56     INFO -  GPU: UNKNOWN
17417
[task 2017-11-18T21:58:56.808Z] 21:58:56     INFO -  Crash reason:  SIGABRT
17418
[task 2017-11-18T21:58:56.808Z] 21:58:56     INFO -  Crash address: 0x3e800000539
17419
[task 2017-11-18T21:58:56.809Z] 21:58:56     INFO -  Process uptime: not available

Comment 10

[Intermittent Failures Robot]

45 failures in 762 pushes (0.059 failures/push) were associated with this bug in the last 7 days. 

This is the #24 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 45

Platform breakdown:
* linux64-ccov: 45

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

I have this patch which should account for the extra coverage changes needed:
https://pastebin.mozilla.org/9073271

then:
./mach talos-test -a tp5o_webext

Comment 12

[Andrei Oprea [:andreio]]

Seems there was a similar crash fixed recently in bug 1411322. I created bug 1419779 for this.

Comment 13

[Intermittent Failures Robot]

14 failures in 744 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 14

Platform breakdown:
* linux64-ccov: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 14

[Ed Lee :Mardak]

andreio tracked this down to the splitting of `createChannel` to have a separate `simulateMessagesForExistingTabs`:
https://github.com/mozilla/activity-stream/pull/3695/files#diff-a12f94b26c6ac483de6e2a17547ced6bR141

Comment 15

[Intermittent Failures Robot]

5 failures in 792 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 5

Platform breakdown:
* linux64-ccov: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 16

[Intermittent Failures Robot]

10 failures in 889 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 10

Platform breakdown:
* linux64-ccov: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 702 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414593&startday=2018-02-05&endday=2018-02-11&tree=all

Comment 18

[Marco Castelluccio [:marco]]

Any plan to fix this? Is this leak affecting real users too?

Comment 19

[Ed Lee :Mardak]

andreio had previously taken a look and sounds like he ran into issues reproducing it on try. The failures seem to be mostly for linux code coverage runs, so probably not affecting real users.

Comment 20

[Marco Castelluccio [:marco]]

(In reply to Ed Lee :Mardak from comment #19)
> andreio had previously taken a look and sounds like he ran into issues
> reproducing it on try. The failures seem to be mostly for linux code
> coverage runs, so probably not affecting real users.

Usually the only difference with code coverage builds is that they are slower than other builds. Maybe the leak somehow depends on timing.

Comment 21

[Intermittent Failures Robot]

1 failures in 5197 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1414593&startday=2019-06-03&endday=2019-06-09&tree=all