Bug 676724 (DMD)

DMD: A Valgrind tool to find dark matter

RESOLVED FIXED

Status

()

defect
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: njn, Assigned: njn)

Tracking

(Blocks 3 bugs)

Trunk
All
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 8 obsolete attachments)

Assignee

Description

8 years ago
[The following text is from bug 563700 comment 23]

I was thinking some more about how to get *serious* about hunting down all the heap-unclassified bytes.

You'd add instrumentation code to maintain a data structure that records 1 bit of information for every malloc'd byte.  A 0 means "not reported", a 1 means "reported".  You'd also add instrumentation code to record, for every heap block, the stack trace at its allocation point.

Then, you'd modify every heap memory reporter so that when it is queried, it sets the "reported" bits of all the heap bytes it counts.

Once about:memory was loaded, you'd iterate over all the heap blocks.  Any heap block fully reported would be ignored.  Any heap block partially or not-at-all reported would be recorded.  You'd aggregate repeated stack traces for recorded heap blocks and present the stack traces in order so that the ones responsible for the most unreported bytes are shown first.

Also, while the memory reporters are doing their thing, you'd complain about any byte on the heap whose "reported" bit was set more than once -- such bytes are double-counted.  You could print out its stack trace and the name of the reporter that was second to count it.

Tracking one-bit-per-byte metadata and per-heap-block stack traces is something that Valgrind excels at, BTW.  A Valgrind tool that implemented this analysis would have to use its own heap allocator rather than jemalloc (that's just how Valgrind works), so it would have to be careful about the rounding up of request sizes, but it could definitely be made to work.  Memory reporters would use client requests to tell the tool which bytes have been accounted for.  A client request would also be used to tell the tool when all the memory reporters have been consulted.
Assignee

Updated

8 years ago
Blocks: DarkMatter
Assignee

Comment 1

8 years ago
In addition to the Valgrind tool, each memory reporter will have to be modified to include annotations that tell the tool which heap bytes have been accounted for.  These annotations wouldn't be landed.
Assignee

Updated

8 years ago
Summary: Write a Valgrind tool to accurately identify what's missing from about:memory's "heap-unclassified" value → Write a Valgrind tool to find dark matter
(In reply to Nicholas Nethercote [:njn] from comment #1)
> In addition to the Valgrind tool, each memory reporter will have to be
> modified to include annotations that tell the tool which heap bytes have
> been accounted for.  These annotations wouldn't be landed.

Why not?  If they're not landed, then:

 * I have to patch FF to run the tool, and
 * that patch probably will bitrot.

I presume this is something we're going to want to run every once in a while; it's not fix once and forget.
Assignee

Comment 3

8 years ago
Well, let's see how intrusive they are.
Assignee

Updated

8 years ago
Alias: DMD
Assignee

Updated

8 years ago
Summary: Write a Valgrind tool to find dark matter → A Valgrind tool to find dark matter
Assignee

Updated

8 years ago
Blocks: 677466
Assignee

Updated

8 years ago
Blocks: 677653
Hardware: x86_64 → All
Version: unspecified → Trunk
Assignee

Updated

8 years ago
Whiteboard: [MemShrink] → [MemShrink:P1]
Assignee

Updated

8 years ago
Blocks: 678126
Blocks: 678376
Assignee

Updated

8 years ago
Blocks: 678811
Assignee

Comment 4

8 years ago
Posted patch DMD, v1 (obsolete) — Splinter Review
This is an in-progress version of DMD.  Features:

- Reports unreported heap blocks.

- Reports double-reported heap blocks.

- Reports how many bytes each annotated reporter reported.  This is useful for cross-checking with about:memory.

- Tracks both requested and slop bytes; emulates jemalloc's round-up behaviour.

- Has a couple of regression tests.

This patch applies to an SVN trunk version of Valgrind (I have r11976).  If you want to use it, you'll need to follow the usual Valgrind tricks for Firefox, see https://developer.mozilla.org/en/Debugging_Mozilla_with_Valgrind.  Trunk builds of Valgrind are easy, follow the instructions in Valgrind's top-level README file.

You'll need the --tool=exp-dmd flag, and you probably want to set --num-callers to something low like 6, otherwise lots of records that could be sensibly merged won't be.  (Even with 6 you'll still get some like that, but if you ask for fewer than 6 sometimes the stack traces will be too shallow to be useful.)
Assignee

Comment 5

8 years ago
Posted patch Firefox annotations, v1 (obsolete) — Splinter Review
Some DMD annotations for Firefox.  Many more need to be done.
Assignee

Updated

8 years ago
Blocks: 678977
Blocks: 680358
Assignee

Comment 6

8 years ago
Posted patch DMD, v2 (obsolete) — Splinter Review
Assignee

Comment 7

8 years ago
Posted patch Firefox annotations, v2 (obsolete) — Splinter Review
Attachment #553103 - Attachment is obsolete: true
Attachment #553104 - Attachment is obsolete: true

Updated

8 years ago
Assignee

Updated

8 years ago
Whiteboard: [MemShrink:P1] → [MemShrink]
Did you mean to change this from [MemShrink:P1] to [MemShrink]?  If so, why?
Assignee

Comment 9

8 years ago
I did, I want to re-triage it.  The bug obviously shouldn't be closed, but I don't think it needs to be a P1 any more now that it's in a state where it's spitting out useful numbers.
(In reply to Nicholas Nethercote [:njn] from comment #9)
> I did, I want to re-triage it.  The bug obviously shouldn't be closed, but I
> don't think it needs to be a P1 any more now that it's in a state where it's
> spitting out useful numbers.

Might we call it fixed?
Assignee

Updated

8 years ago
Whiteboard: [MemShrink] → [MemShrink:P2]
Assignee

Comment 11

8 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #10)
> 
> Might we call it fixed?

I'll still be posting new versions of it, as well as new versions of the patch annotating Firefox.  Marking it closed when I'll still be working on it doesn't feel right.
Blocks: 682215
Blocks: 682216
Assignee

Updated

8 years ago
Blocks: 126212
Assignee

Updated

8 years ago
Blocks: 262234
Blocks: 682219
Assignee

Updated

8 years ago
Blocks: 693016
Assignee

Updated

8 years ago
Summary: A Valgrind tool to find dark matter → DMD: A Valgrind tool to find dark matter
Assignee

Updated

8 years ago
Blocks: 696690
Assignee

Comment 12

8 years ago
Posted patch DMD, v3 (obsolete) — Splinter Review
This version also tracks allocations done with mmap, which led me to bug 696690.
Attachment #554346 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Blocks: 692748
Assignee

Updated

8 years ago
Blocks: 697041
Assignee

Updated

8 years ago
Blocks: 697332
Assignee

Updated

8 years ago
Blocks: 697335
Assignee

Updated

8 years ago
Blocks: 698326
Assignee

Comment 13

8 years ago
Posted patch Firefox annotations, v3 (obsolete) — Splinter Review
Attachment #554347 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Blocks: 699721
Assignee

Comment 14

8 years ago
Posted patch DMD, v4 (obsolete) — Splinter Review
Attachment #569249 - Attachment is obsolete: true
Assignee

Comment 15

8 years ago
Posted patch Firefox annotations, v4 (obsolete) — Splinter Review
Attachment #570932 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Blocks: 704400
Assignee

Updated

8 years ago
No longer blocks: 704400
Depends on: 704400
Assignee

Updated

8 years ago
Blocks: 704723
Assignee

Updated

8 years ago
Blocks: 707842
Assignee

Comment 16

8 years ago
Comment on attachment 571888 [details] [diff] [review]
DMD, v4

Bug 704400 folded DMD into the tree, so this patch is no longer needed.
Attachment #571888 - Attachment is obsolete: true
Assignee

Comment 17

8 years ago
Comment on attachment 571889 [details] [diff] [review]
Firefox annotations, v4

Ugh, wrong patch.
Attachment #571889 - Attachment is obsolete: true
Assignee

Comment 18

8 years ago
Posted patch DMD, v5Splinter Review
Assignee

Updated

8 years ago
Blocks: 708762
Assignee

Updated

8 years ago
Blocks: NewDMD
Assignee

Updated

7 years ago
Blocks: 722595
Assignee

Updated

7 years ago
Blocks: 723799
Assignee

Updated

7 years ago
Blocks: 726764
Assignee

Updated

7 years ago
Blocks: 729008
Assignee

Updated

7 years ago
Blocks: 729403
Assignee

Updated

7 years ago
No longer blocks: NewDMD
Depends on: NewDMD
Assignee

Updated

7 years ago
Whiteboard: [MemShrink:P2]
Assignee

Comment 19

7 years ago
DMDV (i.e. the Valgrind version of DMD) is now in the tree.  There's no need to keep this bug open.
Assignee

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.