Measure peak virtual memory usage of link.exe process during libxul PGO link

RESOLVED FIXED in Firefox 10

Status

()

Core
Build Config
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: ted, Assigned: ted)

Tracking

Trunk
mozilla11
x86
Windows 7
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox9 affected, firefox10 fixed, firefox11 fixed)

Details

(Whiteboard: [qa-])

Attachments

(1 attachment)

(Assignee)

Description

6 years ago
bug 709193 makes it very hard for us to do development. We should monitor the peak virtual memory usage of link.exe when we're linking libxul the final time so we have some sort of benchmark for how close we are to the danger zone, so we can at least make reasonable decisions about what we're doing.
(Assignee)

Updated

6 years ago
See Also: → bug 709193
(Assignee)

Updated

6 years ago
Assignee: nobody → ted.mielczarek
(Assignee)

Updated

6 years ago
Blocks: 710840
(Assignee)

Comment 1

6 years ago
I have this mostly working, just testing it currently.
(Assignee)

Comment 2

6 years ago
Created attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

This seems to do the trick locally. I've pushed it to try just to make sure it doesn't screw anything up there, but I think this should do what we want. It currently prints the measurement to stdout, as well as writing it to $objdir/toolkit/library/linker-vsize.

https://tbpl.mozilla.org/?tree=Try&rev=a076385f7cb7
(Assignee)

Updated

6 years ago
Attachment #581941 - Flags: review?(catlee)
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

Review of attachment 581941 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!
Attachment #581941 - Flags: review?(catlee) → review+
(Assignee)

Comment 4

6 years ago
http://hg.mozilla.org/integration/mozilla-inbound/rev/3c8600c97454
Status: NEW → ASSIGNED
Should we make the build automatically go orange if the linker memory usage goes over some danger threshold? Seems like that would be helpful to warn us if we're about to hit the wall.

Comment 6

6 years ago
> Should we make the build automatically go orange

See bug 710840 comment 1 :-)
On the last pgo build on inbound currently (7a0d03046d42):
linker max virtual size: 2266345472

That seems reaaaally low. It would be good to try on one of the changesets that was failing to link.
(Assignee)

Comment 8

6 years ago
On the changeset I pushed to try (which was about a month out of date), it was ~2.83GB. We can certainly push this to try on top of some of the failing changesets to get data points, but I'm fairly confident in the measurement here. It matched Process Explorer's number exactly (which makes sense because it's calling the exact same API).
Which suggests we have managed to reduce the memory usage by more than 700MB, that's an order of magnitude more than what Kyle has seen.
I'm tempted to say this would be interesting to land on beta and aurora.
(Assignee)

Comment 11

6 years ago
On 718a7411a72b, it measured at 2265296896, which is exactly 1MB more, which seems like pretty decent consistency.
(Assignee)

Comment 12

6 years ago
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

This is a build-system-only change that simply adds reporting to the amount of virtual memory used by the linker during our PGO builds. Since exhausting the virtual address space has disastrous consequences, this should give us fair warning if we're about to destroy our ability to ship Firefox.
Attachment #581941 - Flags: approval-mozilla-beta?
Attachment #581941 - Flags: approval-mozilla-aurora?
(In reply to Mike Hommey [:glandium] from comment #9)
> Which suggests we have managed to reduce the memory usage by more than
> 700MB, that's an order of magnitude more than what Kyle has seen.

I would find it incredibly hard to believe that we've reduced linker memory usage by 700 MB.
Another data point that would be interesting to have (but not necessary tracked, Try is probably enough) is the value for non-PGO builds.
https://hg.mozilla.org/mozilla-central/rev/3c8600c97454
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla11
(Assignee)

Comment 16

6 years ago
Turns out that gavin broke PGO builds in an insidious way in bug 696436 (bug 711478), so I can't give you the non-PGO number, but I can tell you that those numbers above are "PGO without any profiling data", which is why they're much lower than expected. A non-broken inbound run measured at 3027816448, which is 2.82GB, which sounds a lot more reasonable.

(If you wanted to test this on non-PGO runs, you could just drop the ifdefs around the Makefile hunk in this patch.)
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

[Triage Comment]
Approved for Aurora and Beta.
Attachment #581941 - Flags: approval-mozilla-beta?
Attachment #581941 - Flags: approval-mozilla-beta+
Attachment #581941 - Flags: approval-mozilla-aurora?
Attachment #581941 - Flags: approval-mozilla-aurora+
https://hg.mozilla.org/releases/mozilla-aurora/rev/b15863afb438
status-firefox10: --- → fixed
status-firefox11: --- → fixed
status-firefox9: --- → affected
Marking qa- as I don't think this is something QA needs to verify. Please mark qa+ if this is not the case.
Whiteboard: [qa-]

Updated

5 years ago
Blocks: 750717
You need to log in before you can comment on or make changes to this bug.