Last Comment Bug 710712 - Measure peak virtual memory usage of link.exe process during libxul PGO link
: Measure peak virtual memory usage of link.exe process during libxul PGO link
Status: RESOLVED FIXED
[qa-]
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Windows 7
: -- normal with 2 votes (vote)
: mozilla11
Assigned To: Ted Mielczarek [:ted.mielczarek]
:
Mentors:
Depends on:
Blocks: 710840 750717
  Show dependency treegraph
 
Reported: 2011-12-14 07:56 PST by Ted Mielczarek [:ted.mielczarek]
Modified: 2012-05-01 07:14 PDT (History)
16 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
affected
fixed
fixed


Attachments
Measure peak virtual memory usage of link.exe, print and output to a file (8.15 KB, patch)
2011-12-15 05:56 PST, Ted Mielczarek [:ted.mielczarek]
catlee: review+
akeybl: approval‑mozilla‑aurora+
akeybl: approval‑mozilla‑beta+
Details | Diff | Review

Description Ted Mielczarek [:ted.mielczarek] 2011-12-14 07:56:40 PST
bug 709193 makes it very hard for us to do development. We should monitor the peak virtual memory usage of link.exe when we're linking libxul the final time so we have some sort of benchmark for how close we are to the danger zone, so we can at least make reasonable decisions about what we're doing.
Comment 1 Ted Mielczarek [:ted.mielczarek] 2011-12-14 12:27:44 PST
I have this mostly working, just testing it currently.
Comment 2 Ted Mielczarek [:ted.mielczarek] 2011-12-15 05:56:24 PST
Created attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

This seems to do the trick locally. I've pushed it to try just to make sure it doesn't screw anything up there, but I think this should do what we want. It currently prints the measurement to stdout, as well as writing it to $objdir/toolkit/library/linker-vsize.

https://tbpl.mozilla.org/?tree=Try&rev=a076385f7cb7
Comment 3 Chris AtLee [:catlee] 2011-12-15 08:13:28 PST
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

Review of attachment 581941 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!
Comment 4 Ted Mielczarek [:ted.mielczarek] 2011-12-15 10:31:45 PST
http://hg.mozilla.org/integration/mozilla-inbound/rev/3c8600c97454
Comment 5 Robert O'Callahan (:roc) (Exited; email my personal email if necessary) 2011-12-15 16:04:55 PST
Should we make the build automatically go orange if the linker memory usage goes over some danger threshold? Seems like that would be helpful to warn us if we're about to hit the wall.
Comment 6 Ed Morley [:emorley] 2011-12-15 16:07:58 PST
> Should we make the build automatically go orange

See bug 710840 comment 1 :-)
Comment 7 Mike Hommey [:glandium] 2011-12-15 22:16:09 PST
On the last pgo build on inbound currently (7a0d03046d42):
linker max virtual size: 2266345472

That seems reaaaally low. It would be good to try on one of the changesets that was failing to link.
Comment 8 Ted Mielczarek [:ted.mielczarek] 2011-12-16 05:04:24 PST
On the changeset I pushed to try (which was about a month out of date), it was ~2.83GB. We can certainly push this to try on top of some of the failing changesets to get data points, but I'm fairly confident in the measurement here. It matched Process Explorer's number exactly (which makes sense because it's calling the exact same API).
Comment 9 Mike Hommey [:glandium] 2011-12-16 05:09:14 PST
Which suggests we have managed to reduce the memory usage by more than 700MB, that's an order of magnitude more than what Kyle has seen.
Comment 10 Mike Hommey [:glandium] 2011-12-16 05:10:29 PST
I'm tempted to say this would be interesting to land on beta and aurora.
Comment 11 Ted Mielczarek [:ted.mielczarek] 2011-12-16 05:12:11 PST
On 718a7411a72b, it measured at 2265296896, which is exactly 1MB more, which seems like pretty decent consistency.
Comment 12 Ted Mielczarek [:ted.mielczarek] 2011-12-16 05:16:48 PST
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

This is a build-system-only change that simply adds reporting to the amount of virtual memory used by the linker during our PGO builds. Since exhausting the virtual address space has disastrous consequences, this should give us fair warning if we're about to destroy our ability to ship Firefox.
Comment 13 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-12-16 05:23:27 PST
(In reply to Mike Hommey [:glandium] from comment #9)
> Which suggests we have managed to reduce the memory usage by more than
> 700MB, that's an order of magnitude more than what Kyle has seen.

I would find it incredibly hard to believe that we've reduced linker memory usage by 700 MB.
Comment 14 Mike Hommey [:glandium] 2011-12-16 05:38:00 PST
Another data point that would be interesting to have (but not necessary tracked, Try is probably enough) is the value for non-PGO builds.
Comment 15 Ed Morley [:emorley] 2011-12-16 06:11:21 PST
https://hg.mozilla.org/mozilla-central/rev/3c8600c97454
Comment 16 Ted Mielczarek [:ted.mielczarek] 2011-12-16 08:12:23 PST
Turns out that gavin broke PGO builds in an insidious way in bug 696436 (bug 711478), so I can't give you the non-PGO number, but I can tell you that those numbers above are "PGO without any profiling data", which is why they're much lower than expected. A non-broken inbound run measured at 3027816448, which is 2.82GB, which sounds a lot more reasonable.

(If you wanted to test this on non-PGO runs, you could just drop the ifdefs around the Makefile hunk in this patch.)
Comment 17 Alex Keybl [:akeybl] 2011-12-16 12:31:52 PST
Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

[Triage Comment]
Approved for Aurora and Beta.
Comment 18 Matt Brubeck (:mbrubeck) 2011-12-16 13:26:40 PST
https://hg.mozilla.org/releases/mozilla-aurora/rev/b15863afb438
Comment 19 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-28 13:11:20 PST
Marking qa- as I don't think this is something QA needs to verify. Please mark qa+ if this is not the case.

Note You need to log in before you can comment on or make changes to this bug.