710712 - Measure peak virtual memory usage of link.exe process during libxul PGO link

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

bug 709193 makes it very hard for us to do development. We should monitor the peak virtual memory usage of link.exe when we're linking libxul the final time so we have some sort of benchmark for how close we are to the danger zone, so we can at least make reasonable decisions about what we're doing.

Comment 1

[(not currently active) Ted Mielczarek]

I have this mostly working, just testing it currently.

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: Measure peak virtual memory usage of link.exe, print and output to a file

This seems to do the trick locally. I've pushed it to try just to make sure it doesn't screw anything up there, but I think this should do what we want. It currently prints the measurement to stdout, as well as writing it to $objdir/toolkit/library/linker-vsize.

https://tbpl.mozilla.org/?tree=Try&rev=a076385f7cb7

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

Review of attachment 581941 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!

Comment 4

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/integration/mozilla-inbound/rev/3c8600c97454

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Should we make the build automatically go orange if the linker memory usage goes over some danger threshold? Seems like that would be helpful to warn us if we're about to hit the wall.

Comment 6

[Ed Morley [:emorley]]

> Should we make the build automatically go orange

See bug 710840 comment 1 :-)

Comment 7

[Mike Hommey [:glandium]]

On the last pgo build on inbound currently (7a0d03046d42):
linker max virtual size: 2266345472

That seems reaaaally low. It would be good to try on one of the changesets that was failing to link.

Comment 8

[(not currently active) Ted Mielczarek]

On the changeset I pushed to try (which was about a month out of date), it was ~2.83GB. We can certainly push this to try on top of some of the failing changesets to get data points, but I'm fairly confident in the measurement here. It matched Process Explorer's number exactly (which makes sense because it's calling the exact same API).

Comment 9

[Mike Hommey [:glandium]]

Which suggests we have managed to reduce the memory usage by more than 700MB, that's an order of magnitude more than what Kyle has seen.

Comment 10

[Mike Hommey [:glandium]]

I'm tempted to say this would be interesting to land on beta and aurora.

Comment 11

[(not currently active) Ted Mielczarek]

On 718a7411a72b, it measured at 2265296896, which is exactly 1MB more, which seems like pretty decent consistency.

Comment 12

[(not currently active) Ted Mielczarek]

Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

This is a build-system-only change that simply adds reporting to the amount of virtual memory used by the linker during our PGO builds. Since exhausting the virtual address space has disastrous consequences, this should give us fair warning if we're about to destroy our ability to ship Firefox.

Comment 13

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Mike Hommey [:glandium] from comment #9)
> Which suggests we have managed to reduce the memory usage by more than
> 700MB, that's an order of magnitude more than what Kyle has seen.

I would find it incredibly hard to believe that we've reduced linker memory usage by 700 MB.

Comment 14

[Mike Hommey [:glandium]]

Another data point that would be interesting to have (but not necessary tracked, Try is probably enough) is the value for non-PGO builds.

Comment 15

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/3c8600c97454

Comment 16

[(not currently active) Ted Mielczarek]

Turns out that gavin broke PGO builds in an insidious way in bug 696436 (bug 711478), so I can't give you the non-PGO number, but I can tell you that those numbers above are "PGO without any profiling data", which is why they're much lower than expected. A non-broken inbound run measured at 3027816448, which is 2.82GB, which sounds a lot more reasonable.

(If you wanted to test this on non-PGO runs, you could just drop the ifdefs around the Makefile hunk in this patch.)

Comment 17

[Alex Keybl [:akeybl]]

Comment on attachment 581941 [details] [diff] [review]
Measure peak virtual memory usage of link.exe, print and output to a file

[Triage Comment]
Approved for Aurora and Beta.

Comment 18

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b15863afb438

Comment 19

[u279076]

Marking qa- as I don't think this is something QA needs to verify. Please mark qa+ if this is not the case.