483199 - Not enough memory to link on try server linux unittest builder ["/usr/bin/ld: final link failed: Memory exhausted"]

Status: VERIFIED DUPLICATE of bug 498393

(Release Engineering :: General, defect, P2)

Description

[Henri Sivonen (:hsivonen)]

I pushed to try a trivial whitespace-only patch that shouldn't have changed the linking characteristics of the build compared to trunk: http://hg.mozilla.org/try/rev/9d0ba628b36f (I did this when the try server was idle in order to establish a baseline for unit tests and Tp for a real patch.)

The build went red due to insufficient memory to link the compiled code:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1236940857.1236946414.27015.gz

Comment 1

[[e-mail] (:bhearsum)]

I see lots more of these. They're all on try-linux-slave01. I'll have a look and see what's up on it.

Comment 2

[[e-mail] (:bhearsum)]

I'm not really sure what's up here...it's got the same amount of RAM as the rest of the Linux VMs and its ESX host seems fine. I'm going to reboot it as soon as its idle and see if that helps..

Comment 3

[[e-mail] (:bhearsum)]

Alright, that slave has been rebooted...we'll see if the next build it gets goes OK.

Comment 4

[[e-mail] (:bhearsum)]

Seems that rebooting this slave did the trick.

Comment 5

[Daniel Holbert [:dholbert]]

This happened again today:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1240523248.1240526989.23173.gz
Try linux hg unittest builder on 2009/04/23 14:47:28
/usr/bin/ld: final link failed: Memory exhausted
collect2: ld returned 1 exit status

Note that the following cycle was green with no issues, so this appears to be a sporadic problem.

Reopening, since this doesn't seem to be resolved nor fixed. :)

Comment 6

[Chris AtLee [:catlee]]

Will reboot this slave when it becomes idle.

Comment 7

[Chris AtLee [:catlee]]

rebooted

Comment 8

[Chris AtLee [:catlee]]

FTR, this latest machine was try-linux-slave03.mozilla.org

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Machine building fine (green) after reboot. Closing.

Comment 10

[Daniel Holbert [:dholbert]]

Looks like this might have happened again, despite fixed-ness:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245268473.1245271437.30933.gz

Quoting from that log:
/usr/bin/ld: final link failed: Memory exhausted
collect2: ld returned 1 exit status
make[4]: *** [libxul.so] Error 1
make[3]: *** [libs_tier_toolkit] Error 2
make[2]: *** [tier_toolkit] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2

Comment 11

[Daniel Holbert [:dholbert]]

Tentatively reopening, per comment 10.

BTW, looks like the affected machine in that log is try-linux-slave15.mozilla.org

Comment 12

[Aki Sasaki (not active)]

Maybe we should morph this into "auto-reboot try slaves after X number of builds".

Comment 13

[Aki Sasaki (not active)]

Rebooted try-linux-slave15 for the quick symptom fix.
Filed bug 499043 for the long term real fix.

Comment 14

[Daniel Holbert [:dholbert]]

This just happened also on try-linux-slave19.mozilla.org -- Aki, could you reboot that one as well?
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245281607.1245284031.25198.gz

Comment 15

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> Maybe we should morph this into "auto-reboot try slaves after X number of
> builds".

Maybe. It would be nice to understand the root cause here, though...

Comment 16

[Aki Sasaki (not active)]

Rebooting try-linux-slave19 for dholbert after 2days, 21:26 uptime, which doesn't seem all that long at all. Esp. for linux.

Comment 17

[Daniel Holbert [:dholbert]]

Just hit this on try-linux-slave13:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1247716232.1247718964.1412.gz

Comment 18

[Daniel Holbert [:dholbert]]

For the record, this happened on the same box (try-linux-slave13) at least two other times during the last week:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1247635232.1247637697.31700.gz
2009/07/15 20:50:32

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1247273912.1247276682.29815.gz
2009/07/10 17:58:32

Reopening -- perhaps this bug should stay open until this is actually fixed "for good"? :)

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Duping this since there is a dependent bug attached to bug 498393 which is for adding RAM to the Linux builders.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Both the errors listed in comment #18 are from try-linux-slave13, which has probably been intermittently busted since Jun 25 (limited history on the master, so can't confirm that). There's a firefox-bin and ssltunnel from that.

The fix to buildbot.tac was made Jun 28 (bug 498393 comment #10) so I must have failed to clean up old processes then. I've gone around the try linux slaves and verified they're all OK now.

Verifying DUPE.