Closed Bug 483199 Opened 13 years ago Closed 13 years ago

Not enough memory to link on try server linux unittest builder ["/usr/bin/ld: final link failed: Memory exhausted"]

Categories

(Release Engineering :: General, defect, P2)

x86
Linux
defect

Tracking

(Not tracked)

VERIFIED DUPLICATE of bug 498393

People

(Reporter: hsivonen, Assigned: bhearsum)

References

()

Details

I pushed to try a trivial whitespace-only patch that shouldn't have changed the linking characteristics of the build compared to trunk: http://hg.mozilla.org/try/rev/9d0ba628b36f (I did this when the try server was idle in order to establish a baseline for unit tests and Tp for a real patch.)

The build went red due to insufficient memory to link the compiled code:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1236940857.1236946414.27015.gz
I see lots more of these. They're all on try-linux-slave01. I'll have a look and see what's up on it.
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Priority: -- → P2
I'm not really sure what's up here...it's got the same amount of RAM as the rest of the Linux VMs and its ESX host seems fine. I'm going to reboot it as soon as its idle and see if that helps..
Alright, that slave has been rebooted...we'll see if the next build it gets goes OK.
Seems that rebooting this slave did the trick.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
This happened again today:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1240523248.1240526989.23173.gz
Try linux hg unittest builder on 2009/04/23 14:47:28
/usr/bin/ld: final link failed: Memory exhausted
collect2: ld returned 1 exit status

Note that the following cycle was green with no issues, so this appears to be a sporadic problem.

Reopening, since this doesn't seem to be resolved nor fixed. :)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Will reboot this slave when it becomes idle.
rebooted
FTR, this latest machine was try-linux-slave03.mozilla.org
Machine building fine (green) after reboot. Closing.
Status: REOPENED → RESOLVED
Closed: 13 years ago13 years ago
Resolution: --- → FIXED
Looks like this might have happened again, despite fixed-ness:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245268473.1245271437.30933.gz

Quoting from that log:
/usr/bin/ld: final link failed: Memory exhausted
collect2: ld returned 1 exit status
make[4]: *** [libxul.so] Error 1
make[3]: *** [libs_tier_toolkit] Error 2
make[2]: *** [tier_toolkit] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2
Tentatively reopening, per comment 10.

BTW, looks like the affected machine in that log is try-linux-slave15.mozilla.org
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Maybe we should morph this into "auto-reboot try slaves after X number of builds".
Rebooted try-linux-slave15 for the quick symptom fix.
Filed bug 499043 for the long term real fix.
Status: REOPENED → RESOLVED
Closed: 13 years ago13 years ago
Resolution: --- → FIXED
This just happened also on try-linux-slave19.mozilla.org -- Aki, could you reboot that one as well?
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245281607.1245284031.25198.gz
(In reply to comment #12)
> Maybe we should morph this into "auto-reboot try slaves after X number of
> builds".

Maybe. It would be nice to understand the root cause here, though...
Rebooting try-linux-slave19 for dholbert after 2days, 21:26 uptime, which doesn't seem all that long at all. Esp. for linux.
For the record, this happened on the same box (try-linux-slave13) at least two other times during the last week:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1247635232.1247637697.31700.gz
2009/07/15 20:50:32

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1247273912.1247276682.29815.gz
2009/07/10 17:58:32

Reopening -- perhaps this bug should stay open until this is actually fixed "for good"? :)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Not enough memory to link on try server linux unittest builder → Not enough memory to link on try server linux unittest builder ["/usr/bin/ld: final link failed: Memory exhausted"]
Duping this since there is a dependent bug attached to bug 498393 which is for adding RAM to the Linux builders.
Status: REOPENED → RESOLVED
Closed: 13 years ago13 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 498393
Both the errors listed in comment #18 are from try-linux-slave13, which has probably been intermittently busted since Jun 25 (limited history on the master, so can't confirm that). There's a firefox-bin and ssltunnel from that.

The fix to buildbot.tac was made Jun 28 (bug 498393 comment #10) so I must have failed to clean up old processes then. I've gone around the try linux slaves and verified they're all OK now.

Verifying DUPE.
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.