481402 - momo-vm-15/13 don't have enough resources for nightlies?

Status: RESOLVED FIXED

(Mozilla Messaging Graveyard :: Server Operations, defect)

Description

[Mark Banner (:standard8)]

Twice now momo-vm-15 has failed checking out due to not enough resources:

http://build.mozillamessaging.com/buildbot/production/builders/Win2k3%20comm-central%20trunk%20nightly/builds/69
http://build.mozillamessaging.com/buildbot/production/builders/Win2k3%20comm-central%20trunk%20nightly/builds/71

I had a quick look around but couldn't see anything obvious going wrong.

Comment 1

[Philippe M. Chiasson (:gozer)]

For the record, that's mercurial/python aborting with:

abort: Insufficient system resources exist to complete the requested service

Comment 2

[Philippe M. Chiasson (:gozer)]

At least one other Mercurial user has experienced something similar...

http://www.selenic.com/mercurial/bts/issue1418

Not a very helpfull bug report, but still, worth linking here.

Comment 3

[Mark Banner (:standard8)]

momo-vm-13 had this error today: http://build.mozillamessaging.com/buildbot/production/builders/Win2k3%20comm-central%20trunk%20nightly/builds/78

Comment 4

[Philippe M. Chiasson (:gozer)]

Smelling more and more like some sort of Mercurial bug, or some mercurial-triggered win32/python bug.

I wonder if upgrading to Mercurial 1.2 might help (or at least change things)

Comment 5

[Philippe M. Chiasson (:gozer)]

Upgraded momo-vm-15 to Mercurial 1.2, let's see if that helps some.

Comment 6

[Mark Banner (:standard8)]

momo-vm-15 failed on a nightly build today, then threw a complete wobbly, turning red 3 other builds before it had time to stop for a few minutes and recover.

getting layout/reftests/bugs/413286-1-ref.html
abort: Insufficient system resources exist to complete the requested service
Executing command: ['hg', 'clone', '--debug', 'http://hg.mozilla.org/mozilla-central', '.\\mozilla']
Traceback (most recent call last):
  File "client.py", line 281, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev)
  File "client.py", line 135, in do_hg_pull
    check_call_noisy([hg, 'clone'] + hgopts + [repository, fulldir])
  File "client.py", line 54, in check_call_noisy
    check_call(cmd, *args, **kwargs)
  File "d:\mozilla-build\python25\lib\subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'clone', '--debug', 'http://hg.mozilla.org/mozilla-central', '.\\mozilla']' returned non-zero exit status -1
program finished with exit code 1

Could it be that --debug is just too much for these VMs?

I also can't access that VM via RDC at the moment so something isn't happy there.

Comment 7

[Philippe M. Chiasson (:gozer)]

(In reply to comment #6)
> momo-vm-15 failed on a nightly build today, then threw a complete wobbly,
> turning red 3 other builds before it had time to stop for a few minutes and
> recover.

Now that is seriously strange indeed.

> [...]
> 
> Could it be that --debug is just too much for these VMs?

That's possible, I think it's worth trying to disable --debug for a while,
see if it makes a difference,

> I also can't access that VM via RDC at the moment so something isn't happy
> there.

Worked for me just now.

Comment 8

[Philippe M. Chiasson (:gozer)]

Replaced --debug with --verbose --time on the production buildbot master, let's see what happens next.

Comment 9

[Mark Banner (:standard8)]

(In reply to comment #8)
> Replaced --debug with --verbose --time on the production buildbot master, let's
> see what happens next.

The last bad checkout was the day after you made this comment I think (http://build.mozillamessaging.com/buildbot/production/builders/Win2k3%20comm-central%20trunk%20nightly/builds/115), since then the pulls have been fine even if the later steps haven't.

We can probably close this now, thoughts?

Comment 10

[Philippe M. Chiasson (:gozer)]

Probably, but I suspect the underlying problem is still hiding.

let's close this bug and open new ones if/when new symptoms hit us.