Intermittent Windows build Automation Error: mozprocess timed out after 2400 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v']

RESOLVED INCOMPLETE

Status

defect
RESOLVED INCOMPLETE
5 years ago
Last year

People

(Reporter: philor, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

No description provided.
will look at this at some point on monday. I will be on mergeduty but I hope to have time available in the afternoon
> python diagnose_intermittent_bug.py 1145507                                                                                                                 on default at tip!?
{'builders': [('WINNT 6.1 x86-64 mozilla-aurora build', 9),
              ('WINNT 6.1 x86-64 b2g-inbound pgo-build', 8),
              ('WINNT 6.1 x86-64 mozilla-inbound pgo-build', 5),
              ('WINNT 6.1 x86-64 try build', 3),
              ('WINNT 5.2 try build', 2),
              ('WINNT 6.1 x86-64 mozilla-central pgo-build', 2),
              ('WINNT 5.2 b2g-inbound build', 1),
              ('WINNT 5.2 fx-team pgo-build', 1),
              ('WINNT 6.1 x86-64 fx-team pgo-build', 1),
              ('WINNT 5.2 mozilla-inbound pgo-build', 1),
              ('WINNT 6.1 x86-64 fx-team leak test build', 1),
              ('WINNT 5.2 fx-team leak test build', 1),
              ('WINNT 5.2 mozilla-central pgo-build', 1)],
 'repos': [('mozilla-aurora', 9),
           ('b2g-inbound', 9),
           ('mozilla-inbound', 6),
           ('try', 5),
           ('fx-team', 4),
           ('mozilla-central', 3)],
 'slaves': [('b-2008-ix-0150', 6),
            ('b-2008-ix-0091', 2),
            ('b-2008-ix-0069', 2),
            ('b-2008-ix-0148', 2),
            ('b-2008-ix-0149', 1),
            ('b-2008-ix-0072', 1),
            ('b-2008-ix-0018', 1),
            ('b-2008-ix-0175', 1),
            ('b-2008-ix-0119', 1),
            ('b-2008-ix-0171', 1),
            ('b-2008-ix-0079', 1),
            ('b-2008-ix-0117', 1),
            ('b-2008-ix-0013', 1),
            ('b-2008-ix-0111', 1),
            ('b-2008-ix-0056', 1),
            ('b-2008-ix-0132', 1),
            ('b-2008-ix-0131', 1),
            ('b-2008-ix-0135', 1),
            ('b-2008-ix-0094', 1),
            ('b-2008-ix-0095', 1),
            ('b-2008-ix-0096', 1),
            ('b-2008-ix-0142', 1),
            ('b-2008-ix-0020', 1),
            ('b-2008-ix-0146', 1),
            ('b-2008-ix-0145', 1),
            ('b-2008-ix-0107', 1),
            ('b-2008-ix-0105', 1),
            ('b-2008-ix-0062', 1)],
 'times': {'2015-03-19': 8,
           '2015-03-20': 4,
           '2015-03-21': 7,
           '2015-03-22': 3,
           '2015-03-23': 8,
           '2015-03-24': 6}}

win64, pgo, and m-a are obviously worse but nothing is standing out from the logs so I don't think these failures are actual hung failures. Because this is parallelized, it's hard to know what is running and not outputting anything. Either way, 2400 seconds is a lot to ask for coming from 10800s so let's see if doubling the timeout to 4800 affects the rate this bug is hit.
Blocks: 1147271
FYI, I've been hearing complaints from devs hitting this on their Try pushes too.
Flags: needinfo?(jlund)
will update mh json once trees are open
Flags: needinfo?(jlund)
uplifted m-c mh json bump to aurora to fix comment 59-62

remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/9f3e4590ece1
I've been seeing this error on OS X try server repeatedly.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1436c62cf452
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fa8c9ce7dacd
https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d29442fa8c
OS: Windows Server 2003 → All
Summary: Intermittent Windows build Automation Error: mozprocess timed out after 2400 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] → Intermittent build Automation Error: mozprocess timed out after 2400 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v']
OS: All → Windows Server 2003
Hardware: x86_64 → All
Summary: Intermittent build Automation Error: mozprocess timed out after 2400 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] → Intermittent Windows build Automation Error: mozprocess timed out after 2400 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v']
(In reply to Hiroyuki Ikezoe (:hiro) from comment #71)
> I've been seeing this error on OS X try server repeatedly.
> 

thanks. to star this happening, we need to match the osx error exactly. since this bug is for windows and contains logs like 'c:\\mozilla-build\\python27\\python.exe', this won't catch osx. I've filed 1154377 to start tracking how often we hit this for mac.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.