Closed Bug 692708 Opened 9 years ago Closed 9 years ago

Talos old_tp perma-red on mozilla-beta on all platforms: NS_ERROR_FILE_NOT_FOUND at pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 605

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: mbrubeck, Assigned: armenzg)

References

Details

(Whiteboard: [red])

These failures started on 2011-09-05 (Wednesday).  On this push:
https://tbpl.mozilla.org/php/getParsedLog.php?id=6697342&tree=Mozilla-Beta

...there was a green old_tp run at 13:00 that day:
https://tbpl.mozilla.org/php/getParsedLog.php?id=6689229&tree=Mozilla-Beta

...but by 21:56 it was red on the same push, same platform:
https://tbpl.mozilla.org/php/getParsedLog.php?id=6697342&tree=Mozilla-Beta

and has been perma-red ever since.  Example log:

https://tbpl.mozilla.org/php/getParsedLog.php?id=6717876&tree=Mozilla-Beta
Rev3 MacOSX Leopard 10.5.8 mozilla-beta talos old_tp on 2011-10-06 20:49:54 PDT for push 570515d49188

========= Started 'python run_tests.py ...' failed (results: 2, elapsed: 27 secs) ==========
python run_tests.py --noisy 20111006_2012_config.yml
 in dir /home/cltbld/talos-slave/test/../talos-data/talos/ (timeout 21600 secs)
 watching logfiles {}
 argv: ['python', 'run_tests.py', '--noisy', '20111006_2012_config.yml']
 environment:
  CVS_RSH=ssh
  DISPLAY=:0
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoreboth
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=talos-r3-fed-063.build.mozilla.org
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_NO_REMOTE=1
  NO_EM_RESTART=1
  PATH=/home/cltbld/bin:/tools/buildbot-0.8.0/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
  PWD=/home/cltbld/talos-slave/talos-data/talos
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  TERM=xterm
  USER=cltbld
  XPCOM_DEBUG_BREAK=warn
  _=/home/cltbld/bin/python
 using PTY: False
RETURN:s: talos-r3-fed-063
RETURN:id:20111006165549
RETURN:<a href = "http://hg.mozilla.org/releases/mozilla-beta/rev/8f6cce824102">rev:8f6cce824102</a>
talos-r3-fed-063: 
		Started Thu, 06 Oct 2011 20:12:33
Running test tp5: 
		Started Thu, 06 Oct 2011 20:12:33
	Screen width/height:1600/1200
	colorDepth:24
	Browser inner width/height: 1024/650

NOISE: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileInputStream.init]"  nsresult: "0x80520012 (NS_ERROR_FILE_NOT_FOUND)"  location: "JS frame :: chrome://pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 605"  data: no]
NOISE: __startSecondTimestamp1317957170603__endSecondTimestamp
Traceback (most recent call last):
  File "run_tests.py", line 540, in <module>
    test_file(arg, screen, amo)
  File "run_tests.py", line 485, in test_file
    browser_dump, counter_dump, print_format = mytest.runTest(browser_config, test)
  File "/home/cltbld/talos-slave/talos-data/talos/ttest.py", line 290, in runTest
    cm = self.cmanager.CounterManager(self._ffprocess, browser_config['process'], counters)
  File "/home/cltbld/talos-slave/talos-data/talos/cmanager_linux.py", line 150, in __init__
    self.primaryPid = self.ffprocess.GetPidsByName(process)[-1]
IndexError: list index out of range
program finished with exit code 1
elapsedTime=26.898262
TinderboxPrint:s: talos-r3-fed-063

TinderboxPrint:id:20111006165549

TinderboxPrint:<a href = "http://hg.mozilla.org/releases/mozilla-beta/rev/8f6cce824102">rev:8f6cce824102</a>
======== Finished 'python run_tests.py ...' failed (results: 2, elapsed: 27 secs) ========
This is probabaly a fallout from the PGO changes last night for releases.
A mozconfig[1] change landed at Oct 06 at 17:44:15 -0700

A better way to look at this is:
https://tbpl.mozilla.org/?tree=Mozilla-Beta&noignore=1&jobname=old_tp

mbrubeck thanks for filing this.

jmaher, could you please give me a hand at looking at the error?
old_tp is the only suite that started failing.

As mbrubeck says:
* The last round that went green was on Oct. 6th at 15:04 PDT
* The first one that went red was Oct. 6th at 21:56 PDT - retriggered for the same changeset

This cannot be seen anywhere else as we don't run old_tp anywhere else anymore.

Removing "randomorange" dependency as this is a "permared".

[1] http://hg.mozilla.org/build/buildbot-configs/rev/876f80347ed7
Assignee: nobody → armenzg
No longer blocks: 438871
Depends on: 692646
Summary: Talos old_tp/tp5 perma-red on mozilla-beta on all platforms: NS_ERROR_FILE_NOT_FOUND at pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 605 → Talos old_tp perma-red on mozilla-beta on all platforms: NS_ERROR_FILE_NOT_FOUND at pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 605
No longer depends on: 692646
Blocks: 658313
Hidden on tbpl since apparently nobody's going to stop pushing just because the tree is covered with unstarred red, and all we're doing is training them that it's fine to do so.
odd, I ensured I had the latest talos and pageloader bits from hg, and ran tp5 (without mozafterpaint- old_tp5 parity) against a mozilla-beta build.  This ran with no problems.  I even copied the command lines from the tbpl log for a failing instance and just changed the location of the firefox binary as well as the graph servers.

There has to be something wrong in the way we have things setup, but really odd that this is only old_tp5.
Whiteboard: [orange][red] → [red]
(In reply to Joel Maher (:jmaher) from comment #3)
> odd, I ensured I had the latest talos and pageloader bits from hg, ...

Did you take into account that the tip of the repo has changes since the last update on production ? True of talos at least.
maybe tp5 pageset isn't download and/or unzipped?  this error in pageloader.js is for when we are trying to open the tp5.manifest file.
Priority: -- → P3
We disabled old_tp for beta a couple of weeks ago.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.