987540 - 'gnome-panel.desktop' killed by signal 15 on Ubuntu machines

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect, P2)

Description

[Andrei Eftimie]

We've seen from time to time our Ubuntu machines lose the ability to receive VNC connections.
Sometimes this is solved by restarting lightdm. Sometimes a complete reboot is necessary.

On mm-ub-1310-32-3 we had such a failure last night.

Here is the relevant part (I'm making assumptions here):
> Mar 24 23:56:22 mm-ub-1310-32-3 gnome-session[1212]: WARNING: Application 'gnome-panel.desktop' killed by signal 15
> Mar 24 23:56:23 mm-ub-1310-32-3 whoopsie[912]: online
> Mar 24 23:56:27 mm-ub-1310-32-3 colord: device removed: xrandr-Virtual1
> Mar 24 23:56:28 mm-ub-1310-32-3 acpid: client 961[0:0] has disconnected
> Mar 24 23:56:28 mm-ub-1310-32-3 acpid: client connected from 32333[0:0]
> Mar 24 23:56:28 mm-ub-1310-32-3 acpid: 1 client rule loaded
> Mar 24 23:56:30 mm-ub-1310-32-3 dbus[357]: [system] Activating service name='org.freedesktop.systemd1' (using servicehelper)
> Mar 24 23:56:30 mm-ub-1310-32-3 dbus[357]: [system] Successfully activated service 'org.freedesktop.systemd1'
> Mar 24 23:56:30 mm-ub-1310-32-3 gnome-session[32443]: WARNING: Could not parse desktop file gwibber.desktop or it references a not found TryExec binary
> Mar 24 23:56:30 mm-ub-1310-32-3 rtkit-daemon[1606]: Successfully made thread 32517 of process 32517 (n/a) owned by '1000' high priority at nice level -11.
> Mar 24 23:56:30 mm-ub-1310-32-3 rtkit-daemon[1606]: Supervising 1 threads of 1 processes of 1 users.
> Mar 24 23:56:30 mm-ub-1310-32-3 pulseaudio[32517]: [pulseaudio] pid.c: Stale PID file, overwriting.
> Mar 24 23:56:30 mm-ub-1310-32-3 colord: Device added: xrandr-Virtual1
> Mar 24 23:56:30 mm-ub-1310-32-3 rtkit-daemon[1606]: Successfully made thread 32527 of process 32527 (n/a) owned by '1000' high priority at nice level -11.
> Mar 24 23:56:30 mm-ub-1310-32-3 rtkit-daemon[1606]: Supervising 2 threads of 2 processes of 1 users.
> Mar 24 23:56:30 mm-ub-1310-32-3 pulseaudio[32527]: [pulseaudio] pid.c: Daemon already running.
> Mar 24 23:56:30 mm-ub-1310-32-3 dbus[357]: [system] Activating service name='org.freedesktop.locale1' (using servicehelper)
> Mar 24 23:56:30 mm-ub-1310-32-3 dbus[357]: [system] Successfully activated service 'org.freedesktop.locale1'
> Mar 24 23:56:34 mm-ub-1310-32-3 goa[32764]: goa-daemon version 3.8.3 starting [main.c:113, main()]
> Mar 24 23:57:23 mm-ub-1310-32-3 whoopsie[912]: online

I can assume that the window manager failed. This made our future testruns fail (I can assume that without a proper working window manager Firefox wasn't able to properly start).

Here's a sample log from a testrun. Notice that it failed when trying to actually start Firefox and start running tests:
> 19:40:39 [mozilla-aurora_addons] $ mozmill-env-$ENV_PLATFORM/run testrun_addons --with-untrusted --repository=mozmill-tests --junit=report.xml --workspace=data --report=$REPORT_URL builds/
> 19:42:12 *** Installing build: /home/mozauto/jenkins/workspace/mozilla-aurora_addons/builds/2014-03-24-15-04-30-mozilla-aurora-firefox-30.0a2.it.linux-i686.tar.bz2
> 19:42:12 *** Application: Firefox 30.0a2 (/home/mozauto/jenkins/workspace/mozilla-aurora_addons/data/binary/firefox/firefox)
> 19:42:12 *** Platform: Linux Ubuntu 13.10 32bit
> 19:42:12 *** Cloning test repository to '/home/mozauto/jenkins/workspace/mozilla-aurora_addons/data/mozmill-tests'
> 19:42:12 *** Updating branch of test repository to 'mozilla-aurora'
> 19:42:12 *** Downloading http://release.seleniumhq.org/selenium-ide/editor/2.3.0/selenium-ide-editor-2.3.0.xpi to /home/mozauto/jenkins/workspace/mozilla-aurora_addons/data/addons/selenium-ide-editor-2.3.0.xpi
> 19:42:12 *** Creating profile: /home/mozauto/jenkins/workspace/mozilla-aurora_addons/data/profile
> 19:42:12 
> 19:42:12 (process:31913): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
> 19:42:12 RESULTS | Passed: 0
> 19:42:12 RESULTS | Failed: 0
> 19:42:12 RESULTS | Skipped: 0
> 19:42:13 Traceback (most recent call last):
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/mozmill_automation/testrun.py", line 465, in run_tests
> 19:42:13     TestRun.run_tests(self)
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/mozmill_automation/testrun.py", line 302, in run_tests
> 19:42:13     self._mozmill.run(tests, self.options.restart)
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/mozmill/__init__.py", line 429, in run
> 19:42:13     frame = self.run_test_file(frame or self.start_runner(),
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/mozmill/__init__.py", line 338, in start_runner
> 19:42:13     self.create_network()
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/mozmill/__init__.py", line 298, in create_network
> 19:42:13     self.jsbridge_port)
> 19:42:13   File "/home/mozauto/jenkins/workspace/mozilla-aurora_addons/mozmill-env-linux/python-lib/jsbridge/__init__.py", line 45, in wait_and_create_network
> 19:42:13     raise Exception("Cannot connect to jsbridge extension, port %s" % port)
> 19:42:13 Exception: Cannot connect to jsbridge extension, port 60878

All testruns failed in a similar fashion.
For now I have put the machine back online.

`signal 15` indicates that this is an active command, not a failure / crash. 
We should try figuring out why the 'gnome-panel.desktop' process is getting killed, and by whom.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Does it mean that whenever we see the jsbridge not connected failure on Ubuntu, it could be because of this bug?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The information as given by Andrei is very helpful and might help us to nail this down. Given that we had a lot of those jsbridge failures, we should prioritize this bug. Who can work on that?

Comment 3

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Does it mean that whenever we see the jsbridge not connected failure on
> Ubuntu, it could be because of this bug?

In this particular case all testruns failed. I do think we've seen this before as it's not uncommon for us to find one of the Linux machines in a state where it doesn't accept VNC connections and it ultimately needs a reboot. Half the time restarting lightdm seems to be enough. I do think that whenever I saw this Jenkins was also disconnected.

**
Some food for thought. I'm assuming we use Ubuntu distributions because of their relative high market share among linux distributions. In this case why are we switching Unity from Gnome? Shouldn't we use as much of the default as possible? I have no idea if this would be any different with Unity instead of Gnome. (Btw. I really dislike Unity)

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We didn't make this switch. It happens because we updated older versions of Ubuntu and unity hasn't been made the default yet. Something we should do is to get a fresh Ubuntu 14.04 setup once it is out. Maybe we are even facing other crufty old backlog stuff in our machines, which we might get rid of with a fresh install. So lets keep an eye on the new release.

Comment 5

[Andrei Eftimie]

Attachment: mm-ub-1310-64-3.syslog.txt

This happened again, this time on mm-ub-1310-64-3:
> Apr  7 05:59:51 mm-ub-1310-64-3 gnome-session[1393]: WARNING: Child process 1750 was already dead.
> Apr  7 05:59:51 mm-ub-1310-64-3 gnome-session[1393]: WARNING: Application 'gnome-panel.desktop' killed by signal 15

Same resolution as before, the machine needed a reboot, all testruns during this period failed.

Comment 6

[Andreea Matei [:AndreeaMatei]]

Haven't seen this so much and it will be fixed once we have Puppet running.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Mozmill-CI will be shutdown on Monday next week. So this is not needed anymore.