Last Comment Bug 720636 - Permanent orange: TEST-UNEXPECTED-FAIL | test_TelemetryPing.js | false == true
: Permanent orange: TEST-UNEXPECTED-FAIL | test_TelemetryPing.js | false == true
Status: RESOLVED FIXED
[qa-]
: intermittent-failure
Product: Core
Classification: Components
Component: Graphics (show other bugs)
: Trunk
: All Mac OS X
: -- major (vote)
: mozilla13
Assigned To: Mark Banner (:standard8)
:
:
Mentors:
Depends on:
Blocks: 706340
  Show dependency treegraph
 
Reported: 2012-01-24 01:50 PST by Mark Banner (:standard8)
Modified: 2012-11-25 19:31 PST (History)
11 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
fixed


Attachments
Possible fix (3.20 KB, patch)
2012-02-10 13:08 PST, Mark Banner (:standard8)
jd.bugzilla: review+
akeybl: approval‑mozilla‑aurora+
Details | Diff | Splinter Review

Description Mark Banner (:standard8) 2012-01-24 01:50:34 PST
Since bug 706340 landed in mozilla-central, we're seeing this failure:

TEST-UNEXPECTED-FAIL | /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetryPing.js | false == true - See following stack:
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_throw :: line 462
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: _do_check_eq :: line 556
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_check_eq :: line 577
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_check_true :: line 591
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetryPing.js :: checkHistograms :: line 85
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/Daily.app/Contents/MacOS/components/httpd.js :: <TOP_LEVEL> :: line 2272
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/Daily.app/Contents/MacOS/components/httpd.js :: <TOP_LEVEL> :: line 1169
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/Daily.app/Contents/MacOS/components/httpd.js :: <TOP_LEVEL> :: line 1617
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/Daily.app/Contents/MacOS/components/httpd.js :: <TOP_LEVEL> :: line 1465
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/Daily.app/Contents/MacOS/components/httpd.js :: <TOP_LEVEL> :: line 1334

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-INFO | (xpcshell/head.js) | test 1 finished

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-UNEXPECTED-FAIL | /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetryPing.js | false == true - See following stack:
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_throw :: line 462
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: _do_check_eq :: line 556
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_check_eq :: line 577
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: do_check_true :: line 591
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetryPing.js :: <TOP_LEVEL> :: line 195
JS frame :: /builds/slave/comm-central-macosx64-opt-unittest-xpcshell/build/xpcshell/head.js :: _execute_test :: line 363
JS frame :: -e :: <TOP_LEVEL> :: line 1

It looks like:

Cc["@mozilla.org/gfx/info;1"].getService(Ci.nsIGfxInfo).adapterVendorID (and .adapterDeviceID)

are both null for us.

I don't understand why that would be.

Could we be having an issue if graphics acceleration is turned off by default?
Comment 1 Mark Banner (:standard8) 2012-01-24 03:56:27 PST
Ok, I get this if I sudo to another user on my mac and run the tests - hence the sudo shell isn't able to access the display drivers.

John/Gozer, can you guys have a look and see if we're running our Mac setup differently to Firefox somehow? (this is only seen on Mac).

I do find it a bit strange that MozMill tests are fine, but xpcshell isn't though.
Comment 2 John Daggett (:jtd) 2012-01-24 04:49:14 PST
(In reply to Mark Banner (:standard8) from comment #1)
> Ok, I get this if I sudo to another user on my mac and run the tests - hence
> the sudo shell isn't able to access the display drivers.
> 
> John/Gozer, can you guys have a look and see if we're running our Mac setup
> differently to Firefox somehow? (this is only seen on Mac).
> 
> I do find it a bit strange that MozMill tests are fine, but xpcshell isn't
> though.

Any idea if it's possible to detect that situation?  We can just pull the check for those two values but it would be nice to have it tested somewhere (it's already limited to osx/win for the same reason as this bug).
Comment 3 Mark Banner (:standard8) 2012-01-24 06:35:44 PST
(In reply to John Daggett (:jtd) from comment #2)
> Any idea if it's possible to detect that situation?  We can just pull the
> check for those two values but it would be nice to have it tested somewhere
> (it's already limited to osx/win for the same reason as this bug).

I don't know of any way. I'm not sure we're really designed to run without a window ;-)
Comment 4 John Hopkins (:jhopkins) 2012-01-26 14:03:47 PST
I'm currently trying to reproduce this on staging and will see if I can find a fix.
Comment 5 Joe Drew (not getting mail) 2012-01-27 09:05:11 PST
Thunderbird disables hardware acceleration, and I'm pretty sure that results in us never initializing GfxInfo, so we never add these fields to Telemetry.

The best way to fix this is to always initialize GfxInfo, I think. Is there any reason for us not to do that?
Comment 6 Mark Banner (:standard8) 2012-01-27 09:41:19 PST
I pushed this to Thunderbird try:

pref("layers.acceleration.disabled", true);
pref("layers.acceleration.disabled", false);

but it didn't fix it, does it need something else as well?

I also could reproduce this failure on my Mac locally.

- If I ran the tests in my normal development login, then they passed.

- However if I logged into a different account, used su to get to my development account, and then ran them, they failed.

Hence why I was suspecting more of a builder issue than a code issue.
Comment 7 Mark Banner (:standard8) 2012-02-09 05:40:40 PST
Latest debug from try server:

Init GfxInfoBase
GfxInfo::Init
Getting device info
Vendor: , Device: 

So for some reason we're not getting the device info when we should.

gozer tells me that the builders are from the exact same system image that Firefox uses, so there shouldn't be an issue there.

My only thought currently, is that as we're testing the telementry ping, and not the graphics info, is how would people feel about inserting a fake gfxInfo for debug mode tests, the same way as is done for some other tests? For example:

http://mxr.mozilla.org/comm-central/source/mozilla/toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Equal_OK.js#36
Comment 8 Mark Banner (:standard8) 2012-02-09 06:01:49 PST
I just noticed SeaMonkey is seeing this as well. So this definitely isn't hardware accel, as they have it enabled by default.

Looking back at out 10.5 builders now that we have had lots more builds over the week - tb2-darwin9-slave70 is passing, the rest aren't. Hence this does look more like a buildbot config issue. I wonder if there's a setting or something in Mac that could cause this?

I'll talk with gozer and jhopkins about this later.
Comment 9 Philippe M. Chiasson (:gozer) 2012-02-09 09:20:11 PST
Just decided to compare tb2-darwin9-slave70 with tb2-darwin9-slave72, and after removing the cruft, there are some differences, looks like.

Library/ColorSync/Profiles/Displays/Display-5B81C5C0.icc
Library/Keychains/System.keychain
Library/LaunchDaemons/com.reductivelabs.puppet.plist
Library/Logs/DirectoryService/DirectoryService.server.log
Library/Logs/DirectoryService/DirectoryService.server.log.1
Library/Logs/DirectoryService/DirectoryService.server.log.2
Library/Logs/DirectoryService/DirectoryService.server.log.3
Library/Logs/DirectoryService/DirectoryService.server.log.4
Library/Logs/DirectoryService/DirectoryService.server.log.5
Library/Logs/DirectoryService/DirectoryService.server.log.6
Library/Logs/DirectoryService/DirectoryService.server.log.7
Library/Logs/DirectoryService/DirectoryService.server.log.8
Library/Preferences/.GlobalPreferences.plist
Library/Preferences/DirectoryService/.DSIsRunning
Library/Preferences/SystemConfiguration/NetworkInterfaces.plist
Library/Preferences/SystemConfiguration/com.apple.network.identification.plist
Library/Preferences/SystemConfiguration/com.apple.smb.server.plist
Library/Preferences/SystemConfiguration/preferences.plist
Library/Preferences/SystemConfiguration/preferences.plist.old
Library/Preferences/cfx#61mDDsC
Library/Preferences/cfx#69gaWVw
Library/Preferences/cfx#75noL37
Library/Preferences/cfx#FxSOI14
Library/Preferences/cfx#M1oGY2E
Library/Preferences/cfx#TTK2qj5
Library/Preferences/cfx#VeXMjeV
Library/Preferences/cfx#hCBeRF9
Library/Preferences/cfx#zqUkSIZ
Library/Preferences/com.apple.ARDAgent.plist
Library/Preferences/com.apple.AppleFileServer.plist
Library/Preferences/com.apple.BezelServices.plist
Library/Preferences/com.apple.Bluetooth.plist
Library/Preferences/com.apple.RemoteManagement.launchd
Library/Preferences/com.apple.RemoteManagement.plist
Library/Preferences/com.apple.TimeMachine.plist
Library/Preferences/com.apple.VNCSettings.txt
Library/Preferences/com.apple.audio.DeviceSettings.plist
Library/Preferences/com.apple.audio.SystemSettings.plist
Library/Preferences/com.apple.loginwindow.plist
Library/Preferences/com.apple.security.systemidentities.plist
Library/Receipts/clang-3.0-r135723.moz0.pkg/Contents/Info.plist
Library/Receipts/repackaging-tools-10.5.pkg/Contents/Info.plist
Library/Receipts/simplejson-2.1.1-py25.pkg/Contents/Info.plist
Library/Receipts/yasm-1.1.0.pkg/Contents/Info.plist
System/Library/CoreServices/.disk_label
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/UserDict.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/codecs.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy_reg.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/encodings/__init__.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/encodings/aliases.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/encodings/ascii.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/linecache.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/os.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/posixpath.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/stat.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/types.pyc
System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/warnings.pyc
System/Library/LaunchDaemons/com.apple.WindowServer.plist
System/Library/LaunchDaemons/com.apple.blued.plist
System/Library/LaunchDaemons/com.apple.kdcmond.plist
System/Library/LaunchDaemons/com.apple.loginwindow.plist
Users/cltbld/.bash_history
Users/cltbld/.lesshst
Users/cltbld/.ssh/known_hosts
Users/cltbld/.viminfo
Users/cltbld/Library/Application Support/AddressBook/.database.lockN
Users/cltbld/Library/Application Support/AddressBook/Metadata/.info
Users/cltbld/Library/Logs/DiscRecording.log
Users/cltbld/Library/Logs/fsck_hfs.log
Users/cltbld/Library/Preferences/ByHost/.GlobalPreferences.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.Bluetooth.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.SoftwareUpdate.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.iCal.helper.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.preference.displays.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.scheduler.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.screensaver.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.systempreferences.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/com.apple.windowserver.0016cbae26ff.plist
Users/cltbld/Library/Preferences/ByHost/pbs.0016cbae26ff.plist
Users/cltbld/Library/Preferences/QuickTime Preferences
Users/cltbld/Library/Preferences/com.apple.BezelServices.plist
Users/cltbld/Library/Preferences/com.apple.DownloadAssessment.plist
Users/cltbld/Library/Preferences/com.apple.LaunchServices.plist
Users/cltbld/Library/Preferences/com.apple.PreferenceSync.plist
Users/cltbld/Library/Preferences/com.apple.PubSubAgent.plist
Users/cltbld/Library/Preferences/com.apple.desktop.plist
Users/cltbld/Library/Preferences/com.apple.dock.db
Users/cltbld/Library/Preferences/com.apple.dock.plist
Users/cltbld/Library/Preferences/com.apple.finder.plist
Users/cltbld/Library/Preferences/com.apple.internetconfigpriv.plist
Users/cltbld/Library/Preferences/com.apple.quicktime.plugin.preferences.plist
Users/cltbld/Library/Preferences/com.apple.recentitems.plist
Users/cltbld/Library/Preferences/com.apple.spotlight.plist
Users/cltbld/Library/Preferences/com.apple.systempreferences.plist
Users/cltbld/Library/Preferences/com.apple.systemuiserver.plist
Users/cltbld/Library/Preferences/org.mozilla.crashreporter.plist
Users/cltbld/Library/Preferences/org.mozilla.daily.plist
Users/cltbld/Library/Preferences/org.mozilla.dailydebug.plist
Users/cltbld/Library/Preferences/org.mozilla.earlybird.plist
Users/cltbld/Library/Preferences/org.mozilla.earlybirddebug.plist
Users/cltbld/Library/Preferences/org.mozilla.thunderbird.plist
Users/cltbld/Library/PubSub/Database/Database.sqlite3
Users/cltbld/Library/Thunderbird/console.log
Users/cltbld/Library/Thunderbird/profiles.ini
Users/cltbld/hg
opt/local/etc/cacert.pem
opt/local/etc/wgetrc
usr/X11/man/whatis
usr/local/bin/addon_id
usr/local/bin/check-for-package.sh
usr/local/bin/jsbridge
usr/local/bin/mozmill
usr/local/bin/mozmill-restart
usr/local/bin/mozmill-thunderbird
usr/local/bin/mozrunner
usr/local/bin/runslave.py
usr/local/bin/setup-configuration.sh
usr/local/bin/setup-nagios-user.sh
usr/local/bin/sleep-and-run-puppet.sh
usr/local/share/man/whatis
usr/share/man/whatis
Comment 10 Mark Banner (:standard8) 2012-02-09 11:58:21 PST
Turning off vnc on the builders hasn't worked. I'm currently pondering if there's other bits we can check.
Comment 11 Justin Wood (:Callek) 2012-02-09 22:02:06 PST
(In reply to Mark Banner (:standard8) from comment #8)
> I just noticed SeaMonkey is seeing this as well. So this definitely isn't
> hardware accel, as they have it enabled by default.
> 

SeaMonkey failures due to hwaccel is not uncommon (our plan for these hwaccel issues is to disable the test/fix the test), Our mac slaves are currently a mix of r3/r2 mini's and all setup as builders. I don't think we even have any of the hwaccel graphic cards installed, but I can't recall for certain.
Comment 12 Mark Banner (:standard8) 2012-02-10 13:08:11 PST
Created attachment 596151 [details] [diff] [review]
Possible fix

I'm currently testing this on try servers. The basic idea for the patch is that with the telemetry ping test we are not trying to test obtaining the information from the hardware.

Therefore we use the fake gfx info available in debug mode and if we're in release, then we don't test.
Comment 13 Mozilla RelEng Bot 2012-02-11 09:16:03 PST
Try run for ce232a5245c6 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=ce232a5245c6
Results (out of 26 total builds):
    exception: 1
    success: 23
    warnings: 2
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bugzilla@standard8.plus.com-ce232a5245c6
Comment 14 Mark Banner (:standard8) 2012-02-11 09:40:25 PST
Comment on attachment 596151 [details] [diff] [review]
Possible fix

John/Joe, how do you feel about this patch? I think it just does enough to make sure we get the result out, whilst not depending on the hardware available for the test.
Comment 15 Mark Banner (:standard8) 2012-02-13 02:07:50 PST
Landed on inbound:

https://hg.mozilla.org/integration/mozilla-inbound/rev/37cecf145092
Comment 16 Marco Bonardo [::mak] 2012-02-13 09:11:03 PST
https://hg.mozilla.org/mozilla-central/rev/37cecf145092
Comment 17 Mark Banner (:standard8) 2012-02-14 11:56:44 PST
Comment on attachment 596151 [details] [diff] [review]
Possible fix

[Approval Request Comment]
Regression caused by (bug #): bug 706340
User impact if declined: None
Testing completed (on m-c, etc.): About 2 days
Risk to taking this patch (and alternatives if risky): Test-only change, no risk
String changes made by this patch: none

This is a test-only change to fix some permanent orange for c-c apps caused by assuming that we have hardware accelerated cards on all machines, when we don't need to be testing that here.
Comment 18 Alex Keybl [:akeybl] 2012-02-15 16:04:43 PST
Comment on attachment 596151 [details] [diff] [review]
Possible fix

[Triage Comment]
Test-only fix, approved for Aurora 12.
Comment 19 Mark Banner (:standard8) 2012-02-20 01:58:55 PST
Landed in aurora: https://hg.mozilla.org/releases/mozilla-aurora/rev/dfb7c3567c49

Note You need to log in before you can comment on or make changes to this bug.