Closed Bug 1146713 Opened 10 years ago Closed 9 years ago

[emulator] mach mochitest fails: expected to find ssltunnel at .../gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel

Categories

(Firefox OS Graveyard :: Gaia::Build, defect, P1)

Product:
Firefox OS Graveyard
Component:
Gaia::Build
Platform:
ARM
Gonk (Firefox OS)
Type:
defect

Tracking

(blocking-b2g:2.5+)

Status:
RESOLVED FIXED
Project Flags:
blocking-b2g 2.5+

People

(Reporter: mikeh, Assigned: rickychien)

References

Details

(Keywords: regression, Whiteboard: [workaround in comment 160])

Attachments

(16 files)

logcat output leading up to abort
86.17 KB, text/plain
Details
0001-Bug-1146713-Don-t-export-NO_PKG_FILES-fix-ssltunnel-.patch
3.17 KB, patch
glandium
: review+
Details | Diff | Splinter Review
[gaia] rickychien:b2g43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
mt
: review+
Details | Review
[gaia] mozilla-b2g:revert-31393-b2g43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:b2g-43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:b2g-43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
timdream
: review+
Details | Review
[gaia] rickychien:echo-LD_LIBRARY_PATH > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:hardcode-gtk3-path > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:fix-tooltool-path > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] mozilla-b2g:revert-31750-fix-tooltool-path > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] mozilla-b2g:revert-31749-hardcode-gtk3-path > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:b2g-43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
[gaia] rickychien:b2g-43 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
B2G patch for downloading b2g_sdk by itself
43 bytes, text/x-github-pull-request
ahal
: review+
Details | Review
follow up for removing mozinstall
43 bytes, text/x-github-pull-request
ahal
: review+
Details | Review
[gaia] rickychien:remove-debug-log > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Details | Review
STR: 1. ./config.sh emulator 2. ./build.sh 3. ./mach mochitest-remote dom/camera/test/test_camera_hardware_init_failure.html Expected: Test runs to successful completion. Observed: MochitestServer : launching [u'/home/mikeh/dev/mozilla/b2g/051_emulator/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/xpcshell', '-g', u'/home/mikeh/dev/mozilla/b2g/051_emulator/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g', '-v', '170', '-f', u'/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpqpGEsH'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.230.153'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/server.js'] runtests.py | Server pid: 5940 runtests.py | Websocket server pid: 5943 0 ERROR INFO | runtests.py | expected to find ssltunnel at /home/mikeh/dev/mozilla/b2g/051_emulator/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel Traceback (most recent call last): File "/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/runtestsb2g.py", line 193, in run_tests self.startServers(options, None) File "/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/runtestsb2g.py", line 336, in startServers MochitestUtilsMixin.startServers(self, options, debuggerInfo) File "/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/runtests.py", line 821, in startServers self.sslTunnel.start() File "/home/mikeh/dev/mozilla/b2g/051_emulator/objdir-gecko-debug/_tests/testing/mochitest/runtests.py", line 1063, in start exit(1) File "/usr/lib/python2.7/site.py", line 375, in __call__ raise SystemExit(code) SystemExit: 1 1 ERROR Automation Error: Received unexpected exception while running application Stopping web server Stopping web socket server Stopping ssltunnel runtestsb2g.py | Running tests: end. SUITE-END | took 162s logcat: 03-24 00:35:39.392 44 44 I Gecko : 1427157339400 Marionette INFO sendToClient: {"from":"0","value":true}, {f6facf86-2d0b-420b-aaca-c8c38d35cfec}, {f6facf86-2d0b-420b-aaca-c8c38d35cfec} 03-24 00:35:40.022 44 44 I Gecko : [44] WARNING: Channel provided to SetRequestContext is not an nsIHttpChannel so referrer is not available for reporting.: file /home/mikeh/dev/mozilla/b2g/051_emulator/gecko/dom/security/nsCSPContext.cpp, line 616 03-24 00:35:40.112 44 44 I Gecko : [44] WARNING: Subdocument container has no frame: file /home/mikeh/dev/mozilla/b2g/051_emulator/gecko/layout/base/nsDocumentViewer.cpp, line 2511 03-24 00:35:40.112 44 44 I Gecko : ++DOMWINDOW == 8 (0x464e1320) [pid = 44] [serial = 8] [outer = 0x450e2100] 03-24 00:35:46.572 44 44 I Gecko : [44] WARNING: Bluetooth service is not ready yet!: file /home/mikeh/dev/mozilla/b2g/051_emulator/gecko/dom/bluetooth/bluez/BluetoothDBusService.cpp, line 2066 03-24 00:35:47.962 44 44 I Gecko : 1427157347962 Marionette DEBUG Got request: deleteSession, data: {"to":"0","sessionId":"{ee847dee-9425-439b-8555-f5d877853311}","name":"deleteSession"}, id: null 03-24 00:35:48.022 44 44 I Gecko : 1427157348029 Marionette INFO sendToClient: {"from":"0","ok":true}, {745e4ca7-4945-4ae6-a808-90f9c82a6766}, {745e4ca7-4945-4ae6-a808-90f9c82a6766} 03-24 00:35:48.892 44 44 I Gecko : [44] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file /home/mikeh/dev/mozilla/b2g/051_emulator/gecko/dom/base/nsScriptLoader.cpp, line 376 03-24 00:35:49.792 44 44 I GeckoConsole: Content JS LOG: BluetoothHelper(): connot get default adapter!!! 03-24 00:35:49.792 44 44 I GeckoConsole: at BluetoothHelper/_fetchAdapter/req.onerror (app://callscreen.gaiamobile.org/shared/js/bluetooth_helper.js:117:8) 03-24 00:35:50.542 33 33 I ServiceManager: service 'media.resource_manager' died 03-24 00:35:50.542 33 33 I ServiceManager: service 'permission' died
Note: I am running with DEBUG=1.
Build: - gecko: master/cfa695b0e3184a77d437dc63e6da2269bd0bdbe1 - gaia: master/d5fb945b48069a6745c60617e3203e3f9ca3ea01
Further to comment 1, I've confirmed that non-DEBUG builds have the same problem.
ahal, any idea what's going on here?
Flags: needinfo?(ahalberstadt)
I just noticed this in the output: expected to find ssltunnel at /home/mikeh/dev/mozilla/b2g/051_emulator/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel This fixed it: # cp b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/ssltunnel b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g
See Also: → 1002545
So is the bug here that ssltunnel is no longer packaged with the latest b2g_sdk? Sounds like a build/packaging bug, though probably good to leave this open for other people who run into the same problem and file a new bug.
Flags: needinfo?(ahalberstadt)
(In reply to Andrew Halberstadt [:ahal] from comment #6) > So is the bug here that ssltunnel is no longer packaged with the latest > b2g_sdk? Sounds like a build/packaging bug, though probably good to leave > this open for other people who run into the same problem and file a new bug. It might be easier to find with the real error message in the summary. (The “service … died” messages show up whenever b2g exits, regardless of how/why, I think?) Interestingly, if I just symlink in the ssltunnel binary from my desktop Firefox objdir, it complains about not finding libnssdbm3.so (required by libsoftokn3.so which is dlopen()ed) and failing to initialize NSS, but tests still run. I assume it would break more dramatically if I were trying to run a test that used HTTPS.
Summary: [Emulator] aborts early: service 'media.resource_manager' died | service 'permission' died → [emulator] mach mochitest-remote fails: expected to find ssltunnel at .../gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel
We noticed this when working on bug 1153774, treeherder log https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/DqT5uuBQQYqKtikMpXcvmg/0/public/logs/live_backing.log | 0 ERROR INFO | runtests.py | expected to find ssltunnel at /home/worker/build/xre/bin/ssltunnel|
Flags: needinfo?(jlal)
Hi Martin, I am also setting NI flag on you as I learned that you are one of the contributors of bug 1002545 which looks very similar to this one. Do you have any idea here? Thank you.
Flags: needinfo?(martin.thomson)
Ricky, could this related to bug 1089710?
Flags: needinfo?(ricky060709)
Or, maybe the question to ask should be, would it possible for us to identify the bug that remove the ssltunnel from B2G package?
:timdream is right. Find out who removed ssltunnel and work from there.
Flags: needinfo?(martin.thomson)
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #11) > Ricky, could this related to bug 1089710? I believe this bug results from bug 1089710; however, it doesn't explicit show how ssltunnel wasn't in nightly build anymore from that bug. Maybe Ricky has more to comment. John and Seinlin have been tried to figure out how and where ssltunnel is packed or removed but still have no ideas. Very appreciate if we could get some help from nightly build peers.
bug 1089710 only pulled in changes that caused this. The real problem is in gecko, where the mulet build was modified to omit ssltunnel. I can't remember which of the files is the one that matters here, but the patches on bug 1019117 include the original changes to add it to the build.
Right. Someone has to bisect the B2G Desktop builds since bug 1019117.
Flags: needinfo?(ricky060709)
I can try to run mozregression on the range but it would be much faster if someone closer to the nightly servers can do it.
I've just finish $ mozregression -n b2g --good 2014-08-12 --bad 2015-03-05 on Mac and ssltunnel is always there.
... and for some reason mozregression on Ubuntu ignore |-n| option so I can't bisect B2G.
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #19) > ... and for some reason mozregression on Ubuntu ignore |-n| option so I > can't bisect B2G. I was not using the latest version of mozregression on that machine.
The regressed bug should be Bug 1109136. Michael, do you know why moving packager instruction resulting lost of ssltunnel from Linux B2G Desktop packaging?
Flags: needinfo?(mshal)
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #22) > The regressed bug should be Bug 1109136. Michael, do you know why moving > packager instruction resulting lost of ssltunnel from Linux B2G Desktop > packaging? It looks like upload-files.mk is being parsed first by moz-automation.mk, which doesn't have MOZ_PKG_MANIFEST set. Since NO_PKG_FILES is exported into the environment, ssltunnel already exists in NO_PKG_FILES when upload-files.mk is parsed again later by packager.mk. Because it's in NO_PKG_FILES, it gets skipped by packager.py: Warning: /home/worker/workspace/object-folder/browser/installer/package-manifest:1241: NO_PKG_FILES contains file listed in manifest: ssltunnel I think we should not be using export here from make, since it makes the lifetime of the variable confusing. Instead we can just add it to the environment of packager.py when it is invoked.
Flags: needinfo?(mshal)
try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea52651757c6 ssltunnel exists in the b2g desktop linux64 package with this. I moved the export of USE_ELF_HACK/ELF_HACK_FLAGS back to packager.mk so they aren't exported for things that aren't run by the packager. Since NO_PKG_FILES uses += everywhere and export is confusing, I just pass NO_PKG_FILES into packager.py's environment instead.
Assignee: nobody → mshal
Attachment #8604831 - Flags: review?(mh+mozilla)
Attachment #8604831 - Flags: review?(mh+mozilla) → review+
We would need to again bump the version of Gecko for Gaia build once this is fixed.
Keywords: regression
Marking leave-open until the version bump.
Keywords: leave-open
Reassigning to handle the version bump. I'm not sure if you're the right person, so feel free to redirect :)
Assignee: mshal → timdream
Ricky could you do the honor here? Either in this bug or in another bug.
Assignee: timdream → ricky060709
Flags: needinfo?(jlal) → needinfo?(ricky060709)
It seems that ssltunnel disappear in B2G linux build. (I found it still locate on b2g 34 or 39 mac64 build) I'm trying to upgrade B2G to 41.0a1 but cannot find newer linux build on ftp, see also http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2015-05-11-16-02-05-mozilla-central/ Does everyone know where can I find newer B2G linux build?
Flags: needinfo?(ricky060709)
ni Michael if you could answer my question on comment 31. =)
Flags: needinfo?(mshal)
(In reply to Ricky Chien [:rickychien] from comment #32) > ni Michael if you could answer my question on comment 31. =) The linux b2g builds are done on taskcluster, so they don't upload to ftp. I like to use the Indexed Artifact Browser from https://tools.taskcluster.net/ - then you can drill down to get the artifacts like: https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.b2g-desktop/gecko.v1.mozilla-central.latest.linux.b2g-desktop.opt and grab public/build/target.linux-x86_64.tar.bz2, which should have ssltunnel. Feel free to ping me if you're having trouble! Also note that the patch only landed on 5-14, so I expect anything before that won't have it.
Flags: needinfo?(mshal)
Flags: needinfo?(ricky060709)
I think it's awful to hear that linux build no longer hosts on ftp server =( Unfortunately, it's hard to specify a B2G version by date & version as usual from ftp, so that we cannot know what's B2G version running on. On the other hand, if older B2G can work properly, I'd like to solve this issue simply by switching back to old version. Tim, do you have any suggestions for this?
Flags: needinfo?(ricky060709) → needinfo?(timdream)
(In reply to Ricky Chien [:rickychien] from comment #34) > On the other hand, if older B2G can work properly, I'd like to solve this > issue simply by switching back to old version. > Tim, do you have any suggestions for this? You can't. Read bug 1089710. We need the new B2G for some feature in the build.
Flags: needinfo?(timdream)
(In reply to Ricky Chien [:rickychien] from comment #34) > I think it's awful to hear that linux build no longer hosts on ftp server =( > Unfortunately, it's hard to specify a B2G version by date & version as usual > from ftp, so that we cannot know what's B2G version running on. Yeah, it's scary because it basically blocks us from upgrading b2g_sdk any further (see bug 1157321). > On the other hand, if older B2G can work properly, I'd like to solve this > issue simply by switching back to old version. We rely on features not available in older B2G. Also, I don't think we should lock ourselves in an old version that will be unsupported very soon. I believe we need to be able to keep our builds in sync with the release, so on master we should update b2g_sdk to 41.0 now...
(In reply to Michael Shal [:mshal] from comment #33) > Feel free to ping me if you're having trouble! Michael, based on what you said, would you recommend updating our Makefile to be able to pull linux builds from taskcluster, and mac/win builds from FTP? How can we match the names/dates to get the right URLs?
Flags: needinfo?(mshal)
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #37) > How can we match the names/dates to get the right URLs? +1 for this. The best solution for Gaia developers to see which version of B2G running is downloading it with a specific version/date.
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #37) > Michael, based on what you said, would you recommend updating our Makefile > to be able to pull linux builds from taskcluster, and mac/win builds from > FTP? > > How can we match the names/dates to get the right URLs? Mac/win builds are also on taskcluster, though we haven't unified the index yet (so it's not ideal in how we grab files from things that are generated by taskcluster vs. things that are generated by buildbot and then uploaded to taskcluster - see bug 1133074 and bug 1159700). Those can still be pulled from FTP until we sort that all out, though. We don't have support for dates in the index yet for nightly builds, though that will be part of those changes. IIRC it is possible to go through the pushlog to grab revision numbers, and then use those to key on the taskcluster index. James, anything I'm missing / other advice?
Flags: needinfo?(mshal) → needinfo?(jlal)
Hi James, Could you please help check comment 39? Thanks!
Redirecting to jonasfj, in case he has thoughts.
Flags: needinfo?(jlal) → needinfo?(jopsen)
@mshal, seems you have it all covered... If you find a place "key1.key2.key3" in index that you want to download artifacts from. You can use the new findArtifactFromTask method [1], constructing your URL as: index.taskcluster.net/v1/task/<namespace>/artifacts/<name> Example: index.taskcluster.net/v1/task/gecko.v1.mozilla-central.latest.../artifacts/public/logs/live.log [1] http://docs.taskcluster.net/services/index/#findArtifactFromTask
Flags: needinfo?(jopsen)
Blocks: 1181561
Ricky, what's the status ? There's a landed patch but this is still badly broken. Anyone needing to investigate locally mochitest on emulator is blocked.
Flags: needinfo?(rchien)
From the aspect of Gaia developer, we don't know where is the revision number from (Does it mean git hash for Gaia commit?). The old fashioned way we downloaded b2g-desktop by a simple url like http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2015/03/2015-03-01-01-02-23-mozilla-central/b2g-39.0a1.multi.mac64.dmg so that anyone can replace this version what they want. Does we still provide such old fashioned way to download b2g-desktop through task-cluster APIs?
Flags: needinfo?(rchien) → needinfo?(jopsen)
> Does we still provide such old fashioned way to download b2g-desktop through task-cluster APIs? Did you create the URL from a pattern? It's not simple to me, unless copy-pasted it from somewhere :) If the URL is constructed from a pattern using nightly dates, then no. But I think mshal is working to index nightlies by date, there is certainly a bug for that. If you're asking if you can get a URL for a b2g-desktop download, then the answer is yes. But at the moment I think the URL patterns available employ revision or just "latest", which updates often...
Flags: needinfo?(jopsen)
(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #45) > > Does we still provide such old fashioned way to download b2g-desktop through task-cluster APIs? > Did you create the URL from a pattern? It's not simple to me, unless > copy-pasted it from somewhere :) > > If the URL is constructed from a pattern using nightly dates, then no. That is exactly what I want =( > But I think mshal is working to index nightlies by date, there is certainly > a bug for that. Could you give me bug number? thanks! > > If you're asking if you can get a URL for a b2g-desktop download, then the > answer is yes. > But at the moment I think the URL patterns available employ revision or just > "latest", which updates > often... During developing, we don't want to fetch latest b2g-desktop every time but just have a fixed version which is able to give us a stable run-time to build Gaia. When I saw revision on https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.revision/gecko.v1.mozilla-central.revision, it seems not easy to figure out what's b2g version with these unreadable hash numbers. So I prefer to put b2g-desktop linux back to ftp server temporary for a while until task-cluster has completed support for hosting all platform builds. (only removing linux build on ftp is so weird for me).
Flags: needinfo?(jopsen)
(In reply to Ricky Chien [:rickychien] from comment #46) > (In reply to Jonas Finnemann Jensen (:jonasfj) from comment #45) > > But I think mshal is working to index nightlies by date, there is certainly > > a bug for that. > > Could you give me bug number? thanks! It's bug 1133074, and I'm actively working on it.
credits to mshal for address my NI :) > So I prefer to put b2g-desktop linux back to ftp server temporary for a while until task-cluster has > completed support for hosting all platform builds. I think we're building b2g-desktop in taskcluster and not buildbot, which is why it's not present on FTP. Adding it FTP would very hard, I think it's easier to move forward and indexing it right, mshal has buildbot uploading to taskcluster and soon it'll be indexed with by-date too. > (only removing linux build on ftp is so weird for me). Can't argue that. The idea was to have buildbot uploading to taskcluster, and then move everything out of buildbot one task/job at the time. But I guess we need the indexing done right to make people happy with FTP going away.
Flags: needinfo?(jopsen)
Depends on: 1133074
I'm going to fix it until bug 1133074 is landed. Workaround for this case would create an another link for linux build b2g and upgrade links of other platform once taskcluster support all platforms indexing.
Priority: -- → P1
Status: NEW → ASSIGNED
No longer blocks: b2g-emulator-x86-KK
(In reply to Ricky Chien [:rickychien] from comment #49) > I'm going to fix it until bug 1133074 is landed. Workaround for this case > would create an another link for linux build b2g and upgrade links of other > platform once taskcluster support all platforms indexing. Hi Ricky, The patches in bug 1133074 should be landed, can you continue to fix this bug now?
Flags: needinfo?(rchien)
[Blocking Requested - why for this release]: This impacts Emulator KK and many codes in 2.5 cannot be tested.
blocking-b2g: --- → 2.5?
The only way to fix it quickly would be falling back to old version b2g to build Gaia (we just need its xpcshell runtime during build time, so I guess it shouldn't break anything). It seems that b2g-36.0a1.multi.linux-x86_64 has built-in ssltunnel, so I'll switch to it if there are no objections.
Flags: needinfo?(rchien)
Flags: needinfo?(gandalf)
Flags: needinfo?(felash)
What's the path forward with this then? Or do you think we should be able to stick to b2g36 for a forseable future?
I forget the b2g 36 has problem in macos64 package so that we cannot just fallback to old one to solve this. A workaround came out my mind is that open another download url for linux but we should stick to a particular version (taskID is YamDhuDgTWa_kWXcSedDHA). https://queue.taskcluster.net/v1/task/YamDhuDgTWa_kWXcSedDHA/artifacts/public/build/target.linux-x86_64.tar.bz2 I know it's not a pretty solution but it should work.
Flags: needinfo?(gandalf)
Flags: needinfo?(felash)
Comment on attachment 8649154 [details] [review] [gaia] rickychien:b2g43 > mozilla-b2g:master Hi Martin, I think you're the best person to review this part change after git blame. Please take a look for this small patch kindly. thanks!
Attachment #8649154 - Flags: review?(martin.thomson)
I don't think this is a future-proof solution; we need to be able to use more recent xpcshell. Here is my favorite solution: We can have direct links to newer versions of B2G Desktop. And they have ssltunnel from what I checked. [1] always links to latest build for B2G Desktop on Linux 64b. [2] is the current latest build for B2G Desktop on Linux 64b as I write now. I can find [2] from [1] using curl: $ curl -s -I https://index.taskcluster.net/v1/task/buildbot.branches.mozilla-central.linux64_gecko/artifacts/public/build/target.linux-x86_64.tar.bz2 | grep ^Location: Location: https://queue.taskcluster.net/v1/task/YamDhuDgTWa_kWXcSedDHA/artifacts/public/build/target.linux-x86_64.tar.bz2 [1] https://index.taskcluster.net/v1/task/buildbot.branches.mozilla-central.linux64_gecko/artifacts/public/build/target.linux-x86_64.tar.bz2 [2] https://queue.taskcluster.net/v1/task/YamDhuDgTWa_kWXcSedDHA/artifacts/public/build/target.linux-x86_64.tar.bz2 See their API documentation [3] for the index. [3] http://docs.taskcluster.net/services/index/ For the build system I think we should fix a specific build and not fetch whatever latest is, just like we does today (so use the "task" URL and not the "latest" URL). Note that OSX and Windows builds are still on ftp.mozilla.org (that we access using HTTP) but they'll likely move to taskcluster as well. Another idea is we can use mozilla-download to download B2G as it uses taskcluster to find what to download. Historically I think we didn't want to download half of npm repository just to build Gaia. I still think this is a valuable goal, that's why I don't recommend this solution. However some possible solutions to avoid downloading all npm dependencies if we only need "mozilla-download": 1. run "npm install mozilla-download" (instead of "npm install") before running the "b2g" goal to download B2G. 2. Have a separate package.json in build/ so that we could do "npm install" in this directory and download only the npm packages we need for building. Still this means we'll need a node/npm environment to download xpcshell we use to build Gaia. I find this quite sad. Also we can make that node/npm environment less strict (could work with node 0.8/0.10 and npm 1). Another question: because we'll stop building B2G Desktop soon in favor of Mulet, do we know if Mulet builds suit our build system?
Bug 1133074 is finally moving. Should we wait for it?
Since this is a blocker, it make sense to take attachment 8649154 [details] [review] as a temp solution though.
(In reply to Julien Wajsberg [:julienw] from comment #57) > I don't think this is a future-proof solution; we need to be able to use > more recent xpcshell. I was answering comment 52 here, not comment 54; we reached the same solution then :)
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #58) > Bug 1133074 is finally moving. Should we wait for it? From what I see the major patches landed in bug 1133074 already, and I think we can use it right away. But it just gives new routes, I don't think we needed this to move forward anyway as we have existing URLs to the builds anyway, and we don't need the latest build in the build system, just one specific recent build.
(In reply to Julien Wajsberg [:julienw] from comment #57) > I don't think this is a future-proof solution; we need to be able to use > more recent xpcshell. As far as I know, b2g desktop provides a Javascript runtime and XPCOM APIs for gaia build system. However, XPCOM APIs could be replaced by node.js :) Thus, I wonder why we have to upgrade b2g to newer version continuously, in addition to support newer es6 or es7 syntax, I don't see any potential benefits from it.
(In reply to Ricky Chien [:rickychien] from comment #62) > (In reply to Julien Wajsberg [:julienw] from comment #57) > > I don't think this is a future-proof solution; we need to be able to use > > more recent xpcshell. > > As far as I know, b2g desktop provides a Javascript runtime and XPCOM APIs > for gaia build system. However, XPCOM APIs could be replaced by node.js :) > > Thus, I wonder why we have to upgrade b2g to newer version continuously, in > addition to support newer es6 or es7 syntax, I don't see any potential > benefits from it. I know the L10n folks use it for preprocessing -- and so maybe they need also the newer API like Intl ? Not sure, just saying that having the same engine to parse apps and to run apps still could have advantages.
Attachment #8649154 - Flags: review?(martin.thomson) → review+
No longer depends on: 1133074
Comment on attachment 8649154 [details] [review] [gaia] rickychien:b2g43 > mozilla-b2g:master Patch has landed on Gaia main trunk. Need feedback from Blake to verify mochitest part.
Attachment #8649154 - Flags: feedback?(bwu)
That is likely burning the tree starting at https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=d255da3b284c. Can you backout please?
master revert: https://github.com/mozilla-b2g/gaia/commit/c73c5648fb50de2dd23454eaf78ba5a44803eca5 According to IRC, gtk3 has something to do with this... philor> INFO - /home/worker/workspace/B2G/gaia/b2g_sdk/43.0a1-2015-08-17-15-02-06/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory 12:45 PM <•fabrice> philor: awesome 12:46 PM <philor> no nice things for you! 12:46 PM <•fabrice> why the hell is xpcshell depending on gtk...
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #70) > 12:46 PM <•fabrice> why the hell is xpcshell depending on gtk... bug 1189125
Unfortunately, there are things we don't want to happen but it happened. After investigating artifacts on task-cluster, I'm still no idea how to find out a previous artifacts. Jonas, do you know how to get a correct b2g url from task-cluster just like https://queue.taskcluster.net/v1/task/YamDhuDgTWa_kWXcSedDHA/artifacts/public/build/target.linux-x86_64.tar.bz2 but it is built before bug 1189125? I would like to get a newer b2g but don't include gtk features.
see comment 72
Flags: needinfo?(jopsen)
No, I think the proper fix is to have the GTK3 libs on our build machines, not to use an older xpcshell.
(In reply to Ricky Chien [:rickychien] from comment #72) > Unfortunately, there are things we don't want to happen but it happened. > > After investigating artifacts on task-cluster, I'm still no idea how to find > out a previous artifacts. > > Jonas, do you know how to get a correct b2g url from task-cluster just like > > https://queue.taskcluster.net/v1/task/YamDhuDgTWa_kWXcSedDHA/artifacts/ > public/build/target.linux-x86_64.tar.bz2 > > but it is built before bug 1189125? I would like to get a newer b2g but > don't include gtk features. If it's in Gaia, why don't you use mozilla-download ? It can fetch from TaskCluster ...
Flags: needinfo?(rchien)
mozilla-download try to pull latest version b2g from task-cluster, but for Gaia build we just want to stick a stable version.
Flags: needinfo?(rchien)
For other reasons why don't we choose mozilla-download you could see comment 57.
Set this as a 2.5+ bug since we need this to run Emulator KK test cases to prevent regressions.
blocking-b2g: 2.5? → 2.5+
Depends on: 1196595
(In reply to Julien Wajsberg [:julienw] from comment #74) > No, I think the proper fix is to have the GTK3 libs on our build machines, > not to use an older xpcshell. Agree! Let's do it on bug 1196595.
Attachment #8649154 - Flags: feedback?(bwu)
Flags: needinfo?(jopsen)
I've checked and mulet has ssltunnel too. Good !
This isn't a Emulator bug. Change it to Gaia:Build. Feel free to correct if this isn't the best fit.
Component: Emulator → Gaia::Build
Push to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3f61845707a I'm going land it soon once try is green.
Try is green along with some unrelated orange failures. Patch is ready! It will be landed in master once Gaia tree is opened, and backout manually if some other unexpected failures show up again.
Backed out for similar-looking errors as last time. https://treeherder.mozilla.org/logviewer.html#?job_id=2621176&repo=b2g-inbound 02:17:54 INFO - /home/worker/workspace/B2G/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory https://github.com/mozilla-b2g/gaia/commit/024c6be4c6a402841062e78043070f0c4e7eef44
Comment on attachment 8653248 [details] [review] [gaia] rickychien:b2g-43 > mozilla-b2g:master There is an another buildbot machine issue that will be solved in bug 1196595 by a gecko patch. From gaia side, we need to accept an additional LD_LIBRARY_PATH from b2g build script to point to a correct libgtk-3.so. I've updated gaia patch and verified locally to see LD_LIBRARY_PATH which was injected correctly. Tim, would you please take a look at this gaia patch for me? thanks! I'm going to land this once bug 1196595 is done.
Attachment #8653248 - Flags: review?(timdream)
Attachment #8653248 - Flags: review?(timdream) → review+
Still burning. You taking care of the backout?
Flags: needinfo?(rchien)
Ryan, I think Ricky is sleeping now.
We didn't see these failures earlier though, are you sure this patch is the issue ?
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #93) > Still burning. You taking care of the backout? I didn't see any failures which are related to this patch as last time. Issue of missing libgtk3 seems to be resovled.
Flags: needinfo?(rchien) → needinfo?(ryanvm)
Comment 92 was looking at the wrong push. https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=d5d566e3cdf8 https://treeherder.mozilla.org/logviewer.html#?job_id=2665747&repo=b2g-inbound 09:39:52 INFO - /builds/slave/b2g_b2g-in_emu_dep-00000000000/build/gaia/b2g_sdk/43.0a1-2015-08-17-15-02-06/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory 09:39:52 INFO - /builds/slave/b2g_b2g-in_emu_dep-00000000000/build/gaia/b2g_sdk/43.0a1-2015-08-17-15-02-06/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory 09:39:52 ERROR - make[1]: *** [preferences] Error 127 09:39:52 INFO - make[1]: Leaving directory `/builds/slave/b2g_b2g-in_emu_dep-00000000000/build/gaia' 09:39:52 INFO - make: *** [gaia-prefs] Error 2 09:39:52 INFO - make: *** Waiting for unfinished jobs....
Flags: needinfo?(ryanvm)
(In reply to Ricky Chien [:rickychien] from comment #96) > (In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #93) > > Still burning. You taking care of the backout? > > I didn't see any failures which are related to this patch as last time. > Issue of missing libgtk3 seems to be resovled. ok I saw bustage here. https://treeherder.allizom.org/#/jobs?repo=b2g-inbound&revision=d5d566e3cdf8 so it probaly is an unsolved issue in bug 1196595 or LD_LIBRARY_PATH doesn't export correctly to gaia.
According to comment 89, LD_LIBRARY_PATH was exported correctly and verified from B2G repo by invoking `LD_LIBRARY_PATH=xxxx build.sh`. I've no clue about why libgtk-3.so is still failing on treeherder. Mike, is there any missing step when trying to verify LD_LIBRARY_PATH? how can we verify LD_LIBRARY_PATH on machine? Could you give me a hand for that? thanks
Flags: needinfo?(mh+mozilla)
Ask someone who actually knows those jobs. I don't, so I have no more clue why LD_LIBRARY_PATH is not propagating like you would like it to, or if it's some other problem.
Flags: needinfo?(mh+mozilla)
Ryan, do you have any clue for comment 99?
Flags: needinfo?(ryanvm)
I need someone who is very familiar with task-cluster. Greg, do you know what's the problem in this bustage from comment 97?
Flags: needinfo?(garndt)
I know nothing about these jobs.
Flags: needinfo?(ryanvm)
I'm not deeply familiar about the internals of what happens with these jobs but willing to dig in and try to help figure it out. From what I see from this [1] treeherder link is that even the emulator ICS builds on buildbot fail with the same error that has been noted in previous comments. "09:38:41 INFO - /builds/slave/b2g_b2g-in_emu-d_dep-000000000/build/gaia/b2g_sdk/43.0a1-2015-08-17-15-02-06/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory" It's not immediately clear to me where the taskcluster specific bug is. [1] https://treeherder.allizom.org/#/jobs?repo=b2g-inbound&revision=d5d566e3cdf8
Flags: needinfo?(garndt)
We ran into a problem I guess it probably relate to TC because missing of libgtk-3.so on build machine. Bug bug 1196595 has landed that try to pull gtk3 through tooltool to centos buildbot but I tried in vain. We also got TC failures with same libgtk-3.so errors but glandium told me that TC already installed it originally. Just want to know what's LD_LIBRARY_PATH actually on buildbot env? I guess it probably is empty so that it is unalbe to export to gaia correctly.
Flags: needinfo?(garndt)
I assume buildbot has exported the path of libgtk3 ('LD_LIBRARY_PATH' = $topsrcdir/gtk3/usr/local/lib) correctly. However, I saw some logs for example https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/QpKmN4hFRUaGZPb6MFz22w/0/public/logs/live_backing.log I found 'LD_LIBRARY_PATH': '/tools/gcc-4.7.3-0moz1/lib64:/tools/gcc-4.7.3-0moz1/lib' ,and it doesn't include gtk3. I'm not familiar about the buildbot machine environment, don't have any idea about what's difference of these machines behind CI infra. I just hope that someone who is familiar with it could help us setup libgtk-3 on all buildbot machines properly.
I do not believe that we currently use tooltool in our builders for b2g products so if this was updated in tooltool,t he change won't make it to the taskcluster jobs. part of the discussion for including tooltool in the builders is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1198726 I believe for the builders to include packages from tooltool, the docker image for the builder needs to be updated to include the tooltool script and then the build tasks need to add a few features to use this [1] [2] I'm not sure the repercussion of pulling these packages from tooltool in our builders to get these packages since this builder is used for multiple products. Dustin, do you have any ideas or could point me in the right direction? [1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/mulet_linux.yml?offset=200#28 [2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/mulet_linux.yml?offset=200#20
(In reply to Ricky Chien [:rickychien] from comment #106) > I assume buildbot has exported the path of libgtk3 ('LD_LIBRARY_PATH' = > $topsrcdir/gtk3/usr/local/lib) correctly. However, I saw some logs for > example > > https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/ > QpKmN4hFRUaGZPb6MFz22w/0/public/logs/live_backing.log > > I found 'LD_LIBRARY_PATH': > '/tools/gcc-4.7.3-0moz1/lib64:/tools/gcc-4.7.3-0moz1/lib' ,and it doesn't > include gtk3. > > I'm not familiar about the buildbot machine environment, don't have any idea > about what's difference of these machines behind CI infra. I just hope that > someone who is familiar with it could help us setup libgtk-3 on all buildbot > machines properly. So there is a mixture of environments right now for b2g. B2G desktop, emulator l, jb, kk are taskcluster builds. Emulator ICS is on our buildbot infrastructure. Just clarifying since there was a mixture of terms used within this bug. As far as updating buildbot instances, I'm not sure what that takes.
Flags: needinfo?(garndt)
Flags: needinfo?(dustin)
Wow, this bug has had four or five different topics, as I count them. Might I suggest filing a new bug for each new topic? I can't really follow the thread well enough to tell what info you need.. I think the latest question is about gtk3. Gtk3 is installed from tooltool, so there's no change to the host environment to support it. :glandium set that up in bug 1186003.
Flags: needinfo?(dustin)
Feel free to file new bugs for appropriate topic you want. If gtk3 has been installed already, why we still get libgtk-3.so.0 errors after pathc is landed? please see also comment 98. Patch in this bug tries to upgrade b2g-desktop from 39.0a1 to 43.0a1 but newer 43.0a1 depends on gtk3 causing bustage in b2g-inbound. Give you an example log of `B2G JB Emulator opt` from comment 98, https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/swKQmdOgSh-_hp3RczkE_g/0/public/logs/live_backing.log Please search keyword `libgtk-3.so`, you can see what's going on in buildbot. According to comment 108, I'm still seeing both buildbot and task-cluster are bustage with libgtk-3.so.0
Flags: needinfo?(dustin)
Flags: needinfo?(garndt)
I don't see any invocation of tooltool in the log in comment 110. Since Gtk3 is installed from tooltool, it follows that Gtk3 is not available for these builds. Which explains errors trying to find libgtk-3.so.0. I don't know anything about the B2G build process, so I don't know how you would go about installing Gtk3. Bug 1186003 might help?
(In reply to Dustin J. Mitchell [:dustin] from comment #111) > Gtk3 is installed from tooltool, it follows that Gtk3 is not available for > these builds. I don't really understand what you mean here :( Although gtk3 has been available in CI machines, I know releng has to setup/export these library paths in buildbot script or test-suite script (ex: mozharness settings?) to execute a job correctly. Gtk3 should be installed in https://bugzilla.mozilla.org/show_bug.cgi?id=1196595#c60, but I'm not sure whether it has been set up comprehensively without patching any part of buildbot script or test-suite script.
I'm going to file an another bug since according to comment 0, mochitest shouldn't depend on gaia b2g_sdk just in order to access ssltunnel. Gaia depends on b2g_sdk's JS run-time to build gaia. In the future, gaia b2g_sdk will be migrated to node.js and no longer install during build time.
Depends on: 1202537
Discussed offline, here are something to try (based on bug 1196595 comment 48). We need the three things below to be true to make this work: 1. Verify treeherder VMs has gtk3 installed (either in the build image itself or with tooltools) 2. Verify buildbot VMs has gtk3 installed 3. Verify B2G/Gaia build scripts are indeed carrying the path variable over when invoking xpcshell Ricky can merge a small patch with |echo| to verify (3), without actually bumping the xpcshell version. For (2), logs after bug 1196595 merged indicates tooltools is in effect. Not sure about (1). We thought it is already a solved issue (when discussed and commented in bug 1196595 comment 48). It is possible to actually |ls| or get the image to verify the presence of the library?
According to comment 114, patch for |echo| to verify (3) Landed https://github.com/mozilla-b2g/gaia/commit/9305742318f201bec7ec84b18d745c11bb699d36 And see treeherder logs should export correct LD_LIBRARY_PATH
Base on https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=e440a0a7c4bb, try to find `[DEBUG_LOG]` in log. We can see: B2G Desktop Linux x64 (opt/debug) a.k.a taskcluster outputs `LD_LIBRARY_PATH=/home/worker/workspace/gecko/gtk3/usr/local/lib` which is pointing to a correct gtk3 path. B2G ICS Emulator (opt/debug) a.k.a buildbot (centos) outputs empty path `LD_LIBRARY_PATH=`. B2G KK Emulator (opt/debug) a.k.a taskcluster outputs `LD_LIBRARY_PATH=/tools/gcc-4.7.3-0moz1/lib64:/tools/gcc-4.7.3-0moz1/lib` which doesn't include gtk3. B2G L Emulator (opt/debug) a.k.a taskcluster outputs `LD_LIBRARY_PATH=/tools/gcc-4.7.3-0moz1/lib64:/tools/gcc-4.7.3-0moz1/lib` which doesn't include gtk3. It's very clear to clarify that case (3) in comment 114 is valid and there are some different LD_LIBRARY_PATH results so I think that issue will need the support of releng or automation.
Update our to-be-fixed things here: (a) Ensure tooltool downloads gtk3 on buildbot (fixed in bug 1196595) (b) Ensure tooltool downloads gtk3 on treeherder (c) Ensure path is set when |./build.sh| (the B2G build system) is called (d) Ensure path is taken from B2G build system to Gaia build system (fixed in comment 115) For (b), Ricky found that https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-b2g-desktop.sh eventually invokes tooltool while https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-emulator.sh does not. For (c), (In reply to Ricky Chien [:rickychien] from comment #118) > B2G Desktop Linux x64 (opt/debug) a.k.a taskcluster outputs > `LD_LIBRARY_PATH=/home/worker/workspace/gecko/gtk3/usr/local/lib` which is > pointing to a correct gtk3 path. > Upon investigation we realized the path was picked up by Gecko build system itself in https://dxr.mozilla.org/mozilla-central/source/build/unix/mozconfig.gtk#23 (from https://dxr.mozilla.org/mozilla-central/source/build/unix/mozconfig.linux and from https://dxr.mozilla.org/mozilla-central/source/b2g/config/mozconfigs/linux64_gecko/nightly) We already know Gaia build is invoked by B2G build system, not Gecko build system in Emulator builds, so we can't copy the same approach for fixing (c). We will need a new approach here...
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #119) > We already know Gaia build is invoked by B2G build system, not Gecko build > system in Emulator builds, so we can't copy the same approach for fixing > (c). We will need a new approach here... Here is a hack: How about ifneq (,$(TOOLTOOL_DIR)) LD_LIBRARY_PATH="$TOOLTOOL_DIR/gtk3/usr/local/lib:$(LD_LIBRARY_PATH)" endif in gaia/Android.mk ? That would fix the problem for now I guess??
Sounds like you folks got it figured out. Note for other readers, since it took me a while to figure out: in comments 114-119, "treeherder" should be "taskcluster".
Flags: needinfo?(dustin)
If anyone needs assistance getting anything setup in taskcluster's docker images or build scripts, I can help out and point people in the right direction. Reading the comments here definitely helped me understand things a bit more about our builds.
Flags: needinfo?(garndt)
(In reply to Dustin J. Mitchell [:dustin] from comment #121) > Sounds like you folks got it figured out. Note for other readers, since it > took me a while to figure out: in comments 114-119, "treeherder" should be > "taskcluster". Ouch. Thanks Dustin. :garndt, it would be good if you could help out (b) in comment 119 in our p.m. hours so we could go ahead attempt to sort out (c) tomorrow. :mwu, I would like to know your opinion on (c), particularly what's discussed in comment 119 and comment 120; what's the best place to resolve gtk3 path from tooltool? Or, could you discuss this with Gaia build system or automation forks to ensure we (B2G+Gaia) could receive the path from them instead? Thanks.
Flags: needinfo?(mwu)
Flags: needinfo?(garndt)
I suspect :glandium is actually the best person to answer that -- as you've noticed, the Gecko build system automatically finds Gtk3 if it's unpacked in the working directory.
Flags: needinfo?(mh+mozilla)
ok, I'm going to ni? wcosta on this. I spoke with him on IRC and he might have some bandwidth to look into this. Updating any taskcluster bits should probably be taken into a separate bug so we can work on that implementation without adding to the comments here.
Flags: needinfo?(garndt) → needinfo?(wcosta)
I don't know too much about how things are setup on the build machines, though the current hacks seem a bit weird to me. Why is gtk3 not installed normally (in /usr/lib or /usr/local/lib rather than in $(TOOLTOOL_DIR)) like the other dependencies? I guess adding the hack suggested in comment 120 is ok too.
Flags: needinfo?(mwu)
comment 120 is fine.
Flags: needinfo?(mh+mozilla)
I'm waiting for b2g-inbound bumper and verify LD_LIBRARY_PATH again as same as comment 118.
Try to find `[DEBUG_LOG]: TOOLTOOL_DIR=` is empty, that means comment 120 doesn't help...
Ricky, my comment 120 is just an illustration on how can we hack our way to find the gtk3 library based on environment variable. There is no environment variable named TOOLTOOL_DIR.... Even in Gecko build it is constructed too. https://dxr.mozilla.org/mozilla-central/source/build/unix/mozconfig.gtk#1-8 Please back out your patches and actually look into the code before trying out your fixes. Flagging for review might be helpful too.
Flags: needinfo?(rchien)
Look into https://dxr.mozilla.org/mozilla-central/source/build/unix/mozconfig.gtk#1-8, I guess LD_LIBRARY_PATH=$TOOLTOOL_DIR/gtk3/usr/local/lib" would be empty since tooltool.py doesn't invoke / install from build-emulator.sh properly. See https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-emulator.sh We should invoke install-packages.sh to launch tooltool.py, see also https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/install-packages.sh Push to try but it failed... https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8e57e51ab52
Flags: needinfo?(rchien)
Please fix the install-packages.sh issue ("b" in comment 119) in another bug as comment 125 addressed. We would still need to fix (c) here -- could you verify comment 137 fixes (c) according to buildbot output? We are getting there, thanks!
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #139) > Please fix the install-packages.sh issue ("b" in comment 119) in another bug > as comment 125 addressed. We would still need to fix (c) here -- could you > verify comment 137 fixes (c) according to buildbot output? > > We are getting there, thanks! Hi, I am working on this.
Flags: needinfo?(wcosta)
Depends on: 1203144
After investigating with John Dai, we found that set LD_LIBRARY_PATH relate to a Dockerfile at https://dxr.mozilla.org/mozilla-central/source/testing/docker/b2g-build/Dockerfile?case=true&from=b2g-build%2FDockerfile#104 I attempted in vain to submit a gecko patch to try with override LD_LIBRARY_PATH, so it means TC will pull docker image from https://quay.io/repository/mozilla/b2g-build. John has already verified it with his docker image (https://hub.docker.com/r/jdai/b2g-build/). See also try log at https://treeherder.allizom.org/#/jobs?repo=try&revision=5ed7b3826bfc And LD_LIBRARY_PATH was override with empty. Thus, to fix gtk3 path in TC would modify https://dxr.mozilla.org/mozilla-central/source/testing/docker/b2g-build/Dockerfile?case=true&from=b2g-build%2FDockerfile#104 to append a correct gtk3 path '/home/worker/workspace/gecko/gtk3/usr/local/lib'. After then update docker image to https://quay.io/repository/mozilla/b2g-build. Dustin, I'm not sure about above steps are correctly, I hope you could help us to do it in a separate bug since you are the Dockerfile maintainer =)
Flags: needinfo?(dustin)
Next step we have to do is to figure out how to set up LD_LIBRARY_PATH for ICS emulator (buildbot machine).
ICS emulator doesn't run on taskcluster yet. AFAIK, buildbot environment bases on mozharness (if I'm wrong please correct me). Take an example of http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-emulator/1441864233/b2g_b2g-inbound_emulator_dep-bm74-build1-build163.txt.gz We can see output from above log: File gtk3.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/b2g_b2g-in_emu_dep-00000000000/build/tmpLyBiSW gtk3 has already installed and untared successfully and tmpLyBiSW was renamed to gtk3.tar.xz The gtk3 path for this machine would be LD_LIBRARY_PATH=/builds/slave/b2g_b2g-in_emu_dep-00000000000/build/gtk3 We want to set up it in an appropriate place in mozharness. Greg, do you know who can help with this? thanks!
Flags: needinfo?(garndt)
Depends on: 1203460
(In reply to Ricky Chien [:rickychien] from comment #141) > After investigating with John Dai, we found that set LD_LIBRARY_PATH relate > to a Dockerfile at > https://dxr.mozilla.org/mozilla-central/source/testing/docker/b2g-build/ > Dockerfile?case=true&from=b2g-build%2FDockerfile#104 > > I attempted in vain to submit a gecko patch to try with override > LD_LIBRARY_PATH, so it means TC will pull docker image from > https://quay.io/repository/mozilla/b2g-build. > > John has already verified it with his docker image > (https://hub.docker.com/r/jdai/b2g-build/). See also try log at > https://treeherder.allizom.org/#/jobs?repo=try&revision=5ed7b3826bfc And > LD_LIBRARY_PATH was override with empty. > > Thus, to fix gtk3 path in TC would modify > https://dxr.mozilla.org/mozilla-central/source/testing/docker/b2g-build/ > Dockerfile?case=true&from=b2g-build%2FDockerfile#104 to append a correct > gtk3 path '/home/worker/workspace/gecko/gtk3/usr/local/lib'. After then > update docker image to https://quay.io/repository/mozilla/b2g-build. > > Dustin, I'm not sure about above steps are correctly, I hope you could help > us to do it in a separate bug since you are the Dockerfile maintainer =) I am working on this bug. Please see bug 1203460.
Flags: needinfo?(dustin)
Depends on: 1203513
(In reply to Ricky Chien [:rickychien] from comment #143) > ICS emulator doesn't run on taskcluster yet. AFAIK, buildbot environment > bases on mozharness (if I'm wrong please correct me). > > Take an example of > http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound- > emulator/1441864233/b2g_b2g-inbound_emulator_dep-bm74-build1-build163.txt.gz > > We can see output from above log: > > File gtk3.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as > /builds/slave/b2g_b2g-in_emu_dep-00000000000/build/tmpLyBiSW > > gtk3 has already installed and untared successfully and tmpLyBiSW was > renamed to gtk3.tar.xz > > The gtk3 path for this machine would be > > LD_LIBRARY_PATH=/builds/slave/b2g_b2g-in_emu_dep-00000000000/build/gtk3 > > We want to set up it in an appropriate place in mozharness. > > Greg, do you know who can help with this? thanks! I am working on this bug. Please see bug 1203513.
Flags: needinfo?(garndt)
cross fingers too...
Thanks for the effort John and Ricky! (In reply to Dustin J. Mitchell [:dustin] from comment #124) > I suspect :glandium is actually the best person to answer that -- as you've > noticed, the Gecko build system automatically finds Gtk3 if it's unpacked in > the working directory. While I appreciate the efforts, I found it hard to believe that build machines expects build systems to find the path of gtk3 in the custom installed locations. What if we need to change the locations in the future?
(In reply to Ricky Chien [:rickychien] from comment #147) > Landed in master: > > https://github.com/mozilla-b2g/gaia/commit/ > 47f0ac6cf4c4525c342d052168153dd56b0039a7 > > cross fingers https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=6402dc30a9ce https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/BnJdYib4T9ebxmg0vIfcUQ/0/public/logs/live_backing.log It burns :'( we need to back this out...
Flags: needinfo?(rchien)
Flags: needinfo?(rchien)
Depends on: 1204800
Depends on: 1204822
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #149) > Thanks for the effort John and Ricky! > > (In reply to Dustin J. Mitchell [:dustin] from comment #124) > > I suspect :glandium is actually the best person to answer that -- as you've > > noticed, the Gecko build system automatically finds Gtk3 if it's unpacked in > > the working directory. > > While I appreciate the efforts, I found it hard to believe that build > machines expects build systems to find the path of gtk3 in the custom > installed locations. What if we need to change the locations in the future? Actually I think it's a fine expectation as long as LD_LIBRARY_PATH is correctly set (as it is here).
(In reply to Julien Wajsberg [:julienw] from comment #153) > Actually I think it's a fine expectation as long as LD_LIBRARY_PATH is > correctly set (as it is here). It is not, as you can see, LD_LIBRARY_PATH is found and set by build systems (B2G & Gecko) and not present in the shell environment.
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #154) > (In reply to Julien Wajsberg [:julienw] from comment #153) > > Actually I think it's a fine expectation as long as LD_LIBRARY_PATH is > > correctly set (as it is here). > > It is not, as you can see, LD_LIBRARY_PATH is found and set by build systems > (B2G & Gecko) and not present in the shell environment. See the patch in https://bug1203460.bmoattachments.org/attachment.cgi?id=8659169 for example: LD_LIBRARY_PATH is set in the environment. Then our build system simply forwards it in [1]. [1] https://github.com/mozilla-b2g/gaia/blob/d130529c626695570c7b5017d9b43955115d68a0/Makefile#L343.
Ever since this landed and was backed out, Raptor automation has been unable to recover. While there were compilation/build errors while this was in tree, once it was backed out it has persistently failed when trying to install the reference workload: make reference-workload-light test_media/reference-workload/makeReferenceWorkload.sh light Waiting for device to be connected... Device connected Populate Databases - light Workload Can't find indexedDB base dir I'm assuming this will go away once I can reflash the base image, but just wanted to give a heads up.
See Also: → 1205118
Hi Ricky, the GijTV test for this seems always fails. Could you please have a look? Thanks!
Flags: needinfo?(rchien)
This patch was backed out in comment 151. It shouldn't affect to GijTV since gaia patch targets on upgrading b2g version.
Flags: needinfo?(rchien)
See Also: 1205118 →
(In reply to Julien Wajsberg [:julienw] from comment #155) > (In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to > queue) from comment #154) > > (In reply to Julien Wajsberg [:julienw] from comment #153) > > > Actually I think it's a fine expectation as long as LD_LIBRARY_PATH is > > > correctly set (as it is here). > > > > It is not, as you can see, LD_LIBRARY_PATH is found and set by build systems > > (B2G & Gecko) and not present in the shell environment. > > See the patch in > https://bug1203460.bmoattachments.org/attachment.cgi?id=8659169 for example: > LD_LIBRARY_PATH is set in the environment. > > Then our build system simply forwards it in [1]. > > [1] > https://github.com/mozilla-b2g/gaia/blob/ > d130529c626695570c7b5017d9b43955115d68a0/Makefile#L343. OK! But actually that's more troubling because that means TC VMs and Buildbot VMs are different in terms on path is set (and un- or little- documented), and the next change would probably takes another 100 comments to be figured out.
Blocks: 1207131
Workaround for anybody else who needs to run ./mach mochitest on b2g emulator, on a linux machine at least: 1) Go to https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.b2g-desktop/gecko.v1.mozilla-central.latest.linux.b2g-desktop.opt (link taken from comment 33) and download public/build/target.linux-x86_64.tar.bz2 2) Rename the $B2G_DIR/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g folder to b2g.bk or something to get it out of the way 3) Unpack the above tarball in $B2G_DIR/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/. This will create a new b2g folder which has ssltunnel 4) Run mochitest command and enjoy!
Summary: [emulator] mach mochitest-remote fails: expected to find ssltunnel at .../gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel → [emulator] mach mochitest fails: expected to find ssltunnel at .../gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02/b2g/ssltunnel
Whiteboard: [workaround in comment 160]
Hey Alexandre Lissy, I think you're the best person to know about mach_b2g_bootstrap.py and here is my patch dealing with downloading b2g_sdk by itself when running emulator mochitest. It's the best solution due to mochitest shouldn't depend on b2g from gaia , and leading gaia build-time crash with gtk3 if we go to upgrade b2g on gaia. Please take a look at patch kindly and I don't know how to verify it on try server. I would appreciate that if you could help me test it. thanks!
Attachment #8668845 - Flags: review?(lissyx+mozillians)
Comment on attachment 8668845 [details] [review] B2G patch for downloading b2g_sdk by itself I am sorry but I have never ever read nor hacked that file, and I don't know who should be reviewing this.
Attachment #8668845 - Flags: review?(lissyx+mozillians)
Patch of B2G does: 1. Download b2g_sdk from ftp (windows / mac) or taskcluster (linux) if b2g.tar doesn't exist 2. Extract b2g.tar if b2g.tar doesn't unpacked I'm not sure where is an appropriate place for b2g_sdk, so I put it on B2G/b2g_sdk currently. Mochitest will try to access gaia's b2g_sdk locally so this patch makes sense to fix this issue, but it's different on CI machines. John Dai will continue the patch for CI machines to redirect xre_path = downloaded_b2g_sdk once patch is landed.
Comment on attachment 8668845 [details] [review] B2G patch for downloading b2g_sdk by itself git blame ahal also modified that file. ahal, could you review that patch for me? thanks!
Attachment #8668845 - Flags: review?(ahalberstadt)
Comment on attachment 8668845 [details] [review] B2G patch for downloading b2g_sdk by itself Please see comments on github. I'm also not convinced this is the right place to do this download, but I don't know of any better places.
Attachment #8668845 - Flags: review?(ahalberstadt) → review-
Comment on attachment 8668845 [details] [review] B2G patch for downloading b2g_sdk by itself Patch updated! Please see it again.
Attachment #8668845 - Flags: review- → review?(ahalberstadt)
Thanks, looks good but I think the patch as is will fail. Have you tried it out? See comment on github.
Done! mozinstall & requests are moved into download function. Both previous and current patch are passed on my machine.
Flags: needinfo?(ahalberstadt)
Comment on attachment 8668845 [details] [review] B2G patch for downloading b2g_sdk by itself Thanks, looks good!
Flags: needinfo?(ahalberstadt)
Attachment #8668845 - Flags: review?(ahalberstadt) → review+
Thank you Andrew! Landed in master at https://github.com/mozilla-b2g/B2G/commit/3a9f9810063e3be891226dd80868361440618b20 I'll keep watching tree status until all green.
According to issue description, this bug focuses on dealing with local running issue for mochitest. There is an another bug for addressing CI issues at bug 1192121. I think it's okay to close this bug!
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Reopen that due to mozinstall not found when running mochitest locally. I forgot I've installed mozinstall on my environment so it's different with usual environment. Andrew, do you know how to get mozinstall & requests to work?
Status: RESOLVED → REOPENED
Flags: needinfo?(ahalberstadt)
Resolution: FIXED → ---
I do a quick fix for previous patch (added python/mozinstall) so that SEARCH_PATHS can find the correct mozinstall path. Landed in master: https://github.com/mozilla-b2g/B2G/commit/858bee24e901f0c53f78bc47977786cc4117cc96
Flags: needinfo?(ahalberstadt)
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Remember to update B2G repo by git pull and enjoy.
You probably don't want to check in the mozinstall.egg-info folder, that's a setuptools artifact. The reason mozinstall wasn't being detected is because it's missing from the gecko bootstrap: https://dxr.mozilla.org/mozilla-central/source/build/mach_bootstrap.py#101 I'll fix that, and once that's added you should be able to remove python/mozinstall again.
Sorry for that, patch is ready once bug 1212840 is landed.
Attachment #8671389 - Flags: review?(ahalberstadt)
Comment on attachment 8671389 [details] [review] follow up for removing mozinstall No worries, thanks for doing this! I landed bug 1212840, so we just need to wait for it to get merged to mozilla-central. Note, people will also need to do a ./repo sync to pull in the latest gecko changes.
Attachment #8671389 - Flags: review?(ahalberstadt) → review+
Follow-up for removing Makefile's [DEBUG_LOG]: LD_LIBRARY_PATH https://github.com/mozilla-b2g/gaia/commit/df7683e4f7329c567deb487d11317578c6930246
Ricky, I think attachment 8661104 [details] [review] is still not landed ? or do you have another bug to update b2g_sdk in Gaia Makefile ?
Flags: needinfo?(rchien)
To upgrade b2g to latest version will depend on gtk3 which requires all CI machines to install gtk3 (see also bug 1204800). If you would like to do this, we can file another bug with attachment 8661104 [details] [review] to upgrade b2g. However, I prefer to move gaia build to node.js and get rid of b2g_sdk from build-time.
Flags: needinfo?(rchien)
Hey Ricky, unfortunately we're currently using b2g_sdk to know the known CSS property in the CSS linter. So I don't think we'll move to node.js really soon, at least for this. And not having latest gecko is impairing us because then using the newest properties yields errors. So I think we should move forward upgrading b2g_sdk. We're nearly there !
Flags: needinfo?(rchien)
Yeah, I can understand we will continue to upgrade b2g_sdk tentatively until node.js migration is complete. Bug of upgrade to latest b2g_sdk will be filed soon.
Flags: needinfo?(rchien)
No longer depends on: 1204800
No longer blocks: 1207131
I think this ( https://github.com/mozilla-b2g/gaia/commit/e544b68c4c5b7b9b9bc7af71db02ebe81a6b5cc1 ) broke device builds: https://treeherder.mozilla.org/logviewer.html#?job_id=3157363&repo=b2g-inbound 16:19:44 INFO - test -f /home/worker/workspace/B2G/gaia/b2g_sdk/44.0a1-2015-10-01-03-02-29/b2g/xpcshell 16:19:44 INFO - host C: libcrypto_static <= external/openssl/crypto/evp/pmeth_lib.c 16:19:44 INFO - /home/worker/workspace/B2G/gaia/b2g_sdk/44.0a1-2015-10-01-03-02-29/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory 16:19:44 INFO - /home/worker/workspace/B2G/gaia/b2g_sdk/44.0a1-2015-10-01-03-02-29/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory 16:19:44 ERROR - make[1]: *** [preferences] Error 127 16:19:44 INFO - make[1]: Leaving directory `/home/worker/workspace/B2G/gaia' 16:19:44 INFO - make: *** [gaia-prefs] Error 2 Reverted in https://github.com/mozilla-b2g/gaia/commit/81e52c867e86c116a096f892bcb3d78f33fb702c to see if it fixes things.
Flags: needinfo?(rchien)
Wes, bug 1215437 is the bug you're looking for, sorry the commit log was wrong.
Flags: needinfo?(wkocher)
Ah, thanks!
Flags: needinfo?(wkocher)
Flags: needinfo?(rchien)
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: