1215084 - [mozcrash] "HTTP Error 404: Not Found" on downloading symbols

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Emulator, defect)

Description

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Snippet from try log https://treeherder.allizom.org/logviewer.html#?job_id=12501318&repo=try

===================
 08:32:46     INFO -  MEMORY STAT | vsize 208MB | residentFast 55MB | heapAllocated 18MB

 08:32:48     INFO -  ############ ErrorPage.js

 08:32:48     INFO -  JavaScript error: chrome://specialpowers/content/SpecialPowersObserver.js, line 110: NS_ERROR_NOT_INITIALIZED: Component returned failure code: 0xc1f30001 (NS_ERROR_NOT_INITIALIZED) [nsIMessageSender.sendAsyncMessage]

 08:32:49     INFO -  [Parent 1614] WARNING: waitpid failed pid:1714 errno:10: file /home/worker/workspace/gecko/ipc/chromium/src/base/process_util_posix.cc, line 267

 08:32:49     INFO -  -*- SettingsManager: Received: inner-window-destroyed for valid innerWindowID=2, cleanup.

 08:32:49     INFO -  1444120364627	Marionette	INFO	Closed connection conn1

 08:32:49     INFO -  AlarmService: observe(): profile-change-teardown

 08:32:49     INFO -  AlarmService: uninit()

 08:32:50     INFO -  JavaScript error: chrome://specialpowers/content/SpecialPowersObserver.js, line 110: NS_ERROR_NOT_INITIALIZED: Component returned failure code: 0xc1f30001 (NS_ERROR_NOT_INITIALIZED) [nsIMessageSender.sendAsyncMessage]

 08:32:51     INFO -  [Parent 1614] WARNING: waitpid failed pid:1624 errno:10: file /home/worker/workspace/gecko/ipc/chromium/src/base/process_util_posix.cc, line 267

 08:32:51     INFO -  JavaScript error: chrome://specialpowers/content/SpecialPowersObserver.js, line 154: NS_ERROR_ILLEGAL_VALUE: Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIObserverService.removeObserver]

 08:32:51     INFO -  -*- TCPPresentationServer.js: TCPPresentationServer - close

 08:32:51     INFO -  JavaScript error: jar:file:///system/b2g/omni.ja!/components/NfcContentHelper.js, line 257: TypeError: setting a property that has only a getter

 08:32:52     INFO -  JavaScript error: resource://gre/modules/IndexedDBHelper.jsm, line 121: UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code.

 08:32:54     INFO -  Segmentation fault

 08:33:00     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/VP27_spKTpyTFMui46sD-w/artifacts/public/build/emulator.crashreporter-symbols.zip

 08:33:00     INFO -  Traceback (most recent call last):

 08:33:00     INFO -    File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozrunner/base/runner.py", line 200, in check_for_crashes

 08:33:00     INFO -      test=test_name)

 08:33:00     INFO -    File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 123, in log_crashes

 08:33:00     INFO -      stackwalk_binary=stackwalk_binary):

 08:33:00     INFO -    File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 202, in __iter__

 08:33:00     INFO -      rv = self._process_dump_file(path, extra)

 08:33:00     INFO -    File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 224, in _process_dump_file

 08:33:00     INFO -      self._get_symbols()

 08:33:00     INFO -    File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 171, in _get_symbols

 08:33:00     INFO -      data = urllib2.urlopen(self.symbols_path)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen

 08:33:00     INFO -      return _opener.open(url, data, timeout)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 406, in open

 08:33:00     INFO -      response = meth(req, response)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 519, in http_response

 08:33:00     INFO -      'http', request, response, code, msg, hdrs)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 444, in error

 08:33:00     INFO -      return self._call_chain(*args)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain

 08:33:00     INFO -      result = func(*args)

 08:33:00     INFO -    File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default

 08:33:00     INFO -      raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

 08:33:01     INFO -  HTTPError: HTTP Error 404: Not Found

Comment 1

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

I saw same error on xpcshell as well [1]. I check the log, it turns out mozcrash use a wrong url to download the symbols from taskcluster. The file name should be b2g.crashreporter-symbols.zip [2].

> 09:25:55 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/Q9XXlVEDRfa3_wyFw9ConQ/artifacts/public/build/emulator.crashreporter-symbols.zip 

[1] https://treeherder.mozilla.org/logviewer.html#?job_id=14606336&repo=try
[2] https://tools.taskcluster.net/task-inspector/#Q9XXlVEDRfa3_wyFw9ConQ/0

Comment 2

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

We have defined the symbol location in task definition in taskcluster [1], but seems not pass the symbols_url to mozharness [2]. And TestingMixin only read the symbols_url from buildbot configuration [3], looks like it needs to add --symbols_url option in order to get symbols_url passed from taskcluster. 

[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/b2g_emulator_base.yml?from=b2g_emulator_base.yml#36
[2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/b2g_emulator_xpcshell.yml#15-23
[3] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#255-290

Comment 3

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Attachment: WIP, Patch, v1

Comment 4

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

All mochiest tests become orange with having correct symbols_url [1]. And I saw a crash on mochiest shutdown (probably same as bug 1208418).

--
12:43:12  WARNING -  PROCESS-CRASH | dom/apps/tests/test_bug_795164.html | application crashed [None]
12:43:12     INFO -  Crash dump filename: /tmp/tmpwzDNfy/28802bb4-1503-e040-3061d4e4-3be2fa96.dmp
12:43:12     INFO -  MINIDUMP_STACKWALK not set, can't process dump.
--

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=54b9e7d32c60&exclusion_profile=false&group_state=expanded&selectedJob=14648070

Comment 5

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

(In reply to Edgar Chen [:edgar][:echen] from comment #4)
> All mochiest tests become orange with having correct symbols_url [1]. And I
> saw a crash on mochiest shutdown (probably same as bug 1208418).
> 
> --
> 12:43:12  WARNING -  PROCESS-CRASH | dom/apps/tests/test_bug_795164.html |
> application crashed [None]
> 12:43:12     INFO -  Crash dump filename:
> /tmp/tmpwzDNfy/28802bb4-1503-e040-3061d4e4-3be2fa96.dmp
> 12:43:12     INFO -  MINIDUMP_STACKWALK not set, can't process dump.

Let me try to make minidump works to see if we can get more useful information.

> --
> 
> [1]
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=54b9e7d32c60&exclusion_profile=false&group_state=expan
> ded&selectedJob=14648070

Comment 6

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Attachment: WIP, Patch, v2

Looks like we have to update docker image to make minidump work.

Comment 7

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Attachment: MozReview Request: Bug 1215084 - Part 1: Add symbols-url option to emulator tests for taskcluster

Review commit: https://reviewboard.mozilla.org/r/29039/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/29039/

Comment 8

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Comment on attachment 8701844 [details]
MozReview Request: Bug 1215084 - Part 1: Add symbols-url option to emulator tests for taskcluster

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/29039/diff/1-2/

Comment 9

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Attachment: MozReview Request: Bug 1215084 - Part 2: Pass correct symbols url to emulator tests

Review commit: https://reviewboard.mozilla.org/r/29365/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/29365/

Comment 10

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Attachment: MozReview Request: Bug 1215084 - Part 3: Set MINIDUMP_STACKWALK to tester image

Review commit: https://reviewboard.mozilla.org/r/29367/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/29367/

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 8701844 [details]
MozReview Request: Bug 1215084 - Part 1: Add symbols-url option to emulator tests for taskcluster

https://reviewboard.mozilla.org/r/29039/#review26163

It would be nice if we could fix https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#159 to be taskcluster aware. We'll need to get rid of buildbot specific stuff like this in mozharness anyway.

But this patch looks reasonable either way, so feel free to land it. Would be good to at least file a follow-up bug though. Thanks for also digging into this!

Comment 12

[Wander Lairson Costa]

Comment on attachment 8703482 [details]
MozReview Request: Bug 1215084 - Part 3: Set MINIDUMP_STACKWALK to tester image

https://reviewboard.mozilla.org/r/29367/#review26349

Comment 13

[Wander Lairson Costa]

Comment on attachment 8703481 [details]
MozReview Request: Bug 1215084 - Part 2: Pass correct symbols url to emulator tests

https://reviewboard.mozilla.org/r/29365/#review26351

Comment 14

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Hi Wander, I don't have permission to push image to taskcluster, could you help me to update the docker image or do you know what process I need to do to get the permission? Thank you.

Comment 15

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

(In reply to Andrew Halberstadt [:ahal] from comment #11)
> Comment on attachment 8701844 [details]
> MozReview Request: Bug 1215084 - Part 1: Add symbols-url option to emulator
> tests for taskcluster
> 
> https://reviewboard.mozilla.org/r/29039/#review26163
> 
> It would be nice if we could fix
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/
> mozilla/testing/testbase.py#159 to be taskcluster aware. We'll need to get
> rid of buildbot specific stuff like this in mozharness anyway.
> 
> But this patch looks reasonable either way, so feel free to land it. Would
> be good to at least file a follow-up bug though. Thanks for also digging
> into this!

File bug 1237182 as a follow-up bug.

Comment 16

[Wander Lairson Costa]

(In reply to Edgar Chen [:edgar][:echen] from comment #14)
> Hi Wander, I don't have permission to push image to taskcluster, could you
> help me to update the docker image or do you know what process I need to do
> to get the permission? Thank you.

Do you have an account in docker hub?

Comment 17

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

(In reply to Wander Lairson Costa [:wcosta] from comment #16)
> (In reply to Edgar Chen [:edgar][:echen] from comment #14)
> > Hi Wander, I don't have permission to push image to taskcluster, could you
> > help me to update the docker image or do you know what process I need to do
> > to get the permission? Thank you.
> 
> Do you have an account in docker hub?

Yes, I have. My account is edgarchen, https://hub.docker.com/u/edgarchen/.

Comment 18

[Wander Lairson Costa]

(In reply to Edgar Chen [:edgar][:echen] from comment #17)
> (In reply to Wander Lairson Costa [:wcosta] from comment #16)
> > (In reply to Edgar Chen [:edgar][:echen] from comment #14)
> > > Hi Wander, I don't have permission to push image to taskcluster, could you
> > > help me to update the docker image or do you know what process I need to do
> > > to get the permission? Thank you.
> > 
> > Do you have an account in docker hub?
> 
> Yes, I have. My account is edgarchen, https://hub.docker.com/u/edgarchen/.

I added you to the firefoxosdev group, please check if you can push the image now

Comment 19

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

(In reply to Wander Lairson Costa [:wcosta] from comment #18)
> I added you to the firefoxosdev group, please check if you can push the
> image now

I can push the image now. Thank you, Wander.

Comment 20

[Edgar Chen [:edgar] (Parental leave until Oct. 09.)]

Try result: https://treeherder.allizom.org/#/jobs?repo=try&revision=4d73e2b4387e&group_state=expanded&exclusion_profile=false

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/4d4178e60cb9
https://hg.mozilla.org/integration/b2g-inbound/rev/9cba1315b62e
https://hg.mozilla.org/integration/b2g-inbound/rev/22b61066991a

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4d4178e60cb9
https://hg.mozilla.org/mozilla-central/rev/9cba1315b62e
https://hg.mozilla.org/mozilla-central/rev/22b61066991a