Closed Bug 532925 Opened 10 years ago Closed 10 years ago

provide windows mobile nightly symbols through symbol server

Categories

(Release Engineering :: General, defect, P3)

ARM
Windows Mobile 6 Professional
defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: blassey, Assigned: aki)

References

Details

(Keywords: mobile)

Attachments

(3 files, 1 obsolete file)

This would make getting crash reports more useful from nightly testers.

Also, we're currently seeing crashes in our nightlies which we don't see in our personal builds, which this would help track down.
Is there anything on the symbol-server side that needs to be done here? 

Does "make buildsymbols" do the right thing currently for WinMo before releng tries to add it (and the corresponding upload) for WinMo builds?
Priority: -- → P3
I suspect "make buildsymbols" will not work at all because we don't have Breakpad support for WinMo yet. It's probably possible to do this manually using symstore.exe, but that creates a whole separate code path.

(The symbol server itself is just a directory being served by Apache.)
what is the product of make buildsymbols? when I run it here I get "nothing to be done for 'buildsymbols'"
(In reply to comment #3)
> what is the product of make buildsymbols? when I run it here I get "nothing to
> be done for 'buildsymbols'"

uh... that's basically because the whole thing is ifdef'd for crashreporter being enabled.
This sounds like either a Fennec bug or a Build:Config bug to me. Perhaps both.
Component: Release Engineering → Windows Mobile
Product: mozilla.org → Fennec
QA Contact: release → mobile-windows
Version: other → Trunk
Attached patch patch to use symstore.exe (obsolete) — Splinter Review
Attachment #421278 - Flags: review?(ted.mielczarek)
I should not that this patch requires symstore.exe to be in the PATH.  Chris, is this a reasonable assumption for our nightly builders?
(In reply to comment #7)
> I should not that this patch requires symstore.exe to be in the PATH.  Chris,
> is this a reasonable assumption for our nightly builders?

As Ted noted in IRC, the Debugging Tools for Windows dir is not in the PATH by default. We currently use a PDBSTR_PATH (full path to pdbstr.exe), and could presumably use SYMBOLSTORE_PATH or similar.
Brad: can you update your patch to be inside an ifdef SYMBOLSTORE_PATH, and use $(SYMBOLSTORE_PATH) as the exe name? Then we'll just have to get the buildbot configs updated, shouldn't be a big deal.
Attachment #421278 - Attachment is obsolete: true
Attachment #421619 - Flags: review?(ted.mielczarek)
Attachment #421278 - Flags: review?(ted.mielczarek)
Comment on attachment 421619 [details] [diff] [review]
patch v.2 [checked in]

Ok, this seems fine as a stopgap until we get crash reporting working on WinCE.

You'll need to file a followup or move this back to RelEng to get SYMBOLSTORE_PATH set in the buildbot configs.
Attachment #421619 - Flags: review?(ted.mielczarek) → review+
Comment on attachment 421619 [details] [diff] [review]
patch v.2 [checked in]

pushed http://hg.mozilla.org/mozilla-central/rev/8bd09155ca14
Attachment #421619 - Attachment filename: buildsymbols.patch → buildsymbols.patch [checked in]
Attachment #421619 - Attachment description: patch v.2 → patch v.2 [checked in]
back over to RelEng
Component: Windows Mobile → Release Engineering
Product: Fennec → mozilla.org
QA Contact: mobile-windows → release
Version: Trunk → other
RelEng will need to:
1) Add SYMBOLSTORE_PATH to the buildbot environment
2) Ensure that "make buildsymbols" gets run before uploading WinCE/WinMo builds
(In reply to comment #14)
> RelEng will need to:
> 1) Add SYMBOLSTORE_PATH to the buildbot environment
> 2) Ensure that "make buildsymbols" gets run before uploading WinCE/WinMo builds

Also, in order to produce symbols the nightly mozconfig's should be updated to include:

export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debugger-info-modules=yes
(In reply to comment #15)
> export MOZ_DEBUG_SYMBOLS=1
> ac_add_options --enable-debugger-info-modules=yes

Only MOZ_DEBUG_SYMBOLS should be necessary.
I'm on buildduty this week, I'll try to have a look at this.
Assignee: nobody → bhearsum
According to the mozconfig, we need to fix bug 514295 before we turn on MOZ_DEBUG_SYMBOLS - is that still true?
its been worked around already, but not fixed. The work around allows us to build symbols for everything but nss.
(In reply to comment #14)
> RelEng will need to:
> 1) Add SYMBOLSTORE_PATH to the buildbot environment
> 2) Ensure that "make buildsymbols" gets run before uploading WinCE/WinMo builds

To be clear, this bug is about what releng calls "winmo" builds. It's more than just a flip of a switch to turn on - we still need to add symbol support into the windows mobile build factory, which I don't have time to do. Aki/jhford - either of you have time?
Assignee: bhearsum → nobody
Assignee: nobody → aki
I've got a patch, testing bug 550319 with this.
1.9.2 has nothing for 'make buildsymbols' and trunk died on a mozalloc (blassey might be working on this?)  Will try again later.
You can nominate this patch for 1.9.2 if it'd make your life easier.
also, the mozalloc issue has be resolved now
Attachment #421619 - Flags: approval1.9.2.2?
Hm, I'm getting an upload_to_stage exception on 1.9.2:

Traceback (most recent call last):
  File "/tools/buildbot-clean/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/buildstep.py", line 689, in startStep
    d.addCallback(self._startStep_2)
  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 194, in addCallback
    callbackKeywords=kw)
  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 185, in addCallbacks
    self._runCallbacks()
  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
  File "/tools/buildbot-clean/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/buildstep.py", line 712, in _startStep_2
    skip = self.start()
  File "/tools/buildbotcustom/buildbotcustom/steps/transfer.py", line 287, in start
    datedDir = self.getLongDatedPath()
  File "/tools/buildbotcustom/buildbotcustom/steps/transfer.py", line 213, in getLongDatedPath
    buildid = self.getBuildID()
  File "/tools/buildbotcustom/buildbotcustom/steps/transfer.py", line 188, in getBuildID
    return strftime("%Y-%m-%d-%H", strptime(buildid[0:10], "%Y%m%d%H"))
  File "/tools/python-2.5.2/lib/python2.5/_strptime.py", line 330, in strptime
    (data_string, format))
exceptions.ValueError: time data did not match format:  data=Section [A  fmt=%Y%m%d%H

I need to track that down.

The m-c build failed since I built during tree breakage; trying again.
I'm going to chalk that one up to production keys on the system (just VNC'ed in and did an openssl sha1 on the key).

We should really fail more gracefully in MozillaStageUpload. However, this should work in 1.9.2.  Still waiting for the trunk build to get to the upload step.
Also, is there a reason why production keys aren't in the staging-stage .ssh/authorized_keys ? =) I mean, if we can't trust production slaves to upload to a staging env ...

Putting this in a bug comment where it will be buried and forgotten because I'm awesome that way.
Filed bug 551621 and bug 551622 for comment 25 and comment 26, respectively.
Ran this in staging. Gacked on upload but that's with the wrong keys; the objdir/dist/*.zip should get this.

Coop: note that objdir/mobile/... and objdir/xulrunner/... become objdir/... for winmo builds with this patch (which includes fixes from bug 550319).

This doesn't actually upload the symbols to the symbol server, but it's a good first step.
Attachment #431813 - Flags: review?(ccooper)
This is the same patch as in bug 550319, but with MOZ_DEBUG_SYMBOLS, and with the -QRarch6 lines commented out.
Attachment #431814 - Flags: review?(bugmail)
aki: with the new Fennec build (no more xulrunner), you should be able to use "make upload" just like the Firefox build to upload. You should also be able to use "make uploadsymbols" to upload to the symbol server.
Comment on attachment 431814 [details] [diff] [review]
mozconfigs for symbols

as noted on irc, nspr has been updated so let's get the -QArch6 in there
Attachment #431814 - Flags: review?(bugmail) → review-
Comment on attachment 431813 [details] [diff] [review]
build+upload winmo symbols

Removing review -- I will attempt make upload* and I need to double check I'm not borking Try.
Attachment #431813 - Flags: review?(ccooper)
I also need to make sure the project branches' mozconfigs are correct, as long as they re-pull from the appropriate main branch (1.9.2 or m-c).
Comment on attachment 421619 [details] [diff] [review]
patch v.2 [checked in]

There's a lot of churn here after the request point, so minusing for now.

Please renominate when you've got a single patch that does everything you want on the branch!
Attachment #421619 - Flags: approval1.9.2.2? → approval1.9.2.2-
afaict, the churn is on the releng side of things and doesn't effect the request for branch landing.
Blocks: 554087
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.