1225488 - Fix build process to automatically put openh264 pdb file into FF symbol server to help debug crash

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Maire Reavy [:mreavy]]

+++ This bug was initially created as a clone of Bug #1223926 +++

Ted is doing this for Rui and me in bug 1223926 as a one-off, but we really want to do this automatically as part of the build process.  This bug is meant to cover the automation work needed to do that.

Comment 1

[Justin Wood (:Callek)]

ted, our scripts to do openh264 builds are in https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/openh264_build.py

Admittedly I did some manual editing my last run of the builds I have more info about the process in Bug 1223925

Much of the automation of openH264 build process is actually in https://github.com/cisco/openh264 however I'm keen to assist in making sure the scripts/build-process/etc does whatever is necessary for symbols.

it seems like Cisco cares most about Windows symbols, but if we're doing it I'd love us to get Android (arm/x86) as well as linux and OSX symbols (note we build a 32 bit openh264 and a 64 bit openh264 for osx)

any help you can give is appreciated.

Maire, is this a requirement in any sense for 1.5.2? (If so, that may push back our timeline next week until after US thanksgiving)

Comment 2

[Maire Reavy [:mreavy]]

Fixing this bug is not a requirement for v1.5.2.  I would like to get the pdb file into v1.5.2 as "a one-off" which I think is pretty quick/easy to do.  (That's what Ted has offered to do in Bug 1223926.)

Comment 3

[Hank Peng]

Justin, it would be much appreciated if the plugin symbols can be uploaded and resolved on all the supported platforms. I guess the reason Rui pointed out the PDB file in particular was that we found the build configuration of openh264 was wrong to generate the PDB file. It has been fixed in the master branch, referring to commit https://github.com/cisco/openh264/commit/b5792a09f9c80c0f0bf3a6faacd8883c086817e6.

Comment 4

[Justin Wood (:Callek)]

(In reply to Hank Peng from comment #3)
> Justin, it would be much appreciated if the plugin symbols can be uploaded
> and resolved on all the supported platforms. I guess the reason Rui pointed
> out the PDB file in particular was that we found the build configuration of
> openh264 was wrong to generate the PDB file. It has been fixed in the master
> branch, referring to commit
> https://github.com/cisco/openh264/commit/
> b5792a09f9c80c0f0bf3a6faacd8883c086817e6.

I just sent e-mail asking on if this is a useful thing to uplift to the 1.5.2 branch, in anticipation of our next build.

:ted, what file(s) or commands would you need run for the android, mac, and linux builds of openh264 to get the relevant symbols for Socorro. E.g. how can I help get the most out of the 1.5.2 build I'll be doing next week. (even if the symbol upload is manual)

Comment 5

[(not currently active) Ted Mielczarek]

We're essentially going to need to add a buildsymbols/uploadsymbols step akin to what our Firefox builds do. I'm poking at the openh264 build right now to see if I can give you an easy command to run.

Are you doing these builds manually, or in buildbot or taskcluster?

Comment 6

[(not currently active) Ted Mielczarek]

Ok, I poked at the openh264 build and this shouldn't be too bad. I submitted a PR to make life slightly easier:
https://github.com/cisco/openh264/pull/2251

With that patch, the packagesymbols.py script from https://gist.github.com/luser/2ad32d290f224782fcfc , and a local dump_syms binary, I can run:
```
packagesymbols.py /path/to/dump_syms `make -s echo-plugin-name`
```

or on Windows:
```
packagesymbols.py /path/to/dump_syms_vc1800.exe `make -s echo-plugin-name OS=msvc`
```

I didn't test on OS X yet, I'll sanity check that.

That produces symbols.zip containing what we want. You can upload that symbols.zip using either the in-tree upload_symbols.py:
https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py

or the uploadsymbols.py from that gist. The in-tree one is somewhat more reliable for automation usage. If you're doing builds in Taskcluster, you could alternately just copy the symbols.zip to `artifacts/target.crashreporter-symbols.zip` and list that directory as public/build in your artifacts section and then use the upload_symbols task as a dependent task, like:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/file/tip/task.json#l51

Otherwise you'll need a valid Socorro auth token (the RelEng builders should all have one in place already for uploading Firefox symbols).

One thing I noticed is that the non-Windows builds don't build with debug symbols, so we only get export symbols. I'll open an issue on GitHub for that.

Comment 7

[(not currently active) Ted Mielczarek]

Opened https://github.com/cisco/openh264/issues/2252 for getting debug symbols on non-Windows.

Comment 8

[Hank Peng]

Hi Ted, thanks for your PR2251 which we have merged into the master branch and would like to pick it into the v1.5.2-Firefox39 branch. 
Also thanks for raising the issue https://github.com/cisco/openh264/issues/2252. However, we don't have anyone in the OpenH264 who is familiar with the build configuration to resolve this issue. Could you please submit another PR to help fix it? Thanks.

Comment 9

[(not currently active) Ted Mielczarek]

I can fix it, it shouldn't be very complicated.

Comment 10

[Hank Peng]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> I can fix it, it shouldn't be very complicated.

Much appreciated. Looking forward to your fix.

Comment 11

[Hank Peng]

There are 3 commits in v1.5.2-Firefox39 branch trying to generate the debug symbols on non-windows platforms:
https://github.com/cisco/openh264/commit/48002c809193e31e8db365ae1336e08ce4f08bfa
https://github.com/cisco/openh264/commit/ac9ca8f879d8226a53ff69e9053ba93ee58772be
https://github.com/cisco/openh264/commit/778e87520e656410d102f642317365ee684cd270

When running the command 'make DEBUGSYMBOLS=True plugin', two libraries will be generated. libgmpopenh264.so(dylib) is the one containing symbols, and libgmpopenh264_stripped.so(dylib) is the one without debug symbols. We are not sure it this could meet the requirement. 

Ted, hope you could help review the commits and update the your build script. Thanks.

Comment 12

[Hank Peng]

The android hang fix has been put into v1.5.2. I'd like to have the v1.5.2 released soon. If the symbol generation is OK, the v1.5.2 could include it as well. Otherwise, will we release v1.5.2 first, Maire?

Comment 13

[Maire Reavy [:mreavy]]

Thanks, Hank.  I believe there's time to get the symbol generation in.  We're all at Orlando this week and next week is a Firefox release week.  (It's a bad idea to release the browser and the plugin during the same week -- unless it's critical to do so.)

I'm talking with Justin (Callek) and Randell (Jesup) now to come up with a reasonable timeline for testing and release of v1.5.3.

Comment 14

[Justin Wood (:Callek)]

(In reply to Hank Peng from comment #11)
> There are 3 commits in v1.5.2-Firefox39 branch trying to generate the debug
> symbols on non-windows platforms:
> https://github.com/cisco/openh264/commit/
> 48002c809193e31e8db365ae1336e08ce4f08bfa
> https://github.com/cisco/openh264/commit/
> ac9ca8f879d8226a53ff69e9053ba93ee58772be
> https://github.com/cisco/openh264/commit/
> 778e87520e656410d102f642317365ee684cd270
> 
> When running the command 'make DEBUGSYMBOLS=True plugin', two libraries will
> be generated. libgmpopenh264.so(dylib) is the one containing symbols, and
> libgmpopenh264_stripped.so(dylib) is the one without debug symbols. We are
> not sure it this could meet the requirement. 
> 
> Ted, hope you could help review the commits and update the your build
> script. Thanks.

ted, while you have the n-i still open, can you confirm if I needed to do this new `make` magic for non-windows to get the symbols, and if so if I should have shipped the _stripped as if it was the s/_stripped// version to Cisco.  And if the lack of me doing this invalidates the attempt on my end to get non-windows symbols into our symbol server?

Comment 15

[(not currently active) Ted Mielczarek]

It's...a little weird. The make command I gave you should work with that new stuff. However! You'll have to ship the _stripped version in this case otherwise the binaries will be much larger than they need to be.

Comment 16

[Haibo Zhu]

I am very sorry that we had a mistake here. As the openh264 v1.5.3 has been released, We will add a note in our backlog or you can give a pull request to openh264 master branch. So it will be OK in next release.

Comment 17

[Justin Wood (:Callek)]

To be clear...

<Callek>	ted: your answer confused me more in Bug 1225488 ...
<ted>	Callek: sorry!
<ted>	Callek: the make command will work fine for non-windows, and produce useful symbols
<ted>	Callek: however! the way they implemented that is weird
<ted>	and means that you'd have to ship the _stripped version
<Callek>	ted: rephrasing attempt: so the way we did the symbols in the existing case was fine.... that is the `packagesymbols.py .. <package_name>` but if we use THEIR new way, we have to ship the _stripped version because their stuff will be much larger?
<ted>	yes


So while this could use some fixing up, 1.5.3 is fine.

Comment 18

[Haibo Zhu]

Hi, we have changed the strange debug symbols make option in openh264 master and latest v1.6 branch. Now when make the openh264 with "BUILDTYPE=Release DEBUGSYMBOLS=True", it will output libopenh264.a, libopenh264.so, libopenh264_debug_symbols.a and libopenh264_debug_symbols.so together. Please modify your scripts if you want to use this make option.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

While working on bug 1380605 (openh264 v1.7.1) our scripts generated symbol archives which contained gmpopenh264.pdb on windows, libgmpopenh264.dylib on mac, and libgmpopenh264.so on linux/android (as well as .sym files for all).

Comment 20

[Aki Sasaki (not active)]

Backlog cleanup.