1087088 - `mach buildsymbols` hangs while gzip asks if I want to overwrite libclearkey.so.dbg.gz

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I'm not sure whose bug this is, but I'll start with Video/Audio because this seems to happen only to libclearkey and nothing else in the build.

STR: run "./mach buildsymbols" after doing a desktop build

At some point, gzip writes to the terminal as follows:

gzip: dist/crashreporter-symbols/libclearkey.so/FBD67515E0566373B3B48BC83385393C0/libclearkey.so.dbg.gz already exists; do you wish to overwrite (y or n)?

This gets buried under hundreds of lines of other output, because a lot of stuff is going on in parallel, so it's very much not obvious that gzip is sitting there waiting patiently for me to type something and blocking the build until that happens.

This also reproduced after deleting dist/crashreporter-symbols.

Comment 1

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1087088-don-t-connect-stdin-to-buildsymbols.patch

I hit this too - looks like fallout from bug 1044742. ted, should libclearkey.so be installed via an INSTALL_TARGET in the Makefile as well as listed as a SharedLibrary in moz.build? I'd guess the INSTALL_TARGET should go, but then the lib won't be available in dist/bin/gmp-clearkey/0.1. The problem is that buildsymbols tries to build both versions, so the second one waits for the user prompt:

16435: Submitting jobs for files: ('dist/bin/libclearkey.so',)
16435: Submitting jobs for files: ('dist/bin/gmp-clearkey/0.1/libclearkey.so',)

Attached is a patch that just disconnects stdin from buildsymbols, which just works around the problem by behaving more like it would in automation. We probably also want to fix the double-library situation.

Comment 2

[(not currently active) Ted Mielczarek]

I assume something is looking for it in that location in dist/bin, so if we stop putting it there we'll need to fix whatever that is. I'm not sure if we can ask the build system to build it there directly instead of putting it in dist/bin and then copying it. glandium: is that a thing we could do?

Comment 3

[Mike Hommey [:glandium]]

Putting SHARED_LIBRARY in CLEARKEY_CDM_FILES in
https://hg.mozilla.org/mozilla-central/diff/3855948c6f30/media/gmp-clearkey/0.1/Makefile.in
causes the file to be copied to dist/bin/gmp-clearkey/0.1.

But SHARED_LIBRARY is also in SHARED_LIBRARY_FILES in
https://hg.mozilla.org/mozilla-central/file/3855948c6f30/config/makefiles/target_binaries.mk#l44
which causes the file to be copied to dist/bin.

Note the destination for the latter is $(FINAL_TARGET).

The right thing to do is to set FINAL_TARGET to 'dist/bin/gmp-clearkey/0.1' in media/gmp-clearkey/0.1/moz.build.

Comment 4

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1087088-Don-t-install-libclearkey-to-dist-bin.patch

I guess we can leave buildsymbols as is for now to help catch any future occurrences.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8514302 [details] [diff] [review]
0001-Bug-1087088-Don-t-install-libclearkey-to-dist-bin.patch

Review of attachment 8514302 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks! Ideally we'd fail with a clearer error here, maybe we ought to take your other patch anyway?

Comment 6

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1087088-buildsymbols-should-fail-if-there-are-du.patch

How's this? The error message can be buried back in the log, but at least searching for "Error" will help reveal the problem.

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8514335 [details] [diff] [review]
0001-Bug-1087088-buildsymbols-should-fail-if-there-are-du.patch

Hmm, actually this fails on OSX because libjsctypes-test.dylib is installed into two locations as well. I'm not quite sure why the same thing doesn't fail on Linux, since it is installed twice there as well. Only OSX processes them in symbolstore.py, though.

Plus it made my try push timeout rather than just fail, which is odd. It didn't do that locally.

Comment 8

[Michael Shal [:mshal]]

inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/3b7bdfbb0988

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3b7bdfbb0988