698702 - Symbol packaging problem in Socorro symbol store

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Since the Thunderbird 10.1 nightly with buildID 20111026030412 we've been getting messages from the script that cleans up the symbol store for Socorro (once a day). The most recent message is:

Missing symbols found.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111026030412-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111026030412-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111026030412-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111026030412-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111027050625-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111027050625-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111027050625-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111027050625-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111028030017-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111028030017-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111028030017-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111028030017-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111031030018-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111031030018-symbols.txt is not readable.
Symbol thunderbird.sym from thunderbird-10.0a1-Darwin-20111031030018-symbols.txt is not readable.
Symbol thunderbird.dSYM.tar.bz2 from thunderbird-10.0a1-Darwin-20111031030018-symbols.txt is not readable.

Looking at the first symbol manifest (thunderbird-10.0a1-Darwin-20111026030412-symbols.txt) it has
thunderbird/273358BB1F13F8B5FFAA975AE07B1A850/thunderbird.sym
thunderbird/273358BB1F13F8B5FFAA975AE07B1A850/thunderbird.dSYM.tar.bz2
thunderbird/F832A0E8DC387EE719A15F6862B3A5390/thunderbird.sym
thunderbird/F832A0E8DC387EE719A15F6862B3A5390/thunderbird.dSYM.tar.bz2
thunderbird-bin/273358BB1F13F8B5FFAA975AE07B1A850/thunderbird-bin.sym
thunderbird-bin/273358BB1F13F8B5FFAA975AE07B1A850/thunderbird-bin.dSYM.tar.bz2
thunderbird-bin/F832A0E8DC387EE719A15F6862B3A5390/thunderbird-bin.sym
thunderbird-bin/F832A0E8DC387EE719A15F6862B3A5390/thunderbird-bin.dSYM.tar.bz2

And indeed none of the thunderbird/ files exist, but the thunderbird-bin ones do. Seems likely that
 http://hg.mozilla.org/comm-central/rev/7582d11356ff
is the cause. Perhaps there is some symbol/packaging code that means we don't hit this for Firefox, but do for Thunderbird ?

Comment 1

[Mark Banner (:standard8)]

From some analysis with Nick on irc, we've found that:

- Thunderbird has another executable called "Thunderbird" in Daily.app/Contents/Library/Spotlight/Thunderbird.mdimporter/Contents/MacOS/Thunderbird
- So we end up with 4 Thunderbird sets of info e.g.

Thunderbird/42D6501A33E9507901230AD7DF21D7C90/Thunderbird.sym
Thunderbird/42D6501A33E9507901230AD7DF21D7C90/Thunderbird.tar.bz2
Thunderbird/2297A4201C7DFF6283ED06B930762FFB0/Thunderbird.sym
Thunderbird/2297A4201C7DFF6283ED06B930762FFB0/Thunderbird.tar.bz2
thunderbird/62DC749649D1FB483A812E99176590B60/thunderbird.sym
thunderbird/62DC749649D1FB483A812E99176590B60/thunderbird.dSYM.tar.bz2
thunderbird/7471D5DEE533A83E2FCFA2797103950F0/thunderbird.sym
thunderbird/7471D5DEE533A83E2FCFA2797103950F0/thunderbird.dSYM.tar.bz2

However, looking at the files on the symbol server:

ls: thunderbird/62DC749649D1FB483A812E99176590B60/thunderbird.sym: No such file or directory
ls: thunderbird/62DC749649D1FB483A812E99176590B60/thunderbird.dSYM.tar.bz2: No such file or directory
ls: thunderbird/7471D5DEE533A83E2FCFA2797103950F0/thunderbird.sym: No such file or directory
ls: thunderbird/7471D5DEE533A83E2FCFA2797103950F0/thunderbird.dSYM.tar.bz2: No such file or directory
-rw-rw-r-- 1 tbirdbld users  1365 Oct 31 16:28 Thunderbird/2297A4201C7DFF6283ED06B930762FFB0/Thunderbird.sym
-rw-rw-r-- 1 tbirdbld users  5278 Oct 31 16:28 Thunderbird/2297A4201C7DFF6283ED06B930762FFB0/Thunderbird.tar.bz2
-rw-rw-r-- 1 tbirdbld users    61 Oct 31 16:28 Thunderbird/42D6501A33E9507901230AD7DF21D7C90/Thunderbird.sym
-rw-rw-r-- 1 tbirdbld users  5278 Oct 31 16:28 Thunderbird/42D6501A33E9507901230AD7DF21D7C90/Thunderbird.tar.bz2

Looking at the logs for the buildsymbols step, something is ignoring the case of Thunderbird versus thunderbird.

I need to dig around in the build to see if it is the buildsymbols script or the zip step that is creating the issue.

Comment 2

[Mark Banner (:standard8)]

I've just done a 64-bit only build, but using the same type of configuration as nightly. When doing make buildsymbols the Contents/Library/Spotlight/Thunderbird.mdimporter/Contents/MacOS/Thunderbird didn't get a symbols file created. That's fine because we wouldn't get crashes for that file, but it does make me wonder why we then get them on a universal build.

I'll look some more...

Comment 3

[Mark Banner (:standard8)]

That was a red herring. A non-universal build does buildsymbols for dist/bin which doesn't include the extra Library files.

Comment 4

[Mark Banner (:standard8)]

Attachment: The fix

Ok, this fixes the issue. In its wonderful wisdom, Mac sees:

ls Thunderbird/14CC9E5B3D44A315FAEF1E7E17493C8C0/thunderbird.sym
ls thunderbird/14CC9E5B3D44A315FAEF1E7E17493C8C0/thunderbird.sym

as the same. We can get around this by lowercasing the file name when it is used to build the directory structure, and this lower-case name will also get printed to the symbols.txt file.

As we should basically never get two executables with the same guid, then this should work for us.

In some ways I think we should probably rename one of the executables, but the only one would be the mdimporter, and I'm not quite sure if that would work properly if we changed that.

Comment 5

[(not currently active) Ted Mielczarek]

I guess I don't quite understand. You have two executables named "thunderbird", but one is capitalized. Are they the same binary, or different?

Comment 6

[Mark Banner (:standard8)]

(In reply to Ted Mielczarek [:ted, :luser] from comment #5)
> I guess I don't quite understand. You have two executables named
> "thunderbird", but one is capitalized. Are they the same binary, or
> different?

Different. One is the thunderbird application executable, the other is in Thunderbird.mdimporter which is used for spotlight integration.

Comment 7

[Mark Banner (:standard8)]

(note, we only added the application executable 'thunderbird' recently, in bug 668869, up until then we just had thunderbird-bin, we didn't even had a script called thunderbird).

Comment 8

[(not currently active) Ted Mielczarek]

Comment on attachment 575178 [details] [diff] [review]
The fix

Review of attachment 575178 [details] [diff] [review]:
-----------------------------------------------------------------

Mark clarified some things on IRC which alleviated my confusion.

Comment 9

[(not currently active) Ted Mielczarek]

Comment on attachment 575178 [details] [diff] [review]
The fix

Actually, right after submitting that I realized that we can't do this. This will break symbols for things with mixed-case filenames (AccessibleMarshal.dll, for example).

The symbol lookup on the server will use the same filename case as what's on the build machine, so it has to match.

Comment 10

[Mark Banner (:standard8)]

Attachment: The fix v2

Ok, I took a look at changing which directory was scanned on mac, but I came to the conclusion that it would probably me we don't get symbols for things in tests that we might want (or that it would break something else, I can't quite remember what now).

So the simpler fix seems to be to try and rename the Thunderbird executable in the spotlight importer to 'thunderbird'.

I haven't fully tested this yet, I'm going to push to try first.

Comment 11

[Mark Banner (:standard8)]

Attachment: Make the mdimporter executable lower-case

Ok, I came to the conclusion the previous approaches just weren't going to work correctly - they'd break something.

Therefore I've gone with renaming the executable in the mdimporter from Thunderbird to thunderbird. I think this should work right - I'm a little concerned about updates, but I'll try and remember to look after we've landed this on trunk.

Comment 12

[Mark Banner (:standard8)]

Oh, and there's a try server build here:

https://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-2fa4fba23bd3/try-comm-central-macosx64/

and I tested it locally and the whole spotlight index, search, view route seemed to work fine.

Comment 13

[David :Bienvenu]

Comment on attachment 583427 [details] [diff] [review]
Make the mdimporter executable lower-case

yeah, this looks reasonable.

Comment 14

[Mark Banner (:standard8)]

As I suspected, removing the "Thunderbird" executable on update from removed-files.in actually broke updates. Therefore I'm going to back that part of the patch out. I've just tested with lower-case in Info.plist and first-caps for the executable, and it seems to work the same, so lets try that ;-)

Comment 15

[Mark Banner (:standard8)]

Comment on attachment 583427 [details] [diff] [review]
Make the mdimporter executable lower-case

a=me with the removed-files.in backout.

Comment 16

[Mark Banner (:standard8)]

This was the original checkin:

http://hg.mozilla.org/comm-central/rev/07d03fadd4ea

I've not backed out the removed-files.in change from trunk yet.

Aurora and Beta landings without removed-files.in changes:

http://hg.mozilla.org/releases/comm-aurora/rev/229f9bf65df3
http://hg.mozilla.org/releases/comm-beta/rev/de6a0d6ada02

Comment 17

[Mark Banner (:standard8)]

Backed out the removed-files.in part from trunk:

http://hg.mozilla.org/comm-central/rev/05cbde022f0b