750637 - No XUL symbols in 12.0 on Linux

Status: RESOLVED WORKSFORME

Description

[Scoobidiver (away)]

There are no XUL symbols in 12.0 on Linux. See https://crash-stats.mozilla.com/topcrasher/byos/Firefox/12.0/Linux/7/browser

Comment 1

[Ben Hearsum (:bhearsum)]

I know we used to have symbols for many distributions...Chris, do you know anything about this? It doesn't seem like it would be a problem on the breakpad server.

Comment 2

[Chris Coulson]

Yes, this is fallout from bug 688186. I updated our automation to push to the new location but realized at the end of last week that the symbols were failing to upload, because the machine that we upload the symbols from can't access the new server.

We do have an internal ticket for resolving that. I will try and find out what is going on

Comment 3

[Ted Mielczarek [:ted.mielczarek]]

Are these actually crashes from Ubuntu builds, or from Mozilla builds?

Comment 4

[Ben Hearsum (:bhearsum)]

(In reply to Ted Mielczarek [:ted] from comment #3)
> Are these actually crashes from Ubuntu builds, or from Mozilla builds?

Not sure about the one in comment #0, but bug 750723 is a Fedora build.

Comment 5

[Chris Coulson]

Ok, our symbol uploads are working again now

Comment 6

[Chris Coulson]

The symbols have been on the server for about 14 hours now, but the latest reports still aren't working. I'm not sure if there is something else wrong too, or if I just need to wait a bit longer

eg, https://crash-stats.mozilla.com/report/index/4f65d7d6-45b6-4276-9c9c-a44f32120502

[[e-mail]1 ~]$ ls -l /mnt/netapp/breakpad/symbols_ubuntu/libxul.so/B8936B88086282FCB8836007BB9E99CF0
total 88288
-rw-r--r-- 1 chrisccoulson chrisccoulson 90220356 Apr 23 08:22 libxul.so.sym

Comment 7

[Ben Hearsum (:bhearsum)]

(In reply to Chris Coulson from comment #6)
> The symbols have been on the server for about 14 hours now, but the latest
> reports still aren't working. I'm not sure if there is something else wrong
> too, or if I just need to wait a bit longer
> 
> eg,
> https://crash-stats.mozilla.com/report/index/4f65d7d6-45b6-4276-9c9c-
> a44f32120502
> 
> [[e-mail]1 ~]$ ls -l
> /mnt/netapp/breakpad/symbols_ubuntu/libxul.so/
> B8936B88086282FCB8836007BB9E99CF0
> total 88288
> -rw-r--r-- 1 chrisccoulson chrisccoulson 90220356 Apr 23 08:22 libxul.so.sym

cc'ing some folks who might know

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> (In reply to Chris Coulson from comment #6)
> > The symbols have been on the server for about 14 hours now, but the latest
> > reports still aren't working. I'm not sure if there is something else wrong
> > too, or if I just need to wait a bit longer
> > 
> > eg,
> > https://crash-stats.mozilla.com/report/index/4f65d7d6-45b6-4276-9c9c-
> > a44f32120502
> > 
> > [[e-mail]1 ~]$ ls -l
> > /mnt/netapp/breakpad/symbols_ubuntu/libxul.so/
> > B8936B88086282FCB8836007BB9E99CF0
> > total 88288
> > -rw-r--r-- 1 chrisccoulson chrisccoulson 90220356 Apr 23 08:22 libxul.so.sym
> 
> cc'ing some folks who might know

I just checked on one of the processors, looks like the dirs in the NFS mount /mnt/socorro/symbols/symbols_ubuntu/libxul.so/ starting May 1st are not readable anymore:

drwxr-xr-x    2 1404 users   4096 Apr 16 06:25 7B27B91E3C336E9888E9F68C7597893D0
drwxr-x---    2 1404  1404   4096 May  1 15:11 6D872AB6C0B0C3FF2AA8B653F87817B00

Comment 9

[Chris Coulson]

Oh, you're right. The directories for all of the symbols I uploaded yesterday have the wrong permissions. I never had to change the permission of files I uploaded to the old server though.

Comment 10

[Chris Coulson]

Oh, I see that upload_symbols.sh actually changes the umask when unpacking the zip file, which is not something that we've ever done when uploading our own symbols. The default umask when I log in is 0027. I wonder if that's different to the old server?

Comment 11

[Ben Hearsum (:bhearsum)]

(In reply to Chris Coulson from comment #10)
> Oh, I see that upload_symbols.sh actually changes the umask when unpacking
> the zip file, which is not something that we've ever done when uploading our
> own symbols. The default umask when I log in is 0027. I wonder if that's
> different to the old server?

That seems likely, I don't think we can verify it at this point, though! Either way, I think this part might be an IT issue?

Comment 12

[Chris Coulson]

I've fixed the permissions and made sure that the next set of symbols we push will also have the correct permissions. The latest crash reports all seem to be ok now.

Comment 13

[Ted Mielczarek [:ted.mielczarek]]

Sounds like this got fixed okay.