650689 - linux mobile desktop symbol uploads failing

Status: RESOLVED DUPLICATE of bug 592244

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

This isn't a key issue -- we're able to upload a number of symbols, but not others.

error:  cannot create TestStringAPI/CE123D606509AC1D785D31298ACE900F0/TestStringAPI.dbg.gz
error:  cannot create TestRefPtr/A0BB9606592A705EDCD652583D90E7310/TestRefPtr.dbg.gz
error:  cannot create grabpage/87B9F02A6E02C9501C24DCCDCE512EFD0/grabpage.dbg.gz
error:  cannot create xpcshell/977FBDFB955FFEBBD69B84068A979A750/xpcshell.dbg.gz
make: *** [uploadsymbols] Error 50

Permissions? disk space?

This happened 4 times in a row on 3 different linux-ix-slaveXX boxes.

Comment 1

[Aki Sasaki (not active)]

http://buildbot-master5.build.mozilla.org:8010/builders/Linux%20Mobile%20Desktop%20mozilla-central%20nightly/builds/4

http://buildbot-master2.build.scl1.mozilla.com:8010/builders/Linux%20Mobile%20Desktop%20mozilla-central%20nightly/builds/52

http://buildbot-master1.build.scl1.mozilla.com:8010/builders/Linux%20Mobile%20Desktop%20mozilla-central%20nightly/builds/17

http://buildbot-master5.build.mozilla.org:8010/builders/Linux%20Mobile%20Desktop%20mozilla-central%20nightly/builds/5

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Ted, any ideas what is going on here?

Comment 3

[(not currently active) Ted Mielczarek]

What are SYMBOL_SERVER_USER  and SYMBOL_SERVER_PATH set to?

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

user is ffxbld and path is /mnt/netapp/breakpad/symbols_mob/

Comment 5

[(not currently active) Ted Mielczarek]

[tmielczarek@dm-symbolpush01 symbols_mob]$ ls -l TestStringAPI/CE123*
total 144
-rwxrwxr-x 1 ffxbld users 45523 Apr 18 05:21 TestStringAPI.dbg.gz

Permissions look fine. There's a bit over 200GB free on that volume, so I don't think that's the problem either.

We also run unzip with -o for overwrite:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.sh#67

Unless something changed here, I'm not sure what to tell you.

Comment 6

[Aki Sasaki (not active)]

Hm, 3 of these nightlies happened on April 16, which is a) making me wonder if our bbdb bug is biting us, and b) making me wonder if they're all trying to upload symbols into the same directory at the same time.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #6)
> Hm, 3 of these nightlies happened on April 16, which is a) making me wonder if
> our bbdb bug is biting us, and b) making me wonder if they're all trying to
> upload symbols into the same directory at the same time.

It sounds like this is fallout from 592244.  Can we dupe to it?