685115 - Win64 on try consistently fails 'make uploadsymbols'

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P4)

Description

[Nick Thomas [:nthomas] (UTC+12)]

In the env:
  SYMBOL_SERVER_HOST=dm-symbolpush01.mozilla.org
  SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
  SYMBOL_SERVER_SSH_KEY=/c/Users/cltbld/.ssh/ffxbld_dsa
  SYMBOL_SERVER_USER=ffxbld

The failure:
/bin/sh /e/builds/moz2_slave/try-w64/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-9.0a1-WINNT-20110906231350-win64-symbols.txt "./dist/firefox-9.0a1.en-US.win64-x86_64.crashreporter-symbols-full.zip"
Transferring symbols... ./dist/firefox-9.0a1.en-US.win64-x86_64.crashreporter-symbols-full.zip
Warning: Identity file /c/Users/cltbld/.ssh/ffxbld_dsa not accessible: No such file or directory.
Permission denied (publickey,gssapi-with-mic).
lost connection
make: *** [uploadsymbols] Error 1
program finished with exit code 2

At this point we've compiled and run 'make buildsymbols', but not packaged or uploaded the actual browser, or done any sendchanges.

By comparison, the win32 try builds set this in the env:
  SYMBOL_SERVER_HOST=build.mozilla.org
  SYMBOL_SERVER_PATH=/symbols/windows
  SYMBOL_SERVER_SSH_KEY=/c/Documents and Settings/cltbld/.ssh/trybld_dsa
  SYMBOL_SERVER_USER=trybld

So we need to
 * fix the four env vars
 * figure out how much space we have at build.m.o:/symbols/windows, and if it's going to be enough to host 64bit symbols too

Might want to disable symbol upload in the meantime.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Disable symbol upload for now

Here's a patch to get you started, given

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1              50G   38G   12G  78% /symbols

Comment 2

[Armen [:armenzg]]

Merged to production and reconfigured.

Comment 3

[Armen [:armenzg]]

Comment on attachment 558768 [details] [diff] [review]
[buildbot-configs] Disable symbol upload for now

2f9fdd317000

Comment 4

[Armen [:armenzg]]

This got fixed:
https://tbpl.mozilla.org/?noignore=1&jobname=WINNT%206.1%20x86-64&usebuildbot=1&rev=787ebf02a8c1

We will have to reenable this ASAP.

Comment 5

[Armen [:armenzg]]

Attachment: [untested] add win64 symbols for try back

It seems that we don't have enough space on build.m.o for both win32 and win64 symbols.

Let's see if we can increase the size.

Comment 6

[Armen [:armenzg]]

I am going to P4 this for now.

We suspect that symbols on the try server are not as important as other branches but to verify that we would need to check at the apache logs.
Unfortunately we can't see those logs on build.m.o because we don't have root.

We would need to either reduce the cronjob [1] from removing the last 14 days to 7 days so we can have enough room for both win32 and win64 symbols OR talk with IT on finding if we can unmount the partition, increase the size, rescan, remount without affecting things like self-serve without a downtime.

[1] 0 3 * * * find /symbols/windows -mtime +14 -prune -exec rm -rf {} \;

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage.

(In reply to Armen Zambrano G. [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #5)
> Created attachment 558948 [details] [diff] [review]
> [untested] add win64 symbols for try back
> 
> It seems that we don't have enough space on build.m.o for both win32 and
> win64 symbols.
> 
> Let's see if we can increase the size.

I know this comment was a few years ago... If capacity on build.m.o is still an issue, let hwine+me know and we'll get that increased.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

We're not uploading win32 symbols from try to build.m.o any more, so this is moot (bug 702337).

Instead, developers can add a patch (https://wiki.mozilla.org/ReleaseEngineering/TryServer#Getting_debug_symbols), which changes the symbols we automatically upload to ftp.m.o. This should work just the same for win64.