Closed Bug 1119238 Opened 9 years ago Closed 9 years ago

Get a Socorro authentication key for symbol upload onto all build machines

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Assigned: coop)

References

Details

Attachments

(4 files)

We want to move from uploading Breakpad symbols using SCP to using Socorro's upload API. We'll need an auth key to do this, though, which will need to live on the build machines.
Depends on: 1119241
coop: I think we want to handle this similar to the Google API key--a puppet'ed secret that winds up in a known location on the build machines so we can pass it to a configure option in mozconfigs.
Assignee: nobody → coop
As an update, I've added the token to hiera (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Secrets) and will roll the puppet patches to deploy this to build machines tomorrow.
Status: NEW → ASSIGNED
(In reply to Chris Cooper [:coop] from comment #3)
> As an update, I've added the token to hiera
> (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Secrets)
> and will roll the puppet patches to deploy this to build machines tomorrow.

Flight delay in your favor, collect patch now.

Token will live in C:/builds/crash-stats-api.token (Windows) or /builds/crash-stats-api.token (POSIX).

This is based on the patch for the google oauth key.
Attachment #8570278 - Flags: review?(catlee)
Attachment #8570278 - Flags: review?(catlee) → review+
Comment on attachment 8570278 [details] [diff] [review]
[puppet] Add API token for crash-stats upload

Review of attachment 8570278 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/a8af1412e229
Attachment #8570278 - Flags: checked-in+
I've merged to the production tag, and verified that builders receive the new token once they re-puppetize.

We'll need to wait a day or two for the golden AMIs to pick up this change, but we should be ready to start using the token by Monday.
Should this be ready to go by now?
Flags: needinfo?(coop)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7)
> Should this be ready to go by now?

For Linux and Mac, yes. I've double-checked that the token is present there.

The Windows builders still need GPO created to deploy the file.
Flags: needinfo?(coop)
Depends on: 1139649
(In reply to Chris Cooper [:coop] from comment #8)
> The Windows builders still need GPO created to deploy the file.

Filed bug 1139649.
Seems fixed now, thanks!
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Apparently this didn't get completely fixed, the nightly build on bld-linux64-spot-128 didn't have the token file:
https://treeherder.mozilla.org/logviewer.html#?job_id=1197057&repo=mozilla-central

08:51:15 INFO - /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.linux-i686.crashreporter-symbols-full.zip'
08:51:15 INFO - Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist!
08:51:15 INFO - gmake[1]: *** [uploadsymbols] Error 1 

Same error on the 32-bit nightly on bld-linux64-spot-1036:
https://treeherder.mozilla.org/logviewer.html#?job_id=1197058&repo=mozilla-central
Status: RESOLVED → REOPENED
Flags: needinfo?(coop)
Resolution: FIXED → ---
The token is there, but the permissions are set to 0600, same as the google-oauth-api.key.

I'll switch this to 0644 in puppet.
Flags: needinfo?(coop)
Except it's failing here:
https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#45

Which indicates that the script can't see the file at all. 0600 should be fine as long as the owner of the file is the user we run builds as.
ahhh of course, we need to copy it into the mock environment, ala:

http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#261
Comment on attachment 8581696 [details] [diff] [review]
[buildbot-configs] Copy crash-stats token into mock env

Review of attachment 8581696 [details] [diff] [review]:
-----------------------------------------------------------------

stamp
Attachment #8581696 - Flags: review?(bugspam.Callek) → review+
Based on IRC chatter and confusion on if this was a flow issue, I edited "firewall-tests" and added a test to ensure this flow remains open:

To ssh://gitolite3@git-internal.mozilla.org/releng/firewall-tests
   d40176b..60a81c1  master -> master

$ git diff -U0 d40176b..60a81c1
diff --git a/test_buildslaves.py b/test_buildslaves.py
index 9697738..ec7e9f3 100644
--- a/test_buildslaves.py
+++ b/test_buildslaves.py
@@ -44,0 +45,5 @@ slaveapi = IPSet([
+crashstats = IPSet([
+    host('crash-stats.mozilla.org'),
+    host('crash-stats.allizom.org')
+])
+
@@ -69,0 +75,8 @@ def test_blobupload_access():
+
+def test_crashstats_access():
+    """All buildslaves can access crashstats"""
+    # XXX Only hosts that compile code need the access
+    rules.assertPermits(
+            slave_vlans,
+            crashstats,
+            'https')
Presumably this should be fixed now. Thanks coop for the patch and Callek for figuring out the issue! Maybe one of you can edit the docs at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Add_new_secrets so this extra step is noted for anyone adding a similar secret file for the build in the future?
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #19)
> Maybe one of you can edit the docs at
> https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/
> Add_new_secrets so this extra step is noted for anyone adding a similar
> secret file for the build in the future?

Done: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Add_new_secrets
This is somehow still broken because I'm not allowed to have nice things:
https://treeherder.mozilla.org/logviewer.html#?job_id=1218661&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=1218662&repo=mozilla-central
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #21)
> This is somehow still broken because I'm not allowed to have nice things:
> https://treeherder.mozilla.org/logviewer.html#?job_id=1218661&repo=mozilla-
> central
> https://treeherder.mozilla.org/logviewer.html#?job_id=1218662&repo=mozilla-
> central

On linux, no less. I'll dig into this today.
The mozharness script, fx_desktop_build.py, is getting its list of copyin files from the mozharness config files rather than buildbot:

https://hg.mozilla.org/build/mozharness/file/e1db791a6e6f/configs/builds/releng_base_linux_32_builds.py#l37
https://hg.mozilla.org/build/mozharness/file/e1db791a6e6f/configs/builds/releng_base_linux_64_builds.py#l37

Patch incoming.
Comment on attachment 8581696 [details] [diff] [review]
[buildbot-configs] Copy crash-stats token into mock env

Review of attachment 8581696 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/5d5b09f943ea
Attachment #8581696 - Flags: checked-in+
Comment on attachment 8587035 [details] [diff] [review]
[mozharness] Copy crash-stats token into mock env

Review of attachment 8587035 [details] [diff] [review]:
-----------------------------------------------------------------

looks good. Do we need to do the same for the b2g desktop builds? 

http://mxr.mozilla.org/build/source/mozharness/configs/b2g/desktop_linux64.py#35
http://mxr.mozilla.org/build/source/mozharness/configs/b2g/desktop_linux32.py#35
Attachment #8587035 - Flags: review?(jlund) → review+
Depends on whether those are going to wind up in taskcluster, but yes, everything that's uploading symbols currently is going to need this treatment.
Comment on attachment 8587035 [details] [diff] [review]
[mozharness] Copy crash-stats token into mock env

Review of attachment 8587035 [details] [diff] [review]:
-----------------------------------------------------------------

Updated the b2g desktop files as well:

https://hg.mozilla.org/build/mozharness/rev/7c12014639c0
Attachment #8587035 - Flags: checked-in+
Okay, so this just needs the in-tree pinned mozharness rev bumped to pick this up now?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #30)
> Okay, so this just needs the in-tree pinned mozharness rev bumped to pick
> this up now?

Yes, and you can do this yourself, or ask jlund to do it for you.
Someone else updated mozharness.json, my procrastination paid off!
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Component: General Automation → General
Attached file GitHub Pull Request

removes crash_stats_api_token from puppet configuration #568

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: