snippet uploading doesn't work in staging

RESOLVED FIXED

Status

Release Engineering
General Automation
P3
minor
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [staging])

Attachments

(5 attachments)

(Reporter)

Description

6 years ago
Looks like it tries to use the wrong key:
python /builds/slave/m-cen-osx64-ntly/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa dev-stage01.srv.releng.scl3.mozilla.com mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20120524060256/en-US'
 in dir /builds/slave/m-cen-osx64-ntly/build (timeout 1320 secs)
 watching logfiles {}
 argv: ['python', '/builds/slave/m-cen-osx64-ntly/tools/buildfarm/utils/retry.py', '-s', '1', '-r', '5', '-t', '1260', 'bash', '-c', 'ssh -l cltbld -i ~/.ssh/cltbld_dsa dev-stage01.srv.releng.scl3.mozilla.com mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20120524060256/en-US']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-4axnDk/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-WDUkbz/org.x:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/m-cen-osx64-ntly/build
  PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-ZRaCDP/Listeners
  TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 using PTY: False
retry: Calling <function run_with_timeout at 0x10052db18> with args: (['bash', '-c', 'ssh -l cltbld -i ~/.ssh/cltbld_dsa dev-stage01.srv.releng.scl3.mozilla.com mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20120524060256/en-US'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['bash', '-c', 'ssh -l cltbld -i ~/.ssh/cltbld_dsa dev-stage01.srv.releng.scl3.mozilla.com mkdir -p /opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_x86_64-gcc3/20120524060256/en-US']
Warning: Identity file /Users/cltbld/.ssh/cltbld_dsa not accessible: No such file or directory.
Permission denied (publickey,gssapi-with-mic,password).
using ffxbld/ffxbld_dsa for staging_config.py:aus2_* should fix the problem, iirc

Updated

6 years ago
Severity: normal → minor
Priority: -- → P3
Whiteboard: [staging]
(Reporter)

Comment 2

6 years ago
Created attachment 647338 [details] [diff] [review]
use ffxbld/ffxbld_dsa

Haven't tested this, but it can't make things worse!
Attachment #647338 - Flags: review?(rail)
Attachment #647338 - Flags: review?(rail) → review+
(Reporter)

Updated

6 years ago
Attachment #647338 - Flags: checked-in+
(Reporter)

Comment 3

6 years ago
Should be fixed now.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Created attachment 647340 [details] [diff] [review]
the same for TB
Attachment #647340 - Flags: review?(bhearsum)
(Reporter)

Updated

6 years ago
Attachment #647340 - Flags: review?(bhearsum) → review+
(Reporter)

Comment 6

6 years ago
Still broken for releases...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 7

6 years ago
Created attachment 666514 [details] [diff] [review]
fix staging release configs
Attachment #666514 - Flags: review?(rail)
(Reporter)

Comment 8

6 years ago
Grrrr, even with that we get:
rsync: mkdir "/opt/aus2/snippets/staging/Firefox-16.0b6-build1-test" failed: Permission denied (13)

Because directory permissions are wrong. Puppet patch incoming.
(Reporter)

Comment 9

6 years ago
Created attachment 666523 [details] [diff] [review]
fix dev-stage01 permissions
Attachment #666514 - Flags: review?(rail) → review+
(Reporter)

Comment 10

6 years ago
Comment on attachment 666523 [details] [diff] [review]
fix dev-stage01 permissions

I did this, plus:
cd /opt/aus2/incoming/3
# these 3 are one-offs because of old snippet pushes
find Firefox -exec chown ffxbld:firefox {} \;
find Fennec -exec chown ffxbld:ffxbld {} \;
find Thunderbird -exec chown tbirdbld:tbirdbld {} \;
# this is how it was checked out in ~cltbld too =\. filed this as bug 795902
# as root
cd ~ffxbld
cvs -d:ext:stgbld@cvs.mozilla.org:cvsroot co -d bin mozilla/tools/release/bin
Attachment #666523 - Flags: review?(rail)
(Reporter)

Updated

6 years ago
Attachment #666514 - Flags: checked-in+
Attachment #666523 - Flags: review?(rail) → review+
(Reporter)

Updated

6 years ago
Attachment #666523 - Flags: checked-in+
(Reporter)

Comment 11

6 years ago
Should be good here now.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 12

6 years ago
Broke some cronjobs:
rm: cannot remove `/builds/opt/aus2/snippets/backup/20120923-nightly-1.time': Permission denied
rm: cannot remove `/builds/opt/aus2/snippets/backup/20120923-nightly-1.tar.bz2': Permission denied
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 13

6 years ago
Created attachment 666958 [details] [diff] [review]
create ffxbld crontab

OK, so we need this patch place moving the following from cltbld's crontab to the new ffxbld one directly on the puppet master:
# to match aus2-staging
@daily          /home/cltbld/bin/backupsnip-nightly > /dev/null 2>&1

# periodically clean aus server
@daily  find /builds/opt/aus2/snippets/staging/ -mindepth 1 -mtime +7 -depth -exec rm -rf {} \;  > /dev/null
@daily  find /builds/opt/aus2/snippets/backup/  -mindepth 1 -mtime +7 -depth -exec rm -rf {} \;  > /dev/null
Attachment #666958 - Flags: review?(rail)
Attachment #666958 - Flags: review?(rail) → review+
(Reporter)

Comment 14

6 years ago
Comment on attachment 666958 [details] [diff] [review]
create ffxbld crontab

OK, I landed this with the following contents in ffxbld's crontab (put in the i686 + x86_64 platforms on scl3, scl1, and mv production puppet masters):
MAILTO=release@mozilla.com

# to match aus2-staging
@daily          /home/ffxbld/bin/backupsnip-nightly > /dev/null 2>&1

# periodically clean aus server
@daily  find /builds/opt/aus2/snippets/staging/ -mindepth 1 -mtime +7 -depth -exec rm -rf {} \;  > /dev/null
@daily  find /builds/opt/aus2/snippets/backup/  -mindepth 1 -mtime +7 -depth -exec rm -rf {} \;  > /dev/null


cltbld's crontab now looks like this:
MAILTO=release@mozilla.com
@hourly cd /builds/opt/aus2/build && for file in `find . -name 'complete.txt' -type f -mtime -10`; do url=`grep ^http $file`; if [ "$url" != "" ]; then echo $file `curl -sI $url | grep ^HTTP` | grep HTTP.*404 || continue; rm $file; fi; done >> snippets_removed.log
@hourly cd /builds/opt/aus2/build && for file in `find . -name 'complete.txt' -type f -mtime -10`; do url=`grep ^url=http $file | grep -o http.*`; if [ "$url" != "" ]; then echo $file `curl -sI $url | grep ^HTTP` | grep HTTP.*404 || continue; rm $file; fi; done >> snippets_removed.log


I also tested the new ffxbld jobs by hand to make sure they work, and they do.
Attachment #666958 - Flags: checked-in+
(Reporter)

Comment 15

6 years ago
We should be done here now. Srsly.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.