Closed Bug 1091582 Opened 10 years ago Closed 10 years ago

Push Socorro Release 20141030

Categories

(Socorro :: Infra, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: lonnen, Unassigned)

Details

whoooops. We want the tar not the rpm.

https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
$ ssh -A crash-stats-prod
Last login: Thu Oct 30 06:10:43 2014 from 10-22-248-42.vpn.scl3.mozilla.com
[lonnen@sp-admin01.phx1 ~]$ /usr/local/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
upgrading collectors (in parallel)
[socorro-collector1.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector2.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector3.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector4.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector5.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector6.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector1.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (6.772s)
[socorro-collector1.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector1.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector1.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector1.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_07
[socorro-collector5.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (7.098s)
[socorro-collector5.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector5.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector5.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector5.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_13_12_07
[socorro-collector6.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (7.097s)
[socorro-collector6.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector6.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector6.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector6.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_13_12_07
[socorro-collector2.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (8.401s)
[socorro-collector2.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector2.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector2.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector2.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_13_12_07
[socorro-collector3.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (17.978s)
[socorro-collector3.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector3.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector3.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector3.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_13_12_07
[socorro-collector4.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (18.110s)
[socorro-collector4.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector4.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro-collector4.webapp.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro-collector4.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_13_12_07
upgrading mware (in parallel)
[sp-mware01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (4.244s)
[sp-mware02.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[sp-mware02.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[sp-mware02.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-mware02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-mware02.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_25
[sp-mware01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (5.432s)
[sp-mware01.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[sp-mware01.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[sp-mware01.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-mware01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware01.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-mware01.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_25
[sp-mware03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (6.697s)
[sp-mware03.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[sp-mware03.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[sp-mware03.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-mware03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-mware03.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-mware03.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_25
upgrading processors (in parallel)
[sp-processor01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor04.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor05.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor06.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor07.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor08.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor09.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor10.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (40.467s)
[sp-processor01.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor01.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor01.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor01.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor01.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor09.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (40.863s)
[sp-processor09.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor09.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor09.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor09.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor09.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor09.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor07.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (41.494s)
[sp-processor07.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor07.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor07.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor07.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor07.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor07.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_33
[sp-processor02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (42.013s)
[sp-processor02.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor02.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor02.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor02.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor02.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor06.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (42.638s)
[sp-processor06.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor06.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor06.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor06.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor06.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor06.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor08.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (43.528s)
[sp-processor08.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor08.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor08.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor08.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor08.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor08.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor05.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (43.872s)
[sp-processor05.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor05.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor05.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor05.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor05.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor05.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_33
[sp-processor03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (43.973s)
[sp-processor03.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor03.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor03.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor03.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor03.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
[sp-processor04.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (44.024s)
[sp-processor04.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor04.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor04.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor04.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor04.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor04.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_33
[sp-processor10.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (46.293s)
[sp-processor10.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor10.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor10.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor10.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[sp-processor10.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[sp-processor10.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_12_32
upgrading db servers (in parallel)
[socorro1.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro2.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro2.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (5.141s)
[socorro2.db.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro2.db.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro2.db.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro2.db.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_13_18
[socorro1.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz (7.915s)
[socorro1.db.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro1.db.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
[socorro1.db.phx1.mozilla.com] out: Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
[socorro1.db.phx1.mozilla.com] out: Backed up original to /data/socorro.30-10-2014_06_13_18
upgrading admin host
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
Checksum: 5780ffcc5ddedbaeab9ec03cb0c20b37
Backed up original to /data/socorro.30-10-2014_06_13_26
upgrading django servers (in parallel)
DIR: /data/crashstats/src/crash-stats.mozilla.org
ENV: crashstats
SITE: crash-stats.mozilla.org
APP: socorro
Oct 30 13:13:41 socorroadm update[426] Upgrading socorro to https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
Oct 30 13:13:41 socorroadm update[426] Entering /data/crashstats/src/crash-stats.mozilla.org
/data/crashstats/src/crash-stats.mozilla.org /home/lonnen
Oct 30 13:13:41 socorroadm update[426] Acquiring https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
Oct 30 13:13:42 socorroadm update[426] Removing old socorro directory
Oct 30 13:13:42 socorroadm update[426] Unpacking build tarball
Oct 30 13:13:45 socorroadm update[426] Merging local.py
Oct 30 13:13:45 socorroadm update[426] Running syncdb
Traceback (most recent call last):
  File "socorro/webapp-django/manage.py", line 7, in <module>
    from funfactory import manage
ImportError: No module named funfactory
Oct 30 13:13:45 socorroadm update[426] Activating symbols_upload symlink
Oct 30 13:13:45 socorroadm update[426] Initiating deployment
[2014-10-30 13:13:45] Running rsync_project
[2014-10-30 13:13:45] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/
[2014-10-30 13:13:50] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ (5.456s)
[2014-10-30 13:13:50] Finished rsync_project (5.457s)
[2014-10-30 13:13:50] Running commit_www
[2014-10-30 13:13:50] [localhost] running: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']'
[2014-10-30 13:13:58] [localhost] finished: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' (7.506s)
[localhost] out: [master c01d140] deploy [crash-stats.mozilla.org]
[localhost] out: 314 files changed, 2984 insertions(+), 7498 deletions(-)
[localhost] out: rename crash-stats.mozilla.org/socorro/application/scripts/{socorro_app => socorro} (100%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/application/socorro/external/fetch_app.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/application/socorro/external/postgresql/postgresqlalchemymanager.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/application/socorro/processor/timemachine.py
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.7-py2.6.egg-info/SOURCES.txt
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._PKG-INFO
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._SOURCES.txt
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._dependency_links.txt
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._not-zip-safe
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._requires.txt
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/._top_level.txt
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/PKG-INFO (99%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.2.8-py2.6.egg-info/SOURCES.txt
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/dependency_links.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/installed-files.txt (71%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/not-zip-safe (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/requires.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{configman-1.2.7-py2.6.egg-info => configman-1.2.8-py2.6.egg-info}/top_level.txt (100%)
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/argparse_.py
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/config_manager.savde.py
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/converters.old.py
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/converters2.py
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/a.ini
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_argparse
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_config_manager.py.lars2
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_def_for_argparse
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_val_for_configparse.py.lars
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_val_for_json.py.orig
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/tests/test_val_for_mapping.py.old
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/value_sources/__init__.py.lars
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/value_sources/__init__.py.orig
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/value_sources/f.ini
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/value_sources/for_configparse.py.lars
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/value_sources/for_json.py.orig
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/crontabber-0.13-py2.6.egg-info/requires.txt
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/PKG-INFO (99%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/SOURCES.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/dependency_links.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/entry_points.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/installed-files.txt (100%)
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/not-zip-safe (100%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/crontabber-0.15-py2.6.egg-info/requires.txt
[localhost] out: rename crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/{crontabber-0.13-py2.6.egg-info => crontabber-0.15-py2.6.egg-info}/top_level.txt (100%)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/stackwalk/bin/core2md (70%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/stackwalk/src/client/linux/microdump_writer/microdump_writer.cc
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/stackwalk/src/client/linux/microdump_writer/microdump_writer.h
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/stackwalk/src/common/android/include/sys/ucontext.h
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/crashstats/base/static/img/2.0/footer-bg.png
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/crashstats/base/static/img/2.0/page-bg.png
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/crashstats/base/static/img/2.0/panel-bg.png
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/crashstats/base/static/img/3rdparty/silk/exclamation.png
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/crashstats/crashstats/templates/crashstats/topchangers.html (77%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{1dc9b057202e.css => 02590f2d7d02.css} (95%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/045e30c291c4.css
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/081c5329ee3d.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/0b539c9571c7.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{9965899265bf.css => 11855c2bf4e6.css} (81%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{0c7511c87d5f.css => 207687a820a1.css} (93%)
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/2f93e6251b11.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{bc2745a7a858.css => 30d0bd60b4ef.css} (95%)
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/33ea4eafc253.css
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/3ca369412c0a.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{0d589506ca72.css => 3cfbd57e9a8b.css} (81%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{58d21dfb4df3.css => 433fb9d5c551.css} (92%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{0afc4750659d.css => 43a5a0c7976b.css} (95%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{cf9d44b13195.css => 46da00a4796d.css} (95%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/518d076abacf.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/591455d49b38.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/59eec88792c2.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/69a52ee1d633.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/6a51534ee391.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/6a69e39ed066.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/7bd9e34c0a06.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{c265d56aded8.css => 8ce755168902.css} (91%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/8d49d107293c.css
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/9474003389a7.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{70e498265fe1.css => 94d808a9738a.css} (79%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/9f8352e30c73.css
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/a4598a674dcd.css
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{5f6dc142325a.css => b141ca82df6d.css} (77%)
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/{6eb6084e9995.css => ca12b22f3f8d.css} (63%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/ccd85308eb6e.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/e04b12ad8f11.css
[localhost] out: delete mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/fa9ab582c5a9.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/3e5d74f04b4d.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/66d5902544f1.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/8a1ec3379747.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/9f16676d06a0.js
[localhost] out: rename crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/{3f9806106475.js => b1d1381234e9.js} (84%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/fb0f38ff0835.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/fff1b05138f6.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/img/3rdparty/silk/exclamation.png
[2014-10-30 13:13:58] Finished commit_www (7.507s)
[2014-10-30 13:13:58] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh
[2014-10-30 13:13:58] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh
[2014-10-30 13:13:58] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh
[2014-10-30 13:13:58] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh
[2014-10-30 13:14:21] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (23.059s)
[socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/
[2014-10-30 13:14:21] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (23.109s)
[2014-10-30 13:14:21] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (23.635s)
[2014-10-30 13:14:21] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (23.636s)
[2014-10-30 13:14:21] [socorro1.webapp.phx1.mozilla.com] running: service httpd restart
[2014-10-30 13:14:21] [socorro2.webapp.phx1.mozilla.com] running: service httpd restart
[2014-10-30 13:14:21] [socorro3.webapp.phx1.mozilla.com] running: service httpd restart
[2014-10-30 13:14:21] [socorro4.webapp.phx1.mozilla.com] running: service httpd restart
[2014-10-30 13:14:23] [socorro3.webapp.phx1.mozilla.com] finished: service httpd restart (1.403s)
[socorro3.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro3.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2014-10-30 13:14:23] [socorro2.webapp.phx1.mozilla.com] finished: service httpd restart (1.427s)
[socorro2.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro2.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2014-10-30 13:14:23] [socorro4.webapp.phx1.mozilla.com] finished: service httpd restart (1.430s)
[socorro4.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro4.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2014-10-30 13:14:23] [socorro1.webapp.phx1.mozilla.com] finished: service httpd restart (1.442s)
[socorro1.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro1.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2014-10-30 13:14:23] [socorro-memcache1.webapp.phx1.mozilla.com] running: service memcached restart
[2014-10-30 13:14:23] [socorro-memcache2.webapp.phx1.mozilla.com] running: service memcached restart
[2014-10-30 13:14:23] [socorro-memcache2.webapp.phx1.mozilla.com] finished: service memcached restart (0.320s)
[socorro-memcache2.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache2.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[2014-10-30 13:14:23] [socorro-memcache1.webapp.phx1.mozilla.com] finished: service memcached restart (0.357s)
[socorro-memcache1.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache1.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[2014-10-30 13:14:23] [socorro1.webapp.phx1.mozilla.com] running: chown apache /mnt/socorro/symbols_upload
[2014-10-30 13:14:23] [socorro2.webapp.phx1.mozilla.com] running: chown apache /mnt/socorro/symbols_upload
[2014-10-30 13:14:23] [socorro3.webapp.phx1.mozilla.com] running: chown apache /mnt/socorro/symbols_upload
[2014-10-30 13:14:23] [socorro4.webapp.phx1.mozilla.com] running: chown apache /mnt/socorro/symbols_upload
[2014-10-30 13:14:24] [socorro2.webapp.phx1.mozilla.com] finished: chown apache /mnt/socorro/symbols_upload (0.655s)
[2014-10-30 13:14:24] [socorro3.webapp.phx1.mozilla.com] finished: chown apache /mnt/socorro/symbols_upload (0.663s)
[2014-10-30 13:14:24] [socorro4.webapp.phx1.mozilla.com] finished: chown apache /mnt/socorro/symbols_upload (0.683s)
[2014-10-30 13:14:24] [socorro1.webapp.phx1.mozilla.com] finished: chown apache /mnt/socorro/symbols_upload (0.689s)
[2014-10-30 13:14:24] [socorro1.webapp.phx1.mozilla.com] running: chmod 2664 /mnt/socorro/symbols_upload
[2014-10-30 13:14:24] [socorro2.webapp.phx1.mozilla.com] running: chmod 2664 /mnt/socorro/symbols_upload
[2014-10-30 13:14:24] [socorro3.webapp.phx1.mozilla.com] running: chmod 2664 /mnt/socorro/symbols_upload
[2014-10-30 13:14:24] [socorro4.webapp.phx1.mozilla.com] running: chmod 2664 /mnt/socorro/symbols_upload
[2014-10-30 13:14:24] [socorro1.webapp.phx1.mozilla.com] finished: chmod 2664 /mnt/socorro/symbols_upload (0.224s)
[2014-10-30 13:14:24] [socorro2.webapp.phx1.mozilla.com] finished: chmod 2664 /mnt/socorro/symbols_upload (0.230s)
[2014-10-30 13:14:24] [socorro4.webapp.phx1.mozilla.com] finished: chmod 2664 /mnt/socorro/symbols_upload (0.230s)
[2014-10-30 13:14:24] [socorro3.webapp.phx1.mozilla.com] finished: chmod 2664 /mnt/socorro/symbols_upload (0.233s)
Oct 30 13:14:24 socorroadm update[426] Fin.
finished upgrading socorro to https://ci.mozilla.org/job/socorro-release/1457/artifact/socorro.tar.gz
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.