Closed Bug 1150935 Opened 9 years ago Closed 9 years ago

Push Socorro Release 20150403

Categories

(Socorro :: Infra, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: rhelmer)

References

Details

Blocks: 1150795
Blocks: 1150881
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
upgrading collectors (in parallel)
[socorro-collector1.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector2.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector3.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector4.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector5.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector6.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector5.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (7.940s)
[socorro-collector5.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector5.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector5.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector5.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector5.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_16_50_28
[socorro-collector3.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (8.367s)
[socorro-collector3.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector3.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector3.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector3.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector3.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_16_50_28
[socorro-collector4.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (9.223s)
[socorro-collector4.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector4.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector4.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector4.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector4.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_16_50_27
[socorro-collector1.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (17.625s)
[socorro-collector1.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector1.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector1.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector1.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector1.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_09_50_27
[socorro-collector6.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (17.941s)
[socorro-collector6.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector6.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector6.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector6.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector6.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_16_50_28
[socorro-collector2.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (19.270s)
[socorro-collector2.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[socorro-collector2.webapp.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro-collector2.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro-collector2.webapp.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro-collector2.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_16_50_27
upgrading mware (in parallel)
[sp-mware01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-mware03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (6.697s)
[sp-mware02.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[sp-mware02.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[sp-mware02.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-mware02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-mware02.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-mware02.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_09_50_47
[sp-mware03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (18.934s)
[sp-mware03.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[sp-mware03.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[sp-mware03.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-mware03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-mware03.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-mware03.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_09_50_47
[sp-mware01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (814.936s)
[sp-mware01.phx1.mozilla.com] out: Stopping httpd: [FAILED]
[sp-mware01.phx1.mozilla.com] out: Starting httpd: [FAILED]
[sp-mware01.phx1.mozilla.com] out: ERROR: could not start httpd
[sp-mware01.phx1.mozilla.com] err: (98)Address already in use: make_sock: could not bind to address [::]:80
[sp-mware01.phx1.mozilla.com] err: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
[sp-mware01.phx1.mozilla.com] err: no listening sockets available, shutting down
[sp-mware01.phx1.mozilla.com] err: Unable to open logs
upgrading processors (in parallel)
[sp-processor01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor04.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor05.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor06.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor07.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor08.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor09.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor10.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (41.702s)
[sp-processor03.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor03.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor03.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor03.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor03.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor04.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (42.565s)
[sp-processor04.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor04.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor04.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor04.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor04.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor04.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor06.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (43.257s)
[sp-processor06.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor06.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor06.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor06.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor06.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor06.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (43.427s)
[sp-processor01.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor01.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor01.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor01.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor01.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (46.434s)
[sp-processor02.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor02.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor02.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor02.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor02.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor10.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (47.067s)
[sp-processor10.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor10.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor10.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor10.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor10.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor10.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor08.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (47.242s)
[sp-processor08.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor08.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor08.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor08.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor08.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor08.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor07.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (47.616s)
[sp-processor07.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor07.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor07.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor07.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor07.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor07.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor05.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (49.152s)
[sp-processor05.phx1.mozilla.com] out: Stopping socorro-processor: [  OK  ]
[sp-processor05.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor05.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor05.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor05.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor05.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
[sp-processor09.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (56.146s)
[sp-processor09.phx1.mozilla.com] out: Stopping socorro-processor: [FAILED]
[sp-processor09.phx1.mozilla.com] out: Starting socorro-processor: [  OK  ]
[sp-processor09.phx1.mozilla.com] out: Socorro build installed successfully!
[sp-processor09.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[sp-processor09.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[sp-processor09.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_10_04_22
upgrading db servers (in parallel)
[socorro4.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro5.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro6.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro6.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (0.635s)
[socorro6.db.phx1.mozilla.com] err: sudo: /data/bin/update-socorro.sh: command not found
[socorro5.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (0.669s)
[socorro5.db.phx1.mozilla.com] err: sudo: /data/bin/update-socorro.sh: command not found
[socorro4.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz (4.596s)
[socorro4.db.phx1.mozilla.com] out: Socorro build installed successfully!
[socorro4.db.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
[socorro4.db.phx1.mozilla.com] out: Checksum: fd625cb7045d8af2ba2ae09cd062728a
[socorro4.db.phx1.mozilla.com] out: Backed up original to /data/socorro.03-04-2015_17_05_18
upgrading admin host
Stopping httpd: [60G[[0;32m  OK  [0;39m]
Starting httpd: [60G[[0;32m  OK  [0;39m]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
Checksum: fd625cb7045d8af2ba2ae09cd062728a
Backed up original to /data/socorro.03-04-2015_10_05_22
upgrading django servers (in parallel)
DIR: /data/crashstats/src/crash-stats.mozilla.org
ENV: crashstats
SITE: crash-stats.mozilla.org
APP: socorro
Apr 03 17:05:39 socorroadm update[11606] Upgrading socorro to https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
Apr 03 17:05:39 socorroadm update[11606] Entering /data/crashstats/src/crash-stats.mozilla.org
/data/crashstats/src/crash-stats.mozilla.org /home/rhelmer
Apr 03 17:05:39 socorroadm update[11606] Acquiring https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
Apr 03 17:05:40 socorroadm update[11606] Removing old socorro directory
Apr 03 17:05:41 socorroadm update[11606] Unpacking build tarball
Apr 03 17:05:44 socorroadm update[11606] Merging local.py
Apr 03 17:05:44 socorroadm update[11606] Running syncdb
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > funfactory
 > compressor
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.staticfiles
 > commonware.response.cookies
 > south
 > session_csrf
 > crashstats.base
 > crashstats.dataservice
 > crashstats.crashstats
 > crashstats.api
 > crashstats.manage
 > crashstats.supersearch
 > crashstats.signature
 > crashstats.auth
 > django.contrib.messages
 > raven.contrib.django.raven_compat
 > eventlog

Not synced (use migrations):
 - crashstats.tokens
 - crashstats.symbols
 - waffle
(use ./manage.py migrate to migrate these)
Apr 03 17:05:44 socorroadm update[11606] Running migrate
Running migrations for tokens:
- Nothing to migrate.
 - Loading initial data for tokens.
Installed 0 object(s) from 0 fixture(s)
Running migrations for symbols:
- Nothing to migrate.
 - Loading initial data for symbols.
Installed 0 object(s) from 0 fixture(s)
Running migrations for waffle:
- Nothing to migrate.
 - Loading initial data for waffle.
Installed 0 object(s) from 0 fixture(s)
Apr 03 17:05:45 socorroadm update[11606] Activating symbols_upload symlink
Apr 03 17:05:45 socorroadm update[11606] Initiating deployment
Apr 03 17:06:21 socorroadm update[11606] Fin.
finished upgrading socorro to https://ci.mozilla.org/job/socorro-release/1686/artifact/socorro.tar.gz
No longer blocks: 1150795, 1150881
Blocks: 1150795, 1150881
(In reply to Robert Helmer [:rhelmer] from comment #1)
> [sp-mware01.phx1.mozilla.com] out: Stopping httpd: [FAILED]
> [sp-mware01.phx1.mozilla.com] out: Starting httpd: [FAILED]
> [sp-mware01.phx1.mozilla.com] out: ERROR: could not start httpd
> [sp-mware01.phx1.mozilla.com] err: (98)Address already in use: make_sock:
> could not bind to address [::]:80
> [sp-mware01.phx1.mozilla.com] err: (98)Address already in use: make_sock:
> could not bind to address 0.0.0.0:80
> [sp-mware01.phx1.mozilla.com] err: no listening sockets available, shutting
> down
> [sp-mware01.phx1.mozilla.com] err: Unable to open logs
> upgrading processors (in parallel)

I investigated this failure, some httpd processes were hanging and didn't shut down:

[rhelmer@sp-mware01.phx1 httpd]$ ps -ef | grep http
apache    2348     1  0 08:18 ?        00:00:26 /usr/sbin/httpd
apache    2672     1  0 08:20 ?        00:00:25 /usr/sbin/httpd
apache    2700     1  0 08:20 ?        00:00:24 /usr/sbin/httpd
apache    2704     1  0 08:20 ?        00:00:27 /usr/sbin/httpd
rhelmer  11266 10809  0 10:06 pts/0    00:00:00 grep http
apache   13983     1  0 04:29 ?        00:01:39 /usr/sbin/httpd
apache   13985     1  0 04:29 ?        00:02:00 /usr/sbin/httpd
apache   13986     1  0 04:29 ?        00:02:02 /usr/sbin/httpd
apache   29935     1  0 07:06 ?        00:01:24 /usr/sbin/httpd
apache   30027     1  0 07:10 ?        00:00:25 /usr/sbin/httpd

Killed these and restarted apache:

[rhelmer@sp-mware01.phx1 httpd]$ sudo pkill -9 httpd
[rhelmer@sp-mware01.phx1 httpd]$ sudo /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]
Status: NEW → ASSIGNED
Ran migrations:

INFO  [alembic.migration] Running upgrade 63d05b930c3 -> 3e9fd64194df, Fixes bug 1149716 update partitioned-by-date table functions to properly handle partitioned tables
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.