1055812 - Run manage_foopies.py after reconfig is completed in end_to_end_reconfig.sh

Assignee

Description

•

10 years ago

I frequently hit bug 1039982 while running end_to_end_reconfig.sh. Until that bug is fixed, we should move the manage_foopies.py steps to after the reconfig so that the reconfig always completes in its entirety, and we can Ctrl-C out of the foopy update if need be.

Chris Cooper [:coop] (he/him)

Assignee

Updated

•

10 years ago

Status: NEW → ASSIGNED

Priority: -- → P2

Chris Cooper [:coop] (he/him)

Assignee

Comment 1

•

10 years ago

Attached patch Run manage_foopies.py after reconfig is completed. — Details — Splinter Review

Attachment #8475909 - Flags: review?(sbruno)

Simone Bruno [:simone]

Comment 2

•

10 years ago

Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

It looks ok to me, and I tested this patch with reconfig on 2014-08-20 07:47 PT

Attachment #8475909 - Flags: review?(sbruno) → review+

Chris Cooper [:coop] (he/him)

Assignee

Comment 3

•

10 years ago

Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/tools/rev/a57f32992270

Attachment #8475909 - Flags: checked-in+

Chris Cooper [:coop] (he/him)

Assignee

Updated

•

10 years ago

Status: ASSIGNED → RESOLVED

Closed: 10 years ago

Resolution: --- → FIXED

Jordan Lund (:jlund)

Comment 4

•

10 years ago

looks like this worked. reconfig finished fine but foopy part hung so I ctrl-c'd it.

output of foopy log for documentation:

jlund@Hastings163:~MOZ/reconfigs
> tail -f manage_foopies-1408555585.log
[foopy97.build.mozilla.org] out: adding manifests
[foopy97.build.mozilla.org] out: (run 'hg update' to get a working copy)
[foopy98.build.mozilla.org] out: adding file changes
[foopy97.build.mozilla.org] out: added 2 changesets with 4 changes to 4 files
[foopy97.build.mozilla.org] out: 4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy99.build.mozilla.org] out:

[foopy99.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
(run 'hg update' to get a working copy)
[foopy97.build.mozilla.org] out: 4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy96.build.mozilla.org] out:

[foopy96.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy98.build.mozilla.org] out:

[foopy98.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy97.build.mozilla.org] out:

[foopy97.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
[OK]   updated foopy99 tools to a57f32992270
[OK]   updated foopy98 tools to a57f32992270
[OK]   updated foopy96 tools to a57f32992270
[OK]   updated foopy97 tools to a57f32992270
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy64.build.mozilla.org] out:

[foopy64.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
[OK]   updated foopy52 tools to a57f32992270
[OK]   updated foopy64 tools to a57f32992270
Process PoolWorker-30:
Process PoolWorker-28:
Process PoolWorker-26:
Process PoolWorker-23:
Process PoolWorker-29:
Process PoolWorker-32:
Process PoolWorker-27:
Process PoolWorker-31:
Process PoolWorker-25:
Process PoolWorker-24:
Process PoolWorker-22:
Process PoolWorker-20:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-21:
Process PoolWorker-17:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
Failed to run  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
 update on foopy116
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
    self.run()
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
    task = get()
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    racquire()
    racquire()
    task = get()
    racquire()
    racquire()
    racquire()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    return recv()
KeyboardInterrupt
    racquire()
    racquire()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    racquire()
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
KeyboardInterrupt
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    racquire()
KeyboardInterrupt
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    racquire()
KeyboardInterrupt
Traceback (most recent call last):
  File "./manage_foopies.py", line 68, in run_action_on_foopy
    action_func(foopy)
  File "/Users/jlund/devel/mozilla/cleanRepos/tools/buildfarm/maintenance/foopy_fabric.py", line 71, in update
    run('hg pull && hg update -r %s' % revision)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/network.py", line 639, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 1042, in run
    shell_escape=shell_escape)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 911, in _run_command
    stderr=stderr, timeout=timeout)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 797, in _execute
    time.sleep(ssh.io_sleep)
KeyboardInterrupt

Traceback (most recent call last):
  File "./manage_foopies.py", line 165, in <module>
    if not result.get():
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 548, in get
    self.wait(timeout)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 543, in wait
    self._cond.wait(timeout)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 339, in wait
    waiter.acquire()
KeyboardInterrupt

^C

Jordan Lund (:jlund)

Comment 5

•

10 years ago

Attached file manage_foopies-1408645327.log — Details

more foopy fails

Pete Moore [:pmoore][:pete]

Comment 6

•

10 years ago

Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/end_to_end_reconfig.sh
@@ +383,5 @@
>  cat "${RECONFIG_DIR}/reconfig_update_for_maintenance.wiki" | sed 's/^/        /'
>  echo "  * Reconfig completed. Directory '${RECONFIG_DIR}' contains artefacts from reconfig process."
> +
> +# Manage foopies after everything else.
> +if merge_to_production || [ "${FORCE_RECONFIG}" == '1' ]; then

This calls a second merge to production, which means that if there changes since the previous merge before the masters were updated, the foopies and the masters could now be potentially out-of-sync.

Better would be:

if [ -f "${RECONFIG_DIR}/pending_changes" ] || [ "${FORCE_RECONFIG}" == '1' ]; then

I'll raise a separate bug for this.

Pete Moore [:pmoore][:pete]

Updated

•

10 years ago

Updated

•

7 years ago

Component: Tools → General

Run manage_foopies.py after reconfig is completed. 10 years ago Chris Cooper [:coop] (he/him) 5.30 KB, patch	sbruno : review+ coop : checked-in+	Details \| Diff \| Splinter Review
manage_foopies-1408645327.log 10 years ago Jordan Lund (:jlund) 4.05 KB, text/x-log		Details

Bugzilla

Quick Search

Run manage_foopies.py after reconfig is completed in end_to_end_reconfig.sh

Categories

(Release Engineering :: General, defect, P2)

Tracking

(Not tracked)

People

(Reporter: coop, Assigned: coop)

References

Details

Crash Data

Security

(public)

User Story

Attachments

(2 files)

Description

Updated

Comment 1

Comment 2

Comment 3

Updated

Comment 4

Comment 5

Comment 6

Updated

Updated

Attachment

General

Description

File Name

Content Type