The default bug view has changed. See this FAQ.

Run manage_foopies.py after reconfig is completed in end_to_end_reconfig.sh

RESOLVED FIXED

Status

Release Engineering
Tools
P2
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: coop, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 years ago
I frequently hit bug 1039982 while running end_to_end_reconfig.sh. Until that bug is fixed, we should move the manage_foopies.py steps to after the reconfig so that the reconfig always completes in its entirety, and we can Ctrl-C out of the foopy update if need be.
(Assignee)

Updated

3 years ago
Status: NEW → ASSIGNED
Priority: -- → P2
(Assignee)

Comment 1

3 years ago
Created attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.
Attachment #8475909 - Flags: review?(sbruno)
Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

It looks ok to me, and I tested this patch with reconfig on 2014-08-20 07:47 PT
Attachment #8475909 - Flags: review?(sbruno) → review+
(Assignee)

Comment 3

3 years ago
Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/tools/rev/a57f32992270
Attachment #8475909 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Comment 4

3 years ago
looks like this worked. reconfig finished fine but foopy part hung so I ctrl-c'd it.

output of foopy log for documentation:

jlund@Hastings163:~MOZ/reconfigs
> tail -f manage_foopies-1408555585.log
[foopy97.build.mozilla.org] out: adding manifests
[foopy97.build.mozilla.org] out: (run 'hg update' to get a working copy)
[foopy98.build.mozilla.org] out: adding file changes
[foopy97.build.mozilla.org] out: added 2 changesets with 4 changes to 4 files
[foopy97.build.mozilla.org] out: 4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy99.build.mozilla.org] out:

[foopy99.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
(run 'hg update' to get a working copy)
[foopy97.build.mozilla.org] out: 4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy96.build.mozilla.org] out:

[foopy96.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy98.build.mozilla.org] out:

[foopy98.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy97.build.mozilla.org] out:

[foopy97.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
[OK]   updated foopy99 tools to a57f32992270
[OK]   updated foopy98 tools to a57f32992270
[OK]   updated foopy96 tools to a57f32992270
[OK]   updated foopy97 tools to a57f32992270
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
[foopy64.build.mozilla.org] out:

[foopy64.build.mozilla.org] run: find /builds/tools -name \*.pyc -exec rm {} \;
[OK]   updated foopy52 tools to a57f32992270
[OK]   updated foopy64 tools to a57f32992270
Process PoolWorker-30:
Process PoolWorker-28:
Process PoolWorker-26:
Process PoolWorker-23:
Process PoolWorker-29:
Process PoolWorker-32:
Process PoolWorker-27:
Process PoolWorker-31:
Process PoolWorker-25:
Process PoolWorker-24:
Process PoolWorker-22:
Process PoolWorker-20:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-21:
Process PoolWorker-17:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
Failed to run  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
 update on foopy116
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
    self.run()
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
    task = get()
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    racquire()
    racquire()
    task = get()
    racquire()
    racquire()
    racquire()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    return recv()
KeyboardInterrupt
    racquire()
    racquire()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    racquire()
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
KeyboardInterrupt
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    racquire()
KeyboardInterrupt
    task = get()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 374, in get
    racquire()
KeyboardInterrupt
Traceback (most recent call last):
  File "./manage_foopies.py", line 68, in run_action_on_foopy
    action_func(foopy)
  File "/Users/jlund/devel/mozilla/cleanRepos/tools/buildfarm/maintenance/foopy_fabric.py", line 71, in update
    run('hg pull && hg update -r %s' % revision)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/network.py", line 639, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 1042, in run
    shell_escape=shell_escape)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 911, in _run_command
    stderr=stderr, timeout=timeout)
  File "/Users/jlund/venv/bbot/lib/python2.7/site-packages/fabric/operations.py", line 797, in _execute
    time.sleep(ssh.io_sleep)
KeyboardInterrupt

Traceback (most recent call last):
  File "./manage_foopies.py", line 165, in <module>
    if not result.get():
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 548, in get
    self.wait(timeout)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 543, in wait
    self._cond.wait(timeout)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 339, in wait
    waiter.acquire()
KeyboardInterrupt

^C

Comment 5

3 years ago
Created attachment 8476999 [details]
manage_foopies-1408645327.log

more foopy fails
Comment on attachment 8475909 [details] [diff] [review]
Run manage_foopies.py after reconfig is completed.

Review of attachment 8475909 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/end_to_end_reconfig.sh
@@ +383,5 @@
>  cat "${RECONFIG_DIR}/reconfig_update_for_maintenance.wiki" | sed 's/^/        /'
>  echo "  * Reconfig completed. Directory '${RECONFIG_DIR}' contains artefacts from reconfig process."
> +
> +# Manage foopies after everything else.
> +if merge_to_production || [ "${FORCE_RECONFIG}" == '1' ]; then

This calls a second merge to production, which means that if there changes since the previous merge before the masters were updated, the foopies and the masters could now be potentially out-of-sync.

Better would be:

if [ -f "${RECONFIG_DIR}/pending_changes" ] || [ "${FORCE_RECONFIG}" == '1' ]; then

I'll raise a separate bug for this.
See Also: → bug 1057327
You need to log in before you can comment on or make changes to this bug.