Closed Bug 1039982 Opened 10 years ago Closed 8 years ago

manage_foopies.py should not hang in case of problems

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: sbruno, Unassigned)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2735] ) 

end_to_end_reconfig.sh currently halts in case of issues while updating foopies.
Changes should be made to manage_foopies.sh (called in end_to_end_reconfig.sh) so that end to end reconfig does not halt in case of issues with foopies, but reports what foopies were problematic so that further troubleshooting is possible on them after reconfig.
Ideally, attempts to update foopies should be run with some timeout mechanism to make sure execution can be completed in a reasonable amount of time also in case of issues.
You know we could also background the foopy updates - so that the foopy updates happen in parallel to the buildbot master updates (and these are written to separate log files already)... This would also potentially bring down the
Summary: end_to_end_reconfig.sh should not halt in case of foopies issues, and report what foopies were problematic → manage_foopies.py should not hang in case of problems
... wall time. That said, it might not help, since we are already bottle-necked by the number of active ssh connections we can manage at a time - so running manage_foopies.py and manage_masters.py in parallel probably means we would need to reduce parallelisation flag (-j) on each of them anyway.

Maybe it makes sense instead for foopies to have production branch checked out, and to have a cron to update (much like we have with mozharness). In other words, foopies pull changes in, rather than us pushing them out. This is more scalable. After all, it would just be an hg pull -u on a cron on the foopies. Then we can ditch manage_foopies.py altogether! :)
Will raise a separate bug for this now...
Ok scope creep! I raised a separate bug, to ditch reconfigs altogether! ;)

See bug 1040013 ...
I think it's still useful to be able to force an update on the foopies. I think more logging, or even just output of the current state, would help debug this particular issue.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2728]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2728] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2735]
Foopies are no longer a thing...
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.