1522762 - log the occ dsc run in real time

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

occ runs a number of bootstrapping functions that are logged in real time, then occ triggers the dsc run and the instance goes quiet for several minutes while the dsc run takes place. when it completes it releases a transcript file which is sent to papertrail and shown with a program heading of dsc-run.

sometimes, if something goes fatally wrong in the dsc run, the dsc script terminates and no further logs are received from either occ or dsc.

patch the dsc script xDynamicConfig, so that it logs in real time and makes it easier to see and debug where in the dsc run a fatal exception has occurred.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=17e8c741c4fe6498d4f421666726118825c18694

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: GitHub Pull Request

this pr adds:

• event logging to SetScript components within xDynamicConfig
• nxlog configurations to forward those logs to papertrail

most setscript component logic is now wrapped in a try catch all like so:

try {
  do_something_that_might_succeed_or_fail
  write_success_to_eventlog
}
catch {
  write_failure_to_eventlog
  throw_or_reattempt
}

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

merged: https://github.com/mozilla-releng/OpenCloudConfig/commit/c61c3805f06e179a69c423c64a8886cb1c037b42