1307095 - Logs for atmo v1 scheduled jobs are missing

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Comment 1

[Frank Bertsch [:frank]]

Attachment: foo.20161004183138.log.gz

Again, cannot reproduce. Tried with TxP Event-based Install Counts New Bucket.ipynb, which both failed and was missing the log files in S3. Successfully ran, and the log files are in S3.

Location: s3://telemetry-public-analysis-2/foo/logs

Successful Job: https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-3PGK7BLKBENWU
Failed Job: https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-36T7D4OV0EGBW

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

Besides the number of machines in the clusters the rest looks the same to me. I noticed though that the elapsed time is 2 hours & 7 minutes for the failed job; maybe the job has a timeout of 2 hours?

Sunah, could you please tell us what the configuration of the timeout for the "txp_install_counts" job is?

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

I confirm that the timeout for this job is set to 2h. Sunah, this job has been failing for a while, is it still being used?

Comment 4

[Frank Bertsch [:frank]]

So I just realized we can tell which ones are failed and which are timed out by looking at the step status - it's cancelled for timed out, and failed for failure. I have a hunch that all of these timed out, but I'm checking into it.

Comment 5

[bugzilla]

I was keeping it around to verify the real-time count, but it's not really necessary (and I can just run it manually for verification)

Comment 6

[Frank Bertsch [:frank]]

Can confirm, these are all cancelled jobs that exceed their limits. The issue is that when a job times out, the cluster automatically shuts down, which does not give the step a chance to move the logs over.

Comment 7

[Frank Bertsch [:frank]]

Tried fixing this by catching the SIGINT (and all other varieties) signal and uploading the files then, but unfortunately it doesn't work. Ex (at beginning of batch.sh):


upload_log ()
{
    cd ..
    gzip "$LOG"
    aws s3 cp "${LOG}.gz" "$S3_BASE/logs/$(basename "$LOG").gz" --content-type "text/plain" --content-encoding gzip
}

trap upload_log SIGINT
trap upload_log SIGTERM 
trap upload_log SIGKILL
trap upload_log EXIT

Comment 8

[Frank Bertsch [:frank]]

I have a tentative fix that requires adding the --job-name and --data-bucket parameters to telemetry.sh, and copies over the log files before shutting down. Still testing to ensure that it works. Note that this will also require minor changes to telemetry-analysis-service.

Comment 9

[Frank Bertsch [:frank]]

We've decided that we won't fix this, and will just use the logs provided by bug 1307528.