1334566 - Autophone - rca tests start but fail to complete on Beta and Release

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=autophone&exclusion_profile=false&group_state=expanded

https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&filter-searchStr=autophone&exclusion_profile=false&group_state=expanded

Show that rca starts running but fails to complete and fails to mark the job as complete in Treeherder for Beta/52 and Release/51.

This began around merge day:

Mon Jan 23, 6:19:46
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=76101b503191468724851ba294a3d1fe7d2df50f

Tue Jan 24, 3:12:58
https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=480623a791a0077d161378b9e7786a847b0977cd

Note that bug 1332916 began shortly thereafter.

We also have Bug 1331966 which morphed into Bug 1333657.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: bug-1334566-upload.patch

The actual issue was due to the upload directory containing directories which caused s3 to raise an uncaught IOError when it tried to read the directory as a file. This prevented the upload from completing and left the job in a running state.

I introduced find_files(path) into utils.py which uses os.path.walk to find all files contained in the upload directory.

I also added the ability to save a copy of the log file on the local disk if the log level is set to DEBUG. This helped find the issue which was otherwise lost.

I'm r=self on this patch since I want to get it in production and stop the rca tests from failing due to this.

Comment 2

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/80a79c4e93dd37a3a17dc4e9370e1b7d4d2d37e9
deployed 2017-01-28 05:00

Comment 3

[Bob Clary [:bc] (inactive)]

Comment on attachment 8831429 [details] [diff] [review]
bug-1334566-upload.patch

jmaher: If you want to review this after the fact and have me make any changes, I would be happy to do so.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8831429 [details] [diff] [review]
bug-1334566-upload.patch

Review of attachment 8831429 [details] [diff] [review]:
-----------------------------------------------------------------

moar utility functions