Closed Bug 1315988 Opened 3 years ago Closed 3 years ago

Debug WebRTC Autophone jobs haven't run since some time after October 31, 2016

Categories

(Testing :: Autophone, defect)

Version 3
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dminor, Assigned: bc)

References

Details

Attachments

(1 file)

This [1] seems to be last push where the Mw and Cw jobs ran on central. They are also missing from Try [2]. Maybe a hardware problem?

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=944cb0fd05526894fcd90fbe7d1e625ee53cd73d

[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b1efc0a8d835687772571a5688a067dcafab06d
Here is a try push where they ran from Oct 31: https://treeherder.mozilla.org/#/jobs?repo=try&revision=57d62bdd4d041f624fbca424a350fbb7c0e6d284.

On a subsequent push on Nov 1: they are missing: 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5e327dbc4cf9a4dd193843388cd2675b790a47ab
Summary: Debug WebRTC Autophone jobs haven't run since October 27, 2016 → Debug WebRTC Autophone jobs haven't run since some time after October 31, 2016
May be related to the work in bug 1313779 which I hadn't deployed. I've shut everyone down, updated ubuntu, pulled the patch and restarted. If this doesn't fix it, my next step will be to enable debug level logging and see what that can show us.
It doesn't appear we are seeing the pulse messages for exchange/taskcluster-queue/v1/task-completed. jlund: Did anything change about how we should consume taskcluster messages around Oct 31?
Flags: needinfo?(jlund)
jonasfj is listed as the queue maintainer and would probably be the most appropriate person to ask.

jonasfj: It appears I am still receiving messages from this queue for debug builds on mozilla-aurora, mozilla-beta and mozilla-release but not on any "trunk" repos like mozilla-central, mozilla-inbound, or autoland. Do you have an idea about what has changed and what I need to do to support all repos?
Flags: needinfo?(jopsen)
@bc,
For the things running on taskcluster: (following Comment 1, artifact from decision task)
https://public-artifacts.taskcluster.net/IR6yui54QgiGMiEbCn7Jfg/0/public/label-to-taskid.json
(See no change in what is scheduled on taskcluster)

Everything is defined in-tree.

But:
> Android 4.2 API15+ debug	A[tier 2]   Mw 
> Android 4.4 API15+ debug	A[tier 2]   Mw 
> Android 5.1 API15+ debug	A[tier 2]   Mw 
> Android 6.0 API15+ debug	A[tier 2]   Mw  Mw 

Doesn't appear to be taskcluster tasks. I think they are scheduled in buildbot.
By that logic you won't see messages about them in exchange/taskcluster-queue/v1/task-completed.

However, these appear to be in TC and present on both treeherder links from Comment 1.
> Android 4.0 API15+ opt	B tc[tier 2]   B 
> Android 4.0 API15+ debug	tc   B
Flags: needinfo?(jopsen)
jonasfj: The A[...] jobs are Autophone jobs that are initiated by Autophone itself in response to Pulse messages received from the pulse exchange exchange/taskcluster-queue/v1/task-completed.

For example, I received the following message for a debug build on try:

2016-11-09 11:08:16,658|18885|PulseMonitorThread|root|DEBUG|handle_message: data: {u'status': {u'workerType': u'android-api-15', u'taskGroupId': u'ANp_k2JiRyueY6uYEawbcw', u'runs': [{u'scheduled': u'2016-11-09T18:24:24.681Z', u'reasonCreated': u'scheduled', u'takenUntil': u'2016-11-09T19:16:49.137Z', u'started': u'2016-11-09T18:26:01.779Z', u'workerId': u'i-016322fb8f6292140', u'reasonResolved': u'completed', u'workerGroup': u'us-west-1c', u'state': u'completed', u'runId': 0, u'resolved': u'2016-11-09T19:08:16.166Z'}], u'expires': u'2016-11-23T18:24:23.221Z', u'retriesLeft': 5, u'state': u'completed', u'schedulerId': u'gecko-level-1', u'deadline': u'2016-11-10T18:24:23.221Z', u'taskId': u'RywTmU-cT9245vWrjNQm9w', u'provisionerId': u'aws-provisioner-v1'}, u'workerGroup': u'us-west-1c', u'version': 1, u'runId': 0, u'workerId': u'i-016322fb8f6292140'}, message: <kombu.transport.pyamqp.Message object at 0x7f4e7e7e23e0>

Which corresponds to the task https://tools.taskcluster.net/task-inspector/#RywTmU-cT9245vWrjNQm9w/

I do not appear to be getting these task-completed messages for trunk related repos however.
Found it. The worker type was redefined in bug 1307771. We depended on the old form of the name which included android and the api level.
Assignee: nobody → bob
Blocks: 1307771
Status: NEW → ASSIGNED
Flags: needinfo?(jlund)
Comment on attachment 8809895 [details] [diff] [review]
bug-1315988-v1.patch

switching to jmaher since gbrown is probably wearing a poppy today.
Attachment #8809895 - Flags: review?(gbrown) → review?(jmaher)
Comment on attachment 8809895 [details] [diff] [review]
bug-1315988-v1.patch

Review of attachment 8809895 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!
Attachment #8809895 - Flags: review?(jmaher) → review+
deployed ~2016-11-13 22:20 PST.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.