1315441 - KeyError: 'aws_instance_type' on OS X

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1315041 's call im['aws_instance_type'] fails on OS X:

https://treeherder.mozilla.org/logviewer.html#?job_id=6179040&repo=autoland

02:00:45     INFO - Running post-run listener: _resource_record_post_run
02:00:45  WARNING - error reading instance_metadata: Traceback (most recent call last):
02:00:45  WARNING -   File "/builds/slave/test/scripts/mozharness/base/python.py", line 519, in perfherder_resource_options
02:00:45  WARNING -     instance = im['aws_instance_type'].encode('ascii')
02:00:45  WARNING - KeyError: 'aws_instance_type'

Comment 1

[Gregory Szorc [:gps]]

Sometimes I really dislike Treeherder's overzealous desire to turn harmless warnings into something that looks bad.

Anyway, we're able to load /etc/instance_metadata.json on this machine. However, it doesn't have an "aws_instance_type" key. The plot is further thickened by https://hg.mozilla.org/build/puppet/file/79383527e18d/modules/instance_metadata/files/instance_metadata.py. AFAICT that is the only file that produces instance_metadata.json. And that script always writes out a "aws_instance_type" key.

I suspect some other process is producing instance_metadata.json on this machine class. But I have no idea where to look.

rail: could you please either point me in the direction of the code that produces instance_metadata.json on Macs or paste an example file so I know what data there is to extract?

Comment 2

[Rail Aliiev [:rail]]

I think it's https://dxr.mozilla.org/build-central/source/puppet/modules/instance_metadata/manifests/init.pp#41-89 that installs and runs it, which is enabled only on aws machines: https://dxr.mozilla.org/build-central/source/puppet/modules/instance_metadata/manifests/init.pp#19 and OSX gets https://hg.mozilla.org/build/puppet/annotate/5cf1582da0f8/modules/instance_metadata/manifests/init.pp#l54

Comment 3

[Gregory Szorc [:gps]]

Ahh, I missed the empty JSON file ("{}") in puppet. I'll update the mozharness Python to handle this case.

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1315441 - Handle missing aws_instance_type key;

The previous code threw and logged an exception. TreeHerder found
this (harmless) warning and reported it as something more dangerous.

So handle the missing key without raising.

Review commit: https://reviewboard.mozilla.org/r/91214/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/91214/

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8808406 [details]
Bug 1315441 - Handle missing aws_instance_type key;

https://reviewboard.mozilla.org/r/91214/#review93492

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/16568d2ed7cd
Handle missing aws_instance_type key; r=ted

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/16568d2ed7cd