1049202 - automationutils fakes mozinfo

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Gregory Szorc [:gps]]

Bug 1038943 added code to automationutils.py to check mozinfo.info['toolkit']. This resulted in a KeyError due to 'toolkit' not being present. https://tbpl.mozilla.org/php/getParsedLog.php?id=45277842&tree=B2g-Inbound&full=1

I downloaded the tests zip archive (https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/b2g-inbound-android/1407273498/fennec-34.0a1.en-US.android-arm.tests.zip) and examined the mozinfo.json file:

$ python -mjson.tool mochitest/mozinfo.json
{
    "appname": "fennec",
    "asan": false,
    "bin_suffix": "",
    "bits": 32,
    "buildapp": "mobile/android",
    "crashreporter": true,
    "datareporting": true,
    "debug": false,
    "healthreport": true,
    "mozconfig": "/builds/slave/b2g-in-and-0000000000000000000/build/.mozconfig",
    "os": "android",
    "processor": "arm",
    "tests_enabled": true,
    "toolkit": "android",
    "topsrcdir": "/builds/slave/b2g-in-and-0000000000000000000/build",
    "wave": true,
    "webm": true
}

Since mozinfo.json doesn't populate info['toolkit'] by default, I suspect the problem is that find_and_update_from_json() isn't being called from automation and/or that the mozinfo.json from the tests zip is not being loaded.

I'm pretty sure this is a regression - and a serious one at that, since automation is no longer respecting our build configs :/

Comment 1

[Gregory Szorc [:gps]]

I'm not sure if this ever worked. We attempt to import mozbuild in the "load mozinfo.json" code path and this will never work in automation since mozbuild is not included as part of tests.zip.

https://hg.mozilla.org/mozilla-central/file/71497ed2e0db/testing/mozbase/mozinfo/mozinfo/mozinfo.py#l152

Comment 2

[Andrew Halberstadt [:ahal]]

That code path is just for running locally. The test harnesses should have logic to find the mozinfo.json in the tests.zip. For example, mochitests do: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#417

Comment 3

[Andrew Halberstadt [:ahal]]

I don't think the problem here is that automation isn't respecting our build configs. If you look at the list of skipped tests just before the traceback, they are correctly skipped because 'toolkit == "android"'. So at the very least manifestparser is receiving the correct mozinfo.info. I'm not sure why automationutils.py isn't.

Comment 4

[Andrew Halberstadt [:ahal]]

Oh, here's your problem..

http://mxr.mozilla.org/mozilla-central/source/build/automationutils.py#22

Comment 5

[(not currently active) Ted Mielczarek]

Yeah, bummer. Wonder if that's still true?

Comment 6

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

When this is fixed https://hg.mozilla.org/integration/b2g-inbound/rev/d419d642fb4e should be reverted.

Comment 7

[Andrew Halberstadt [:ahal]]

I have no idea why mozinfo can't be imported, or if it is still the case. However if we want to continue using this hack, setting toolkit to 'android' would "fix" the immediate problem and let us backout khuey's bustage fix.

(Note: I'm not advocating we do that)

Comment 8

[(not currently active) Ted Mielczarek]

So, last I knew that was a workaround for non-mozharness Android test jobs. I believe that since we've killed tegra tests on trunk we don't have any more of those, and we could probably remove that hack. I'll test that theory.

Comment 9

[(not currently active) Ted Mielczarek]

Try push with that junk removed:
https://tbpl.mozilla.org/?tree=Try&rev=c5dd8d66bcb1

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We only take that path if the import fails so that means the import is failing, no?

Comment 11

[Andrew Halberstadt [:ahal]]

Yeah, but it's possible we aren't bothering to copy mozbase or setup sys.path because it previously wasn't working.

We need Joel or Geoff to chime in here.

Comment 12

[(not currently active) Ted Mielczarek]

Yeah, it's definitely still broken:
https://tbpl.mozilla.org/?tree=Try&rev=c5dd8d66bcb1

"ImportError: No module named mozinfo"ImportError: No module named mozinfo" on the Android 2.3 emulator tests. They're running under mozharness, but it looks like they're not installing the mozbase modules.

Comment 13

[(not currently active) Ted Mielczarek]

From quickly reviewing the mozharness scripts, it looks like PandaTest uses MozbaseMixin:
https://hg.mozilla.org/build/mozharness/file/03a994d2ec21/scripts/android_panda.py

where AndroidEmulatorTest does not:
https://hg.mozilla.org/build/mozharness/file/03a994d2ec21/scripts/android_emulator_unittest.py

I'll file another bug to fix that.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

so it looks as though the panda tests work but only the 2.3 emulator tests fail.  Once that is fixed we could remove this!

Comment 15

[Geoff Brown [:gbrown]]

I think we are on the right track here. I just put a patch up for review in bug 1049739.

Comment 16

[(not currently active) Ted Mielczarek]

Attachment: remove automationutils mozinfo workaround for non-mozharness Android tests

Might as well get this reviewed then. This can't land until gbrown's mozharness patch makes it into production.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8470057 [details] [diff] [review]
remove automationutils mozinfo workaround for non-mozharness Android tests

Review of attachment 8470057 [details] [diff] [review]:
-----------------------------------------------------------------

can you verify this on try server?  Are there concerns with b2g here?

Comment 18

[(not currently active) Ted Mielczarek]

I pushed it to try this morning after I saw gbrown's patch go into production:
https://tbpl.mozilla.org/?tree=Try&rev=d3d3c2229522

Pretty green!

Comment 19

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ce924184d6b1

Comment 20

[(not currently active) Ted Mielczarek]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #6)
> When this is fixed
> https://hg.mozilla.org/integration/b2g-inbound/rev/d419d642fb4e should be
> reverted.

I forgot to do this, feel free to do that anytime.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ce924184d6b1