Try to use the same mozharness config files for tbpl and docker images

RESOLVED INVALID

Status

Taskcluster
General
RESOLVED INVALID
4 years ago
3 years ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

Details

(Assignee)

Description

4 years ago
The mozharness config files used for b2g-desktop and emulator tests in docker images are slightly different from the standard ones in the mozharness repo. It would be simpler and less error prone if we could just use the ones from the mozharness repo.
(Assignee)

Comment 1

4 years ago
$ diff b2g-desktop-config.py ~/build/mozharness/configs/b2g/desktop_automation_config.py
6a7,13
>     "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> 
>     "exes": {
>         'python': '/tools/buildbot/bin/python',
>         'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
>         'tooltool.py': "/tools/tooltool.py",
>     },
8a16
>         "http://pypi.pvt.build.mozilla.org/pub",
12a21,22
>     "buildbot_json_path": "buildprops.json",
> 
22a33,36
>     "default_blob_upload_servers": [
>         "https://blobupload.elasticbeanstalk.com",
>     ],
>     "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),
23a38
>     # testsuite options
28c43,44
<    "in_tree_config": "config/mozharness/b2g_desktop_config.py",
---
>     # test harness options are located in the gecko tree
>     "in_tree_config": "config/mozharness/b2g_desktop_config.py",
(Assignee)

Comment 2

4 years ago
$ diff emulator_automation_config.py ~/build/mozharness/configs/b2g/emulator_automation_config.py
12,13c12,14
<         'python': '/usr/bin/python',
<         'tooltool.py': "mozharness/mozharness/mozilla/tooltool.py",
---
>         'python': '/tools/buildbot/bin/python',
>         'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
>         'tooltool.py': "/tools/tooltool.py",
44c45,46
<         "xpcshell": "runtestsb2g.py"
---
>         "xpcshell": "runtestsb2g.py",
>         "cppunittest": "remotecppunittests.py"
(Assignee)

Comment 3

4 years ago
The only significant difference is the /tools references:

>         'python': '/tools/buildbot/bin/python',
>         'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
>         'tooltool.py': "/tools/tooltool.py",
(Assignee)

Comment 4

4 years ago
It seems that on existing test slaves, /tools is populated by puppet, which I am not very familiar with. It would be nice to use the same /tools content in the docker image, but it looks like it would be difficult to use puppet to populate /tools in the docker image....perhaps not worth the effort when we just need a few files.

:armenzg -- Do you have any suggestions?
Flags: needinfo?(armenzg)
I don't know how docker works.
Perhaps, some files that live on the tools repo should be moved to the mozharness repo.
We have done that in the past with the file mouse_and_screen_resolution.py [1][2]

Are you trying to remove the dependency on a tools repo being up-to-date?
If I understand correctly, there is an initiative working towards having a tools repo always checked out and up-to-date.

[1] http://hg.mozilla.org/build/tools/file/default/scripts/support/mouse_and_screen_resolution.py
[2] http://hg.mozilla.org/build/mozharness/file/136cf767a41f/external_tools/mouse_and_screen_resolution.py
Flags: needinfo?(armenzg)
(Assignee)

Comment 6

4 years ago
I talked to Armen this afternoon regarding Comment 5. There is not a single place in hg that we can clone to get the /tools contents found on test slaves (we could copy tooltool.py out of hg, but not, for instance, /tools/buildbot/bin/python, /tools/misc-python/virtualenv.py, /tools/android-sdk18/platform-tools/adb, all of which are referenced from mozharness configs).

:rail, :coop -- Can you suggest a way forward here?

I am trying to set up a proper b2g test environment in a vm (a docker image). Currently we take a Ubuntu image, add some packages, and clone mozharness. That gets us 99% of the way there, but mozharness configs have references to files in /tools. I can use custom mozharness configs to work around those paths, but that seems error prone and maintenance intensive. I'd prefer an approach that gives me the same /tools contents in my vm as on test slaves.

Do I want to use puppet? Something like http://hg.mozilla.org/build/cloud-tools/annotate/c10ba9d1f8b0/scripts/aws_create_instance.py#l134 ?

Armen thought "runner" might be useful -- is that an option?
Flags: needinfo?(rail)
Flags: needinfo?(coop)
(Assignee)

Comment 7

4 years ago
See the discussion in https://github.com/taskcluster/gecko-images/pull/7/files. It sounds like taskcluster should use its own customized mozharness configs and scripts -- no need to make things the same.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Flags: needinfo?(rail)
Flags: needinfo?(coop)
Resolution: --- → INVALID
(Assignee)

Updated

4 years ago
Assignee: nobody → gbrown
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.