1577883 - mach - enable python3 for release-history

Status: RESOLVED DUPLICATE of bug 1639151

(Firefox Build System :: General, task)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Command:
./mach release-history --product=Firefox -b mozilla-central

Expected outcome:

(python35_development) egao@egao-44538:~/src/mozilla-central$ ./mach release-history --product=Firefox -b mozilla-central
release_history:
  Darwin_x86_64-gcc3-u-i386-x86_64:
    ach:
      target.partial-1.mar:
        buildid: '20190830093857'
        mar_url: https://archive.mozilla.org/pub/firefox/nightly/2019/08/2019-08-30-09-38-57-mozilla-central-l10n/firefox-70.0a1.ach.mac.complete.mar
      target.partial-2.mar:
        buildid: '20190829214656'
        mar_url: https://archive.mozilla.org/pub/firefox/nightly/2019/08/2019-08-29-21-46-56-mozilla-central-l10n/firefox-70.0a1.ach.mac.complete.mar
...

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1577883 - enable python3 for mach release-history

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

The patch above doesn't work unfortunately.

The actual command ./mach release-history --product=Firefox -b mozilla-central executes to completion (minus the last part in python/mozbuild/mozbuild/telemetry.py but that is handled in the patch for bug 1577603).

The issue is when ./mach taskgraph full is run, to verify the integrity of the taskgraph, the following is seen:

(python35_development) egao@egao-44538:~/src/mozilla-central$ ./mach taskgraph full
 0:00.92 Loading graph configuration.
 0:01.02 Starting new HTTPS connection (1): index.taskcluster.net
 0:02.32 Loading kinds
 0:03.28 Generating full task set
 0:03.36 Error loading tasks for kind addon:
Traceback (most recent call last):
  File "/Users/egao/src/mozilla-central/taskcluster/mach_commands.py", line 379, in show_taskgraph
    tg = getattr(tgg, graph_attr)
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/generator.py", line 151, in full_task_graph
    return self._run_until('full_task_graph')
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/generator.py", line 351, in _run_until
    k, v = self._run.next()
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/generator.py", line 263, in _run
    new_tasks = kind.load_tasks(parameters, list(all_tasks.values()))
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/generator.py", line 77, in load_tasks
    for task_dict in transforms(trans_config, inputs)]
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1981, in check_run_task_caches
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1922, in check_task_dependencies
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1910, in check_task_identifiers
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1893, in chain_of_trust
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1735, in build_task
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1705, in add_index_routes
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1441, in validate
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1418, in task_name_from_label
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1378, in set_defaults
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/task.py", line 1357, in set_implementation
    for task in tasks:
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/job/__init__.py", line 288, in make_task_description
    configure_taskdesc_for_run(config, job, taskdesc, job['worker']['implementation'])
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/transforms/job/__init__.py", line 347, in configure_taskdesc_for_run
    job['run']['using'], worker_implementation, job['label']))
  File "/Users/egao/src/mozilla-central/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
    raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
Exception: In job.run using u'mach'/u'docker-worker' for job u'addon-tps-xpi':
expected a tuple for dictionary value @ data[u'mach']
{u'comm-checkout': False,
 u'mach': u'tps-build --dest tps-out',
 u'sparse-profile': u'tps',
 u'using': u'mach',
 u'workdir': u'/builds/worker'}
(python35_development) egao@egao-44538:~/src/mozilla-central$

I suspect this is due to the fact that our voluptuous library is vendored.

In order to vendor the python 3 version, I think ./mach vendor needs to support python3, but in order to support python3 it must first have some work done to it to support different python versions.

Since that's a rather large task, it will be handled in a separate bug parallel to this bug and this bug will be revisited when the necessary components are in place.

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

Removing assignment as I do not intend to work on this for some time.

Comment 4

[Hamzah Akhtar]

Attachment: Bug 1577883 - release-history mach command

Comment 5

[Hamzah Akhtar]

Attachment: Bug 1577883 - web-platform-tests-update

Depends on D77619