1515261 - TypeError on mach fuzzy

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

When running | mach try fuzzy --no-artifact |, I get the following error and traceback:

TypeError: environment can only contain strings

  File "c:\mozilla-source\mozilla-central\tools/tryselect/mach_commands.py", line 150, in try_fuzzy
    return run_fuzzy_try(**kwargs)
  File "c:\mozilla-source\mozilla-central\tools\tryselect\selectors\fuzzy.py", line 214, in run_fuzzy_try
    tg = generate_tasks(parameters, full, root=vcs.path)
  File "c:\mozilla-source\mozilla-central\tools\tryselect\tasks.py", line 99, in generate_tasks
    tg = getattr(TaskGraphGenerator(root_dir=root, parameters=params), attr)
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\generator.py", line 169, in target_task_graph
    return self._run_until('target_task_graph')
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\generator.py", line 342, in _run_until
    k, v = self._run.next()
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\generator.py", line 263, in _run
    new_tasks = kind.load_tasks(parameters, list(all_tasks.values()))
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\generator.py", line 76, in load_tasks
    for task_dict in transforms(trans_config, inputs)]
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1926, in check_run_task_caches
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1872, in check_task_dependencies
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1860, in check_task_identifiers
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1843, in chain_of_trust
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1689, in build_task
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1659, in add_index_routes
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1477, in validate
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1454, in task_name_from_label
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\task.py", line 1413, in set_defaults
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\coalesce.py", line 29, in enable_coalescing
    for job in jobs:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\job\__init__.py", line 226, in make_task_description
    for job in jobs:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\job\__init__.py", line 154, in use_fetches
    for job in jobs:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\job\__init__.py", line 117, in rewrite_when_to_optimization
    for job in jobs:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\base.py", line 75, in __call__
    for task in tasks:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 1080, in make_job_description
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 1027, in set_worker_type
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 1013, in single_stylo_traversal_tests
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 1002, in set_test_type
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 994, in set_tag
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 982, in set_profile
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 971, in set_retry_exit_status
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 954, in enable_webrender
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 939, in allow_software_gl_layers
    for test in tests:
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\transforms\tests.py", line 906, in split_chunks
    test['test-name'])
  File "c:\mozilla-source\mozilla-central\python/mozbuild\mozbuild\util.py", line 935, in __call__
    self[args] = self.func(*args)
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\util\perfile.py", line 58, in perfile_number_of_chunks
    changed_files.update(files_changed.get_locally_changed_files(GECKO))
  File "c:\mozilla-source\mozilla-central\python/mozbuild\mozbuild\util.py", line 935, in __call__
    self[args] = self.func(*args)
  File "c:\mozilla-source\mozilla-central\taskcluster\taskgraph\files_changed.py", line 88, in get_locally_changed_files
    return set(vcs.get_outgoing_files('AM'))
  File "c:\mozilla-source\mozilla-central\python/mozversioncontrol\mozversioncontrol\__init__.py", line 310, in get_outgoing_files
    '--template', template, upstream, return_codes=(1,)).split()
  File "c:\mozilla-source\mozilla-central\python/mozversioncontrol\mozversioncontrol\__init__.py", line 256, in _run
    return super(HgRepository, self)._run(*args, **runargs)
  File "c:\mozilla-source\mozilla-central\python/mozversioncontrol\mozversioncontrol\__init__.py", line 92, in _run
    universal_newlines=True)
  File "c:\mozilla-build\python\lib\subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "c:\mozilla-build\python\lib\subprocess.py", line 394, in __init__
    errread, errwrite)
  File "c:\mozilla-build\python\lib\subprocess.py", line 644, in _execute_child
    startupinfo)

Comment 1

[Ionuț Goldan [:igoldan]]

This gets a little bit in the way of checking/confirming performance improvements, such as this one [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1515002#c2

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: temp.patch

Does this patch fix it for you by any chance?

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
> Created attachment 9032412 [details] [diff] [review]
> temp.patch
> 
> Does this patch fix it for you by any chance?

Yes, it does. Thank you!

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1515261 - [mozversioncontrol] Fix unicode env string on Windows, r?sheehan

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5f6139ef7c2c
[mozversioncontrol] Fix unicode env string on Windows, r=sheehan

Comment 6

[Arthur Iakab [arthur_iakab]]

Backed out changeset 5f6139ef7c2c (bug 1515261) for causing Windows build bustages CLOSED TREE

Backout revision https://hg.mozilla.org/integration/autoland/rev/8e2d7fcc07f51574c9b48f3641cdae95ae387197

Failed push https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=5f6139ef7c2cc63904a73d6fe8b52031a295784d

Failure log https://treeherder.mozilla.org/logviewer.html#?job_id=219988689&repo=autoland

:sheehan Pleaste take a look at those bustages. Thanks

Comment 7

[Connor Sheehan [:sheehan]]

It looks like the issue here is that on Windows under Py2, some of the environment
variable key/values are `str` (bytes/ascii) and some are `unicode`. So we can't
encode every value to `utf8` as we tried to in this bug and that is causing the 
failure. There is a more succinct description at https://github.com/pallets/werkzeug/pull/1320,
and a fix at https://github.com/pallets/werkzeug/pull/1321

Comment 8

[Andrew Halberstadt [:ahal]]

Latest version fixes the build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=86463ad5452faf83c5e7c9e218d6e61025e4bdec

Ionuț, would you mind re-testing the latest version from phabricator to make sure it still fixes your issue?

Comment 9

[Andrew Halberstadt [:ahal]]

Oops sorry, I forgot to re-open the revision so my re-push never went through. It's up-to-date now though.

Comment 10

[Ionuț Goldan [:igoldan]]

(In reply to Andrew Halberstadt [:ahal] from comment #8)

Latest version fixes the build:
https://treeherder.mozilla.org/#/
jobs?repo=try&revision=86463ad5452faf83c5e7c9e218d6e61025e4bdec

Ionuț, would you mind re-testing the latest version from phabricator to make
sure it still fixes your issue?

I re tested and it works flawlessly.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4ca7829ff4ed
[mozversioncontrol] Fix unicode env string on Windows, r=sheehan

Comment 12

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/4ca7829ff4ed