1538770 - Linux opt format and tidy bustages with bash: ./mach: Argument list too long

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect, P2)

Description

[Cosmin Sabou [:CosminS]]

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235880733&repo=mozilla-inbound&lineNumber=7715

Raw log: https://taskcluster-artifacts.net/Yz-TrIyuTZy0mqACp_Rn0g/0/public/logs/live_backing.log

Th push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&revision=1b71cd5385b56649bea75400dd008e52a1a6d77d&selectedJob=235880689

Comment 1

[Sylvestre Ledru [:Sylvestre]]

When the command line is too long, we should split into several executions.
(this is what we are doing for clang-format)

Comment 2

[Nathan Froyd [:froydnj]]

Does clang-format already accept @resource-file arguments? mach whatever could accept those too.

Comment 3

[Bastien Abadie [:bastien]]

This error happens because both ./mach clang-format and ./mach static-analysis check CI tasks need the list of files modified in the try push.
This is done in taskcluster/scripts/misc/source-test-clang-setup.sh by creating an environment variable TRY_MODIFIED_FILES, and you can see the export line in the log file from this bug.

We could do this in the Python code directly to avoid creating huge environment variables by adding a mercurial revision option. Andrew, what do you think of this solution ? I can implement it.

Comment 4

[Andrew Halberstadt [:ahal]]

You would need to use all files touched in the push, not just a single revision. Luckily we have a function that gives you that already:
https://searchfox.org/mozilla-central/source/python/mozversioncontrol/mozversioncontrol/__init__.py#309

Another alternative would be to modify the mach_commands.py itself to read that environment variable instead of passing it in on the task command line.

Comment 5

[Bastien Abadie [:bastien]]

Attachment: Bug 1538770 - Replace TRY_MODIFIED_FILES by mozversioncontrol usage, r=ahal

Comment 6

[Cosmin Sabou [:CosminS]]

There is also this failure now on trees: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=236298018&repo=mozilla-inbound&lineNumber=463

[task 2019-03-27T05:22:06.928Z] ++ cd /builds/worker/checkouts/gecko
[task 2019-03-27T05:22:06.928Z] +++ hg status --rev 'max(ancestors(tip) and public())' -q --no-status
[task 2019-03-27T05:22:09.514Z] ++ export TRY_MODIFIED_FILES=
[task 2019-03-27T05:22:09.514Z] ++ TRY_MODIFIED_FILES=
[task 2019-03-27T05:22:09.514Z] + cd /builds/worker/checkouts/gecko
[task 2019-03-27T05:22:09.514Z] + ./mach --log-no-times clang-format --path --output /builds/worker/clang-format.json --format json
[task 2019-03-27T05:22:09.812Z] usage: mach [global arguments] clang-format [command arguments]
[task 2019-03-27T05:22:09.812Z] mach: error: argument --path/-p: expected at least one argument
[taskcluster 2019-03-27 05:22:10.368Z] === Task Finished ===
[taskcluster 2019-03-27 05:22:10.481Z] Artifact "public/code-review/clang-format.json" not found at "/builds/worker/clang-format.json"
[taskcluster 2019-03-27 05:22:11.272Z] Unsuccessful task run with exit code: 2 completed in 338.428 seconds

Comment 7

[Intermittent Failures Robot]

32 failures in 3910 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-inbound: 25
• mozilla-central: 7

Platform breakdown:

• linux64: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1538770&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 8

[Intermittent Failures Robot]

31 failures in 398 pushes (0.078 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 5
• autoland: 25
• mozilla-inbound: 1

Platform breakdown:

• linux64: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1538770&startday=2019-04-01&endday=2019-04-01&tree=all

Comment 9

[Bastien Abadie [:bastien]]

Attachment: Bug 1538770 - Remove _conv_to_abspath from mach static analysis, r=ahal

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/813b970cd40b
Replace TRY_MODIFIED_FILES by mozversioncontrol usage, r=ahal

Comment 11

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/813b970cd40b

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a001a531c01e
Remove _conv_to_abspath from mach static analysis, r=ahal

Comment 13

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/a001a531c01e

Comment 14

[Intermittent Failures Robot]

39 failures in 2843 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 8
• autoland: 25
• mozilla-inbound: 6

Platform breakdown:

• linux64: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1538770&startday=2019-04-01&endday=2019-04-07&tree=all