1455473 - mozharness should use log level ERROR for errors reported by Cargo / Rust

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, enhancement)

Description

[Michael Shal [:mshal]]

We have a few intermittent bugs recorded in the Firefox Build System component that only show the cargo make targets as the error messages. The actual errors are reported by cargo/rust earlier in the build, but since they aren't flagged by treeherder, we end up with a bunch of unrelated failures in the same intermittent bug. For example, bug 1428410 has some failures that look similar in the treeherder error log:

----
19:12:05 INFO - mozmake.EXE[2]: *** [force-cargo-test-run] Error 101
19:12:05 INFO - mozmake.EXE[1]: *** [toolkit/library/rust/check] Error 2
19:12:05 INFO - mozmake.EXE: *** [check] Error 2
19:12:05 ERROR - Return code: 2
19:12:05 ERROR - 'mach build check' did not run successfully. Please check log for errors.
----

These error messages are just from the make targets bubbling up, and appear if anything in the cargo test run fail, regardless of what the failure is.

The actual error messages differ:

1)
error: failed to run custom build command for `style v0.0.1 (file:///Z:/build/build/src/servo/components/style)`
19:12:05     INFO - process didn't exit successfully: `z:/build/build/src/obj-firefox/toolkit/library\release\build\style-e8536e0d0e4052f2\build-script-build` (exit code: 101)
...
thread 'main' panicked at 'Can't find python (tried python2.7.exe, python27.exe, and python.exe)! Try fixing PATH or setting the PYTHON env var', servo\components\style\build.rs:44:5

2)
error: couldn't create a temp dir: The system cannot find the path specified. (os error 3)
error: Could not compile `cfg-if`.

It would be helpful if treeherder would recognize these messages so that we could classify the underlying errors better.

I've marked this as blocking the related intermittent bugs in case it helps to look at other logs.

Comment 1

[Ed Morley [:emorley]]

Hi! I agree it would be good to get these line appearing in the error summary.

We're no longer accepting regex changes in Treeherder for cases where the test harness/runner should be using appropriate log line prefixes, so in this case mozharness/whatever is wrapping this command will need changing to make:

19:12:05 INFO - 

...be output as:

19:12:05 ERROR - 

If it's mozharness that's producing that line, I think the log level determination happens via this file:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/errors.py

Comment 3

[Ed Morley [:emorley]]

Would it be possible to have someone look into this and the other `mozharness/base/errors.py` bugs at some point? :-)
(eg https://bugzilla.mozilla.org/showdependencytree.cgi?id=778688&hide_resolved=1)

Comment 4

[Kartikaya Gupta (email:[e-mail])]

I wrote a patch to add the things to errors.py, but a try push [1] doesn't seem to pick it up. Not sure if this needs deploying elsewhere for it to work, or if I just did it wrong.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=e9749156323c1e1592e998deecb75cc54774bf81

Comment 5

[Jordan Lund (:jlund)]

(In reply to Ed Morley [:emorley] from comment #3)
> Would it be possible to have someone look into this and the other
> `mozharness/base/errors.py` bugs at some point? :-)
> (eg
> https://bugzilla.mozilla.org/showdependencytree.
> cgi?id=778688&hide_resolved=1)

Sure. It's been years since I wrote the mozharness build wrapper and probably that long since it has been touched much but I'll put I'll try to help.

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #4)
> I wrote a patch to add the things to errors.py, but a try push [1] doesn't
> seem to pick it up. Not sure if this needs deploying elsewhere for it to
> work, or if I just did it wrong.
> 
> [1]
> https://treeherder.mozilla.org/#/
> jobs?repo=try&group_state=expanded&revision=e9749156323c1e1592e998deecb75cc54
> 774bf81

Hm, I think I see what's happening. iiuc, we are using `mach build` to build rather than a make command directly through Mozharness. MakefileErrorList[1] seems to be only used by l10n jobs. I believe what we need to do is add whatever regexes we want to parse as errors as an error_list to the actual 'mach build' call here[2]. See an example of how this is done elsewhere here[3]


[1] https://dxr.mozilla.org/mozilla-central/search?q=MakefileErrorList&redirect=true
[2] https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/testing/mozharness/mozharness/mozilla/building/buildbase.py#1145
[3] https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/testing/mozharness/scripts/desktop_l10n.py#386

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Thanks for the pointer! That seems to work: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f97612b96985129b57d9c0f46a6fa66f1f275b3&selectedJob=211581726

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1455473 - Add some rust error-matching strings. r?chmanchester

Comment 8

[Ed Morley [:emorley]]

(In reply to Jordan Lund (:jlund) from comment #5)
> Sure. It's been years since I wrote the mozharness build wrapper and
> probably that long since it has been touched much but I'll put I'll try to
> help.

Thank you for taking a look :-)
(I hadn't actually looked at hg annotate - I needinfoed you since you were listed as the triage owner for this component; perhaps that's out of date?)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4012504e472f
Add some rust error-matching strings. r=chmanchester

Comment 10

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/4012504e472f