1067354 - Improve documentation for non-test type of jobs (e.g. rooting analysis) and find ways to run it locally

Status: RESOLVED INCOMPLETE

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Armen [:armenzg]]

See the issues that Ting-Yu Chou faced trying to run browser rooting analysis:
https://groups.google.com/forum/#!topic/mozilla.dev.platform/RNks2fV0h6E

Comment 1

[Armen [:armenzg]]

I have updated the documentation to help developers not go astray and point them to this bug.

I noticed that for non-tests jobs, we don't even have real developer mode [1] since it is part of testbase.py rather than BaseScript. I guess I had never wanted to think of the wider scope and wait for the time to cross that bridge.
I think we should have a DeveloperMixin and re-use it around.


For the case of root analysis:
I think we can add a configs/hazards/common_dev.py specifically for rooting analysis jobs since I would not want to mix a lot of its values into configs/developer_config.py

I believe we can add a feature to the developer mode to look for associated _dev.py of the config files passed.

jlund: do these things make sense to you?

[1] http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/testbase.py#l138
http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.py#l1125

Comment 2

[Jordan Lund (:jlund)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #1)
> I have updated the documentation to help developers not go astray and point
> them to this bug.
> 
> I noticed that for non-tests jobs, we don't even have real developer mode
> [1] since it is part of testbase.py rather than BaseScript. I guess I had
> never wanted to think of the wider scope and wait for the time to cross that
> bridge.
> I think we should have a DeveloperMixin and re-use it around.

TBH - I think it would have been easier to set it up so mozharness scripts run outside of automation by default and then used an AutomationMixin when we plug it into buildbot. that would have forced us to not define so much reliance on buildprops.json (the biggest problem for local devs), mach wouldn't really be needed, and swapping out our scheduler from say buildbot to taskcluster would have been a breeze.

armen, you seem to have an interest in making this easier for devs. maybe we could do a breakout session/sprint on this. I've done some thinking and work in the past that I think would complement (maybe even make easier) your efforts: 'Bug 753547 - make mozharness test scripts easier to run standalone'

you can see how I tried to remove read-buildbot-config action altogether and made it a prerun listener if we were running in automation. this is the kind of thing I like as it removes the need to hack on pre_config_lock or BaseScript/Config's existing methods.

> 
> 
> For the case of root analysis:
> I think we can add a configs/hazards/common_dev.py specifically for rooting
> analysis jobs since I would not want to mix a lot of its values into
> configs/developer_config.py
> 
> I believe we can add a feature to the developer mode to look for associated
> _dev.py of the config files passed.

sweet. I've done stuff like this and I now regret it as my implementation feels magical and not explicit. depends on your impl I guess

Comment 3

[Armen [:armenzg]]

(In reply to Jordan Lund (:jlund) from comment #2)
> armen, you seem to have an interest in making this easier for devs. maybe we
> could do a breakout session/sprint on this. I've done some thinking and work
> in the past that I think would complement (maybe even make easier) your
> efforts: 'Bug 753547 - make mozharness test scripts easier to run standalone'
> 

I think you have very good ideas in all of those bugs.
I'm happy to talk with jmaher and ask him a week or two to work with you in all of these issues.
To be clear, my main interest is to make it easier for devs and want those wins first.
We should at least meet once for an hour to at least create a common understanding and spotting what each bugs fixes so we know how to sell it to others.

Another thought I have is that we should split mozharness into two parts.
* mozharness core functionality (Mozilla independent)
** create a python package
* mozharness scripts & configs

I believe this would help in the long term for other projects adopting mozharness.