753547 - make mozharness test scripts easier to run standalone

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

Specifically if we want to run without downloading anything.

Currently that's peptest and talos and jetperf; I think I can get them in the same fix.

Comment 1

[Aki Sasaki (not active)]

Attachment: fix installer_path

Comment 2

[Aki Sasaki (not active)]

Not actively working on this.
This is still a big want, and I'd be happy to block mozharness 1.0 on this.
We have found that although mozharness is designed to be both user- and automation-runnable, it's definitely being written more for the latter.

Comment 3

[Jordan Lund (:jlund)]

I woul

Comment 4

[Jordan Lund (:jlund)]

I would like to pick this up.

I have some ideas on how we can be more consistent for devs and not so painful for us to change existing scripts.

I can discuss this at our next mozharness meeting.

Comment 5

[Jordan Lund (:jlund)]

==== status update

I have some time today to start working on this

Comment 6

[Jordan Lund (:jlund)]

There are two things that I think we need to accomplish that would help a lot and not take much work:

1) make actions more standalone and only have two places for config items -> bug 1007280
   * the two places would be
       1. self.config: a r/o cfg
           eg: consolidated from in-tree, urls, and/or within mozharness's config/*
       2. self.live_config: r/w cfg
           eg: consolidated from: a cache of previous run state, outside forces like automation that are only known at run time (ie currently: buildbot_config), and also updated while the script runs actions (ie currently: buildbot_properties)
2) hide buildbot integration -> bug 1007292

Comment 7

[Aki Sasaki (not active)]

(In reply to Jordan Lund (:jlund) from comment #6)
> There are two things that I think we need to accomplish that would help a
> lot and not take much work:
> 
> 1) make actions more standalone and only have two places for config items ->
> bug 1007280
>    * the two places would be
>        1. self.config: a r/o cfg
>            eg: consolidated from in-tree, urls, and/or within mozharness's
> config/*
>        2. self.live_config: r/w cfg
>            eg: consolidated from: a cache of previous run state, outside
> forces like automation that are only known at run time (ie currently:
> buildbot_config), and also updated while the script runs actions (ie
> currently: buildbot_properties)

Ah.  Ok, this could be very cool.

Some comments:

* I'd be worried a bit about this thing getting crufty.  I'd be curious if, say,

    scripts/mobile_l10n.py [args] --locale es-ES
    scripts/mobile_l10n.py [args] --locale zh-TW

if the second call does the right thing.  You have the --clobber check, but sometimes clobber will nuke more than you want; in the case of l10n, above, we want to run the zh-TW repack in an un-clobbered workdir.  It's just something to keep an eye on and guard against, rather than a reason not to do this.

* We should log very loudly when any setting in this live config changes (after the initial setup, during runtime).

* self.buildbot_config is very large and ugly, and we only need a subset of it.  However, sometimes we want to access more of it than we initially intended.  Are you planning on dumping all of it into the live config, or only the portions we think are important?

> 2) hide buildbot integration -> bug 1007292

This may be problematic if we have a different live config at the beginning than at the end of the script run -- how does someone then replicate the run without the buildprops.json?


I think we're going to end up with some in-tree mozharness at some point, so that may end up hiding buildbot_config from the developer-facing stuff anyway.  Could still be worth a look, especially for keeping actions independent.

Comment 8

[Jordan Lund (:jlund)]

(In reply to Aki Sasaki [:aki] from comment #7)
> (In reply to Jordan Lund (:jlund) from comment #6)
> > There are two things that I think we need to accomplish that would help a
> > lot and not take much work:
> > 
> > 1) make actions more standalone and only have two places for config items ->
> > bug 1007280
> >    * the two places would be
> >        1. self.config: a r/o cfg
> >            eg: consolidated from in-tree, urls, and/or within mozharness's
> > config/*
> >        2. self.live_config: r/w cfg
> >            eg: consolidated from: a cache of previous run state, outside
> > forces like automation that are only known at run time (ie currently:
> > buildbot_config), and also updated while the script runs actions (ie
> > currently: buildbot_properties)
> 
> Ah.  Ok, this could be very cool.
> 
> Some comments:
> 
> * I'd be worried a bit about this thing getting crufty.  I'd be curious if,
> say,
> 
>     scripts/mobile_l10n.py [args] --locale es-ES
>     scripts/mobile_l10n.py [args] --locale zh-TW
> 

ok so how about cache_config is usable if: 1) clobber is not in actions 2) class names are the same 3) c.get('disable_cache') is False

where we set (3) in configs like ones that use mobile_l10n.py. It means for now things like mobile_l10n.py will not use a cache but it is a quick clean way to remove chance of cruft.

> * We should log very loudly when any setting in this live config changes
> (after the initial setup, during runtime).

/me makes verbose methods for anything that mutates self.live_config

> 
> * self.buildbot_config is very large and ugly, and we only need a subset of
> it.  However, sometimes we want to access more of it than we initially
> intended.  Are you planning on dumping all of it into the live config, or
> only the portions we think are important?

my reply below also is a reply to your other comments regarding buildbot_config,

self.buildbot_config is turning into a bit of a beast to deal with. Since we are likely splitting mozharness, I don't think it's worth solving today. But I think there is still a use in some of what I am doing: 

How about we still replace self.set_buildbot_property() cases with live_config?

We can also replace some self attr assignments (eg: self.installer_url from desktop builds) with live_config.

This fills live_config with only props that are important and set during individual actions. This, with its caching capabilities, should still help make actions more independent and lessen the mention of 'buildbot' in mozharness scripts.

Comment 9

[Aki Sasaki (not active)]

(In reply to Jordan Lund (:jlund) from comment #8)
> How about we still replace self.set_buildbot_property() cases with
> live_config?
> 
> We can also replace some self attr assignments (eg: self.installer_url from
> desktop builds) with live_config.
> 
> This fills live_config with only props that are important and set during
> individual actions. This, with its caching capabilities, should still help
> make actions more independent and lessen the mention of 'buildbot' in
> mozharness scripts.

I agree here, except set_buildbot_property() has downstream effects (sent to buildbot, which gets picked up by Pulse and downstream consumers).  The downstream names and our runtime names might not match currently.  We can certainly try to make them match if possible, though.

Comment 10

[Jordan Lund (:jlund)]

==== status update

I have been moved to 'Bug 932396 - implement "disable" action in slaveapi' and 'Bug 965691 - Create a Comprehensive Slave Loan tool'

I am not actively working on this. I intend to take another stab at it once those are completed or I have some breathing room.

Comment 11

[Aki Sasaki (not active)]

I don't think we're going to put more effort into this; people can easily run mozharness tasks in Taskcluster.