Closed Bug 1240184 Opened 4 years ago Closed 4 years ago

run mozharness tests in the gecko task graph

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jlund, Assigned: Callek)

References

Details

Attachments

(1 file, 1 obsolete file)

when we put mozharness into the tree we lost github's test automation for mozharness. There are a few places that come to mind where we can add the automation back in:

1) as an hg hook when pushing
2) prior to any mozharness script call
3) in mozharness archiver (relengapi tool for caching subdir archives from large repos)
4) before TC task graph generation

my vote is for (4). (3) Archiver is not going to be around for a long time®. (1) is an unknown for me and (2) seems like a waste and messy to check multiple times per checkin. (4) is nice because Taskcluster is the future and most automation jobs should be configured/scheduled through the decision task (mach generate_task_graph).

e.g.

1) checkout $gecko && cd testing/mozharness
2) createORsource mozharness/venv; pip install tox
3) run tox
4) if passes: generate_and_submit_tc_graph(); else: fail
See Also: → 1230150
If it is part of the decision job how can we ensure to run the unit tests on all platforms? I would assume there is platform dependent behavior somewhere.
We should also update the tox.ini to use a version of hg that's supported by version-control-tools. I have some of the extensions on, and tox failed cause hg 2.6 doesn't like them.
I could do that when trying to fix the regression one of my patches might have been caused via bug 1230150.
(In reply to Henrik Skupin (:whimboo) from comment #3)
> I could do that when trying to fix the regression one of my patches might
> have been caused via bug 1230150.

Whereby lets make it a separate issue. Feel free to file another bug for it.
fwiw unless mozharness is responsible for running the gecko descision graph generation, I'd argue we should just insert it into the task graph as a dep of everything that uses mozharness (if we don't expect any intermittents from it) or as a side-run thing if there are intermittents from it.
(In reply to Henrik Skupin (:whimboo) from comment #1)
> If it is part of the decision job how can we ensure to run the unit tests on
> all platforms? I would assume there is platform dependent behavior somewhere.

we wouldn't but (1) at least the mh tests failing would get caught and (2) eventually all jobs across all platforms will be through this decision task because it is the futures.

(In reply to Justin Wood (:Callek) from comment #5)
> fwiw unless mozharness is responsible for running the gecko descision graph
> generation, I'd argue we should just insert it into the task graph as a dep
> of everything that uses mozharness (if we don't expect any intermittents
> from it) or as a side-run thing if there are intermittents from it.

sure, valid point.
I think we should just schedule this as a task in the task graph.  I'm not even sure it should be a prerequisite -- that would just increase our e2e time.

This would mix nicely with the mach unit tests, for example.  And also with the ESLint stuff that's being set up -- tasks that can run straight from the source and don't require or produce a binary.
Summary: gecko decision task should run mozharness unittests prior to task graph generation → gecko decision task should run early in the task graph
Assignee: nobody → dustin
Blocks: 1246527
Assignee: dustin → nobody
Summary: gecko decision task should run early in the task graph → run mozharness tests in the gecko task graph
No longer blocks: 1247703
Depends on: 1247703
Attached patch [mc] mozharness tox (obsolete) — Splinter Review
Scratching an itch now that I'm back.

Looking for anything that would block this landing.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0593907de955 is my try push...

When this lands the file_patterns will be `testing/mozharness/**` to be clear.
Assignee: nobody → bugspam.Callek
Status: NEW → ASSIGNED
Attachment #8733105 - Flags: feedback?(jlund)
Attachment #8733105 - Flags: feedback?(dustin)
Cool! Btw. from where do we install the additional python packages? Their download looks pretty slow so I wonder if that could be from pypi and not from our internal mirror.
Comment on attachment 8733105 [details] [diff] [review]
[mc] mozharness tox

Review of attachment 8733105 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/tests/mozharness-gecko.yml
@@ +7,5 @@
> +    name: '[TC] - Mozharness Tox'
> +    description: 'Mozharness integration tests'
> +
> +  scopes:
> +    - 'docker-worker:cache:level-{{level}}-{{project}}-dotcache'

You don't use this cache, so you don't need this scope

@@ +13,5 @@
> +  payload:
> +    image:
> +      type: 'task-image'
> +      path: 'public/image.tar'
> +      taskId: '{{#task_id_for_image}}tester{{/task_id_for_image}}'

"tester" is a b2g image, and may go away or change unexpectedly.  Better to use desktop-test.

@@ +16,5 @@
> +      path: 'public/image.tar'
> +      taskId: '{{#task_id_for_image}}tester{{/task_id_for_image}}'
> +
> +    cache:
> +      level-{{level}}-{{project}}-tc-vcs: '/home/worker/.tc-vcs'

This cache is already included in lint.yml and need not be repeated here
Attachment #8733105 - Flags: feedback?(dustin) → feedback+
Comment on attachment 8733105 [details] [diff] [review]
[mc] mozharness tox

Review of attachment 8733105 [details] [diff] [review]:
-----------------------------------------------------------------

outside of dustins comments, it looks good to me. one concern below in line

::: testing/taskcluster/tasks/branches/base_jobs.yml
@@ +366,5 @@
> +    root: true
> +    when:
> +      file_patterns:
> +        # Taskcluster stuff to trigger my try push
> +        - 'testing/taskcluster/**'

should this be 'testing/mozharness/**' ?
Attachment #8733105 - Flags: feedback?(jlund) → feedback+
Try push with this, and eslint enabled is:  https://treeherder.mozilla.org/#/jobs?repo=try&revision=300113fd744e

(I enabled by merely marking testing/taskcluster/** as triggering files)
Attachment #8733105 - Attachment is obsolete: true
Attachment #8733494 - Flags: review?(dustin)
Attachment #8733494 - Flags: review?(dustin) → review+
Following this mostly as a learning exercise for myself, and wondering:

Should the task also run when the task definitions change? i.e., any of the 4 files here at least?

Also spotted 
  workerType: b2gtest
in lint.yml, unrelated to this patch, but probably wants a follow-up, re comment 10?
(In reply to Axel Hecht [:Pike] from comment #14)
> Following this mostly as a learning exercise for myself, and wondering:
> 
> Should the task also run when the task definitions change? i.e., any of the
> 4 files here at least?

Probably a good idea (I just didn't add them to avoid extra complexity, since the other files: ones didn't do that either)

> Also spotted 
>   workerType: b2gtest
> in lint.yml, unrelated to this patch, but probably wants a follow-up, re
> comment 10?

I suspect a blanket audit of "what uses b2gtest" and then replacing with a better choice is a good idea, however I don't actually know what other worker types exist to know what is more appropriate atm, so I left it alone.
b2gtest will be fine for now -- it's easy to do an 'hg grep' (and get frustrated that it doesn't work and instead use `grep -r` instead) to find uses of b2gtest when we want to turn it off.

As we build out the new task-optimization system in the in-tree configuration system, it will be more careful to cover the task definitions as well.
https://hg.mozilla.org/mozilla-central/rev/93c1fd856330
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1246527
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.