Closed Bug 1426275 Opened 2 years ago Closed 2 years ago

Better documentation for SCHEDULES and optimizations

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla59

People

(Reporter: dustin, Assigned: dustin)

Details

Attachments

(1 file)

We need some .rst for this, to go with the docstrings in context.py.
Andrew, I consider this a very rough draft so please feel free to suggest ways to make it clearer :)
Comment on attachment 8938969 [details]
Bug 1426275: document SCHEDULES in reStructuredText;

https://reviewboard.mozilla.org/r/209420/#review216254

Thanks for writing this up! I think developers will still be confused after reading this, but that's just because it's a complicated concept to understand. This is infintely better than nothing at all, so lgtm!

I have a few suggestions below.

::: taskcluster/docs/optimization-schedules.rst:6
(Diff revision 1)
> +Optimization and SCHEDULES
> +==========================
> +
> +Most optimization of builds and tests is handled with ``SCHEDULES``.
> +The concept is this: we allocate tasks into named components, and associate a set of such components to each file in the source tree.
> +Then, given a set of files changed in a push, we calculate the affected components (the union of the components affected by each file) remove any tasks that are not in one of the affected components.

Given a set of files changed in a push, we then calculate the union of components affected by each file, and remove tasks that are not tagged with any of them.

::: taskcluster/docs/optimization-schedules.rst:22
(Diff revision 1)
> +
> +Exclusive and Inclusive
> +-----------------------
> +
> +The first wrinkle in this "simple" plan is that there are a lot of files, and for the most part they all affect most components.
> +But there are some components which are affected only a well-defined set of files.

nit: affected only by a

::: taskcluster/docs/optimization-schedules.rst:28
(Diff revision 1)
> +A good way to keep this straight is to think of exclusive platform-family components (``macosx``, ``android``, ``windows``, ``linux``) and inclusive linting components (``py-lint``, ``js-lint``).
> +An arbitrary file in the repository affects all platform families, but does not necessarily require a lint run.
> +But we can configure mac-only files such as ``CoreLocationLocationProvider.mm`` to affect exclusively ``macosx``, and Python files like ``preprocessor.py`` to affect ``py-lint`` in addition to the exclusive components.

This paragraph reads a little confusing to me. Here's my attempt at explaining it, not sure it's much better though:

    Exclusive components can be thought of as a series of families. For example, the platform (linux, windows, macosx, android) is a component family. The test suite (mochitest, reftest, xpcshell, etc.) is another. By default, source files are associated with every component in every family. This means tasks tagged with an exclusive component will *always* run, unless none of the modified source files are associated with that component.
    
    But what if we only want to run a particular task when a pre-determined file is modified? This is where inclusive components are used. Any task tagged with an inclusive component will *only* be run when a source file associated with that component is modified. Lint tasks and well separated unittest tasks are good examples of things you might want to schedule inclusively.

::: taskcluster/docs/optimization-schedules.rst:77
(Diff revision 1)
> +
> +Tasks are annotated with the components they belong to using the ``"skip-unless-schedules"`` optimization, which takes a list of components for this task::
> +
> +    task['optimization'] = {'skip-unless-schedules': ['windows', 'gtest']}
> +
> +For tests, this value is set automatically by the test transform based on the suite name and the platform family, doing the correct thing for inclusive test suites

nit: missing period
Attachment #8938969 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8938969 [details]
Bug 1426275: document SCHEDULES in reStructuredText;

https://reviewboard.mozilla.org/r/209420/#review216254

> This paragraph reads a little confusing to me. Here's my attempt at explaining it, not sure it's much better though:
> 
>     Exclusive components can be thought of as a series of families. For example, the platform (linux, windows, macosx, android) is a component family. The test suite (mochitest, reftest, xpcshell, etc.) is another. By default, source files are associated with every component in every family. This means tasks tagged with an exclusive component will *always* run, unless none of the modified source files are associated with that component.
>     
>     But what if we only want to run a particular task when a pre-determined file is modified? This is where inclusive components are used. Any task tagged with an inclusive component will *only* be run when a source file associated with that component is modified. Lint tasks and well separated unittest tasks are good examples of things you might want to schedule inclusively.

Thanks!  I think your text and mine are complementary, so I'll keep both :)
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b3cc94c0accd
document SCHEDULES in reStructuredText; r=ahal
https://hg.mozilla.org/mozilla-central/rev/b3cc94c0accd
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.