Run Searchfox indexing in TaskCluster on a daily basis

RESOLVED FIXED

Status

defect
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: billm, Assigned: billm)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox59 fixed)

Details

Attachments

(1 attachment, 2 obsolete attachments)

Posted patch patch (obsolete) — Splinter Review
This bug is a follow-on to bug 1413738, which added the ability to index Firefox code for searchfox.org (the index results are generated as a build artifact). The goal of this bug is to run regular TaskCluster jobs to do the indexing.

I'm definitely going to need some help here since I don't know what I'm doing. I mostly copied the task configuration for our static analysis builds, which are very similar to the Searchfox builds. To do these builds periodically, I copied the code for DMD builds.

There are a couple problems that I'm aware of:
1. The windows builds don't work yet. I'd like to check in the task configs for them, but I don't want to start running them yet. I don't know how to filter them out, though.

2. DMD uses a special build platform, but Searchfox builds don't. Therefore I don't think that the target_tasks.py code is correct. I don't know how to test it, though.

I was able to test that the Linux and Mac tasks work on try server.

Dustin, can you give me some feedback here? I would appreciate it!
Attachment #8929306 - Flags: feedback?(dustin)
Comment on attachment 8929306 [details] [diff] [review]
patch

Review of attachment 8929306 [details] [diff] [review]:
-----------------------------------------------------------------

For the windows builds, just don't select them in the target tasks method.  It's probably worth adding a comment somewhere -- either in that method or in kind.yml -- explaining that these tasks don't work and therefore don't run.  Otherwise someone will be confused later when they enable them and they break.

You can test the target_task_method by hacking things in a temporary commit so that it is used on a try push instead of the usual try methods.

I don't know much about the mozconfig or mozharness changes here.

::: .cron.yml
@@ +97,5 @@
>            by-project:
>                mozilla-central: [{hour: 10, minute: 0}]
>                # No default
>  
> +    - name: nightly-searchfox

We should probably stop calling these periodic tasks "nightly" (I see that you are not the first person to do this!).  "Nightly" suggests a build of firefox that goes to the Nightly channel, which this definitely isn't.  Maybe just `name: searchfox-index` and `treeherder-symbol: SearchFox(idx)`?  Then it will appear in a "SearchFox" group in treeherder, to which you can add other SearchFox-related tasks.

::: taskcluster/taskgraph/target_tasks.py
@@ +509,5 @@
> +    """Target Searchfox that run nightly on the m-c branch."""
> +    def filter(task):
> +        # FIXME: I don't think that using build_platform here works, but I'm not sure what to use.
> +        platform = task.attributes.get('build_platform', '')
> +        return platform.endswith('-sf')

yeah, that's pretty gross :)

It's only a few tasks you want here, so maybe just a list of labels and check membership?

```python
return ['label-of-task-i-want', 'label-of-second-task', ..]
```
Attachment #8929306 - Flags: feedback?(dustin) → feedback+
Note to self: I need to remember to disable sccache for these builds.
Posted patch patch v2 (obsolete) — Splinter Review
This patch seems to work, although only the Linux build is actually working right now. Bug 1418415 is filed for fixing Mac and Windows.

I can't figure out how to disable sccache. Maybe a reviewer here can help me. Hopefully it's not too hard.
Attachment #8929306 - Attachment is obsolete: true
Attachment #8929622 - Flags: review?(dustin)
Attachment #8929622 - Flags: review?(core-build-config-reviews)
The patch at https://hg.mozilla.org/try/rev/997716d8e8beb8ab11f0ba93eb1d9b442a97cb10 seems to disable sccache. The try push that includes that patch is at https://treeherder.mozilla.org/#/jobs?repo=try&revision=206f2175a511101353d6cb1807f8c48c05993db8. Feel free to steal it and squash it into your patch.
Posted patch patch v3Splinter Review
Thanks Kats. Updated to turn off sccache.
Attachment #8929622 - Attachment is obsolete: true
Attachment #8929622 - Flags: review?(dustin)
Attachment #8929622 - Flags: review?(core-build-config-reviews)
Attachment #8929790 - Flags: review?(dustin)
Attachment #8929790 - Flags: review?(core-build-config-reviews)
Comment on attachment 8929790 [details] [diff] [review]
patch v3

Review of attachment 8929790 [details] [diff] [review]:
-----------------------------------------------------------------

::: .cron.yml
@@ +100,5 @@
>  
> +    - name: searchfox-index
> +      job:
> +          type: decision-task
> +          treeherder-symbol: Nsearchfox

I stand corrected regarding the use of group(symbol) syntax here.  These symbols are automatically nested under cron(..).  Why `N` here?

::: taskcluster/ci/config.yml
@@ +46,5 @@
>          'TM': 'Toolchain builds for OSX'
>          'TMW': 'Toolchain builds for Windows MinGW'
>          'TW32': 'Toolchain builds for Windows 32-bits'
>          'TW64': 'Toolchain builds for Windows 64-bits'
> +        'Searchfox': 'Searchfox builds'

++ :)
Attachment #8929790 - Flags: review?(dustin) → review+
Comment on attachment 8929790 [details] [diff] [review]
patch v3

Review of attachment 8929790 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_debug_searchfox.py
@@ +39,5 @@
> +        'PATH': '/tools/python/bin:/opt/local/bin:/usr/bin:'
> +                '/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin',
> +    },
> +    'mozconfig_variant': 'debug-searchfox',
> +    #######################

I think this would also want `'artifact_flag_build_variant_in_try': None,`.
Attachment #8929790 - Flags: review?(core-build-config-reviews) → review+

Comment 8

2 years ago
Pushed by wmccloskey@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c2e5d5828a5
Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)

Comment 9

2 years ago
Pushed by wmccloskey@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9bd0ee246f1e
Revert "Bug 1418188 - Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)"
Somehow, the Windows, Mac, and Linux indexing jobs were getting run on every inbound build after I pushed this. Is there any way to disable that, Dustin?
Flags: needinfo?(dustin)

Comment 11

2 years ago
Pushed by wmccloskey@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b50216f3d2e6
Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)
I tried adding run-on-projects: [] to the YAML file. Let's see if that works.
FYI you can test this locally (without doing pushes) by:
1) Click on an existing mozilla-inbound decision task in TreeHerder, and download the "parameters.yml" artifact
2) Run "mach taskgraph target-graph -p parameters.yml"

This will list all the tasks that would be generated given the in-tree taskcluster config state, using the properties in the parameters.yml file for the tree you care about. I ran it on your latest inbound push and it looks like it will still generate the searchfox tasks.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #13)
> I ran it on your latest inbound push and it
> looks like it will still generate the searchfox tasks.

Oh wait sorry, I misread the output. The updated patch should be good.
Flags: needinfo?(dustin)
You need to log in before you can comment on or make changes to this bug.