1418188 - Run Searchfox indexing in TaskCluster on a daily basis

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: patch

This bug is a follow-on to bug 1413738, which added the ability to index Firefox code for searchfox.org (the index results are generated as a build artifact). The goal of this bug is to run regular TaskCluster jobs to do the indexing.

I'm definitely going to need some help here since I don't know what I'm doing. I mostly copied the task configuration for our static analysis builds, which are very similar to the Searchfox builds. To do these builds periodically, I copied the code for DMD builds.

There are a couple problems that I'm aware of:
1. The windows builds don't work yet. I'd like to check in the task configs for them, but I don't want to start running them yet. I don't know how to filter them out, though.

2. DMD uses a special build platform, but Searchfox builds don't. Therefore I don't think that the target_tasks.py code is correct. I don't know how to test it, though.

I was able to test that the Linux and Mac tasks work on try server.

Dustin, can you give me some feedback here? I would appreciate it!

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8929306 [details] [diff] [review]
patch

Review of attachment 8929306 [details] [diff] [review]:
-----------------------------------------------------------------

For the windows builds, just don't select them in the target tasks method.  It's probably worth adding a comment somewhere -- either in that method or in kind.yml -- explaining that these tasks don't work and therefore don't run.  Otherwise someone will be confused later when they enable them and they break.

You can test the target_task_method by hacking things in a temporary commit so that it is used on a try push instead of the usual try methods.

I don't know much about the mozconfig or mozharness changes here.

::: .cron.yml
@@ +97,5 @@
>            by-project:
>                mozilla-central: [{hour: 10, minute: 0}]
>                # No default
>  
> +    - name: nightly-searchfox

We should probably stop calling these periodic tasks "nightly" (I see that you are not the first person to do this!).  "Nightly" suggests a build of firefox that goes to the Nightly channel, which this definitely isn't.  Maybe just `name: searchfox-index` and `treeherder-symbol: SearchFox(idx)`?  Then it will appear in a "SearchFox" group in treeherder, to which you can add other SearchFox-related tasks.

::: taskcluster/taskgraph/target_tasks.py
@@ +509,5 @@
> +    """Target Searchfox that run nightly on the m-c branch."""
> +    def filter(task):
> +        # FIXME: I don't think that using build_platform here works, but I'm not sure what to use.
> +        platform = task.attributes.get('build_platform', '')
> +        return platform.endswith('-sf')

yeah, that's pretty gross :)

It's only a few tasks you want here, so maybe just a list of labels and check membership?

```python
return ['label-of-task-i-want', 'label-of-second-task', ..]
```

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Note to self: I need to remember to disable sccache for these builds.

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: patch v2

This patch seems to work, although only the Linux build is actually working right now. Bug 1418415 is filed for fixing Mac and Windows.

I can't figure out how to disable sccache. Maybe a reviewer here can help me. Hopefully it's not too hard.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

The patch at https://hg.mozilla.org/try/rev/997716d8e8beb8ab11f0ba93eb1d9b442a97cb10 seems to disable sccache. The try push that includes that patch is at https://treeherder.mozilla.org/#/jobs?repo=try&revision=206f2175a511101353d6cb1807f8c48c05993db8. Feel free to steal it and squash it into your patch.

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: patch v3

Thanks Kats. Updated to turn off sccache.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8929790 [details] [diff] [review]
patch v3

Review of attachment 8929790 [details] [diff] [review]:
-----------------------------------------------------------------

::: .cron.yml
@@ +100,5 @@
>  
> +    - name: searchfox-index
> +      job:
> +          type: decision-task
> +          treeherder-symbol: Nsearchfox

I stand corrected regarding the use of group(symbol) syntax here.  These symbols are automatically nested under cron(..).  Why `N` here?

::: taskcluster/ci/config.yml
@@ +46,5 @@
>          'TM': 'Toolchain builds for OSX'
>          'TMW': 'Toolchain builds for Windows MinGW'
>          'TW32': 'Toolchain builds for Windows 32-bits'
>          'TW64': 'Toolchain builds for Windows 64-bits'
> +        'Searchfox': 'Searchfox builds'

++ :)

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8929790 [details] [diff] [review]
patch v3

Review of attachment 8929790 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_debug_searchfox.py
@@ +39,5 @@
> +        'PATH': '/tools/python/bin:/opt/local/bin:/usr/bin:'
> +                '/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin',
> +    },
> +    'mozconfig_variant': 'debug-searchfox',
> +    #######################

I think this would also want `'artifact_flag_build_variant_in_try': None,`.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c2e5d5828a5
Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9bd0ee246f1e
Revert "Bug 1418188 - Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)"

Comment 10

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Somehow, the Windows, Mac, and Linux indexing jobs were getting run on every inbound build after I pushed this. Is there any way to disable that, Dustin?

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b50216f3d2e6
Run Searchfox indexing in TaskCluster on a daily basis (r=dustin,chmanchester)

Comment 12

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I tried adding run-on-projects: [] to the YAML file. Let's see if that works.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

FYI you can test this locally (without doing pushes) by:
1) Click on an existing mozilla-inbound decision task in TreeHerder, and download the "parameters.yml" artifact
2) Run "mach taskgraph target-graph -p parameters.yml"

This will list all the tasks that would be generated given the in-tree taskcluster config state, using the properties in the parameters.yml file for the tree you care about. I ran it on your latest inbound push and it looks like it will still generate the searchfox tasks.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #13)
> I ran it on your latest inbound push and it
> looks like it will still generate the searchfox tasks.

Oh wait sorry, I misread the output. The updated patch should be good.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1fb2e661a99b
Fix indentation on a CLOSED TREE

Comment 16

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/b50216f3d2e6
https://hg.mozilla.org/mozilla-central/rev/1fb2e661a99b