1703115 - Use /tree/static/ paths instead of /static/ paths and ensure web-server uses the same mozsearch revisions the indexer was run against.

Status: RESOLVED FIXED

(Webtools :: Searchfox, task)

Description

[Andrew Sutherland [:asuth] (he/him)]

Today's breakage (bug 1703077) ended up being a little more extensive because our use of a single /static/ route across all of the different configN configurations acts in opposition to our isolation of the different machine configurations. We should switch to having the static resources for each configuration and tree/repository be specific to that tree/repository by exposing the static resources at /tree/static/, for example /mozilla-central/static/.

Although slightly more complex from an application perspective, it could also be beneficial for us to reference these paths via relative paths like ../../static/blah/ because this could enable UI experimentation by letting us do tricky like having /mozilla-central/ and /fancy-mozilla-central/ serve a different effective UI for the same static HTML page without actually having to do twice the work and without having to dynamically recompute all the paths. (Alternately, I suppose that could be addressed via reverse proxying.)

Comment 1

[Kartikaya Gupta (email:[e-mail])]

The flip side to this is that when all the webservers are using the same CSS, we don't need to regenerate the config3 web server for every CSS change, which is nice. I don't have strong feelings either way.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

In that case we can also just update the git checkout too.

One other nicety of a change like this is that it could make the rollout of more extensive changes to searchfox a little easier. Specifically, for the structured analysis branch, especially if we go forward with linking as proposed, we run into the issue of the taskcluster jobs for older branches of searchfox potentially not producing what the trunk indexing jobs want and thereby experiencing some level of breakage/degraded functionality. But if we, in effect, let them be totally isolated, we could specify that the config2-4 should continue to use an older "stable" branch of searchfox for indexing until such time as we're able to make sure we've uplifted changes to the older branches (or the changes just get there naturally) or ensured that the we've dealt with or understood and accepted any fallout/drift. (The structured analysis changes themselves I think are fine, but imply removing the existing in-mozsearch XPIDL indexing.)

Logistically: We could presumably have a deployment helper tool that scans the list of existing web-servers, checks their revisions and how far they are from trunk, then determines whether anything has changed that would mandate a re-indexing or if just updating the checkout is sufficient. I guess changes to anything but /static/ or /docs/ would probably want to presume a re-indexing.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Going to do this now. The specific plan:

• Yes to /tree/static/.
  • Re: comment 1 scenario, in general we've only experienced issues where the HTML is inconsistent with CSS changes and rarely a case where we wanted the CSS updated. Also, I've set config3 to run on a weekly cron job now which also makes it very easy to trigger whenever desired by using the lambda console to "test" the function.
  • Cache performance is not currently a concern given the small sizes of our static resources. If we start using the WASM-compiled graphviz, then maybe there could be an argument for a shared /static/ still.
  • The architecture is still very much sharded on a per-tree basis and if we do any moves to use taskcluster more, this will only increase.
• No relative paths, we're going to stick with absolute paths for now.
  • For all the experimental new work, I haven't ended up needing to meaningfully fork the UI, although there may be some transitional inefficiency as a result.
  • Relative paths are unjustified complexity without the above reason.
• We're also going to have trigger-web-server.py pass the exact mozsearch and mozsearch-mozilla revisions used by the indexer so the indexer and web-server checkouts are coherent! aws/web-server.sh will be modified to use this information.
  • This is perhaps the most common breakage we've experienced in this space. We land a change to the HTML generation with matching CSS, but the change lands after the indexer starts and before the web-server starts and so that web-server's CSS is not consistent.
  • This has been previously discussed on #searchfox in chat.mozilla.org but I'm not sure I see a bug. I've expanded this bug's scope and will dupe to this if it turns out I did file one.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: /tree/static/ paths, web-server uses exact git revision as indexer

Here's the top of the web server's update-log where we can see that it was explicitly given the revisions to use:

+ date
Mon Jul 25 00:18:13 UTC 2022
+ '[' 4 '!=' 4 ']'
+ MOZSEARCH_REPO=https://github.com/mozsearch/mozsearch
+ MOZSEARCH_REV=bcfc4739d2eb31cc905e43fac9c13e46037597bd
+ CONFIG_REPO=https://github.com/mozsearch/mozsearch-mozilla
+ CONFIG_REV=7f450c8db915e4fa43f0ca71321c7bd711199084
+ echo Mozsearch repository is https://github.com/mozsearch/mozsearch rev bcfc4739d2eb31cc905e43fac9c13e46037597bd
Mozsearch repository is https://github.com/mozsearch/mozsearch rev bcfc4739d2eb31cc905e43fac9c13e46037597bd
+ echo Config repository is https://github.com/mozsearch/mozsearch-mozilla rev 7f450c8db915e4fa43f0ca71321c7bd711199084
Config repository is https://github.com/mozsearch/mozsearch-mozilla rev 7f450c8db915e4fa43f0ca71321c7bd711199084
+ rm -rf mozsearch
+ git clone https://github.com/mozsearch/mozsearch mozsearch
Cloning into 'mozsearch'...
+ pushd mozsearch
~/mozsearch ~
+ git checkout bcfc4739d2eb31cc905e43fac9c13e46037597bd
Note: switching to 'bcfc4739d2eb31cc905e43fac9c13e46037597bd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at bcfc473 Bug 1703115 - address divergence in update functions in provisioning
+ git submodule init
Submodule 'deps/livegrep' (https://github.com/mozsearch/livegrep) registered for path 'deps/livegrep'
Submodule 'tools/ipdl_parser' (https://github.com/mozsearch/ipdl_parser/) registered for path 'tools/ipdl_parser'
+ git submodule update
Cloning into '/home/ubuntu/mozsearch/deps/livegrep'...
Cloning into '/home/ubuntu/mozsearch/tools/ipdl_parser'...
Submodule path 'deps/livegrep': checked out 'e95ae815d66da7eb38434f0c4324db8464348a25'
Submodule path 'tools/ipdl_parser': checked out '87cba37f3e23d87703e3e45873745398d18a8894'
+ popd
~
+ rm -rf config
+ git clone https://github.com/mozsearch/mozsearch-mozilla config
Cloning into 'config'...
+ pushd config
~/config ~
+ git checkout 7f450c8db915e4fa43f0ca71321c7bd711199084
Note: switching to '7f450c8db915e4fa43f0ca71321c7bd711199084'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 7f450c8 Bug 1759071 - Use coverage aggregation jobs as basis for m-c rev selection.
+ popd

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: address divergence in update functions in provisioning