1589710 - [meta] Investigate corruptions in the _engines list stored in the search cache

Status: RESOLVED WORKSFORME

(Firefox :: Search, defect, P2)

Description

[Mark Banner (:standard8)]

We've been seeing various corruptions in the engines list stored in the search cache. The general symptoms are:

• Default engine can be changed to something random, e.g. Amazon (bug 1578807)
• Not all the expected engines are listed in Search displays, and can't be restored by "Restore Default Engines".
• Sometimes hidden engines may also be restored (e.g. bug 1586078).

Bug 1580737 found one cause, which was rebuilding the cache whilst trying to shutdown/restart. It added a prevention to prevent rebuilding the cache.

In bug 1578807 we've attempted a workaround to rebuild the cache if it detects the amount of built-in engines in the cache are different to the expected built-in engines. We also added telemetry, which I am monitoring.

Both of those have landed in Firefox 70.

This bug is for tracking the issues and potential fixes. Currently we're working towards a new configuration backend, which may or may not help with this. However, in the meantime we're going to monitor it.

Comment 1

[Mark Banner (:standard8)]

Having been tracking the stats for the last few cycles, after having eradicated a few differences, there's now a background level of < 0.1% of startups which have engine lists out of date and are marked as "corrupted".

I noticed on the nightly stats that there was a bump when we removed Twitter. Having tested it, the cause is that the update is received, and maybeReloadEngines kicks in. This updates the builtin list saved in the cache, but doesn't remove now-obsolete engines, they are left until next restart. On restart we compare the saved engines with the separate list and it doesn't match, hence we mark it as corrupted, though in this case it is a false alarm.

For new users, it is certainly the case on various configurations that they will get a different engine configuration when maybeReloadEngines runs after getting region information on startup. It is my suspicion therefore that the remaining background level is due to those.

I'm planning on fixing bug 1610293 soon which should hopefully fix the background issue.

Comment 2

[Mark Banner (:standard8)]

We have now removed the probe that was looking at corrupted. Most of the issues we've found via the probe that we had, and over the course of looking at this have been false alarm related where for example, we've not had the correct checks in place to properly detect corruption as we've missed a parameter.

We did fix a few potential issues, like writing during shutdown, so that has been useful.

There's currently no obvious evidence of corruption (where the engines list gets out of date), with the latest Firefox code base. We also haven't received any reports from users that look like this.

Therefore closing this out as works for me.