2055619 - Add query-derivation module for the search CTA

Status: RESOLVED FIXED

(Firefox :: Security, task)

Description

[Jack Brown]

Add a module that turns a failed URL into a { action, query, reason } result.

See:
https://docs.google.com/document/d/1c8ZPioin4Bnj_5u9_su7vsfv6L3FSgnmORz7EM0ucAI/edit?tab=t.i5r0fb3p704v#heading=h.lhd46c2jtkx3

The full routine looks like this:

• Check for host viability (see the linked doc for criteria). Either we block here and don't show the CTA, or we continue.
• Not part of this bug's scope, but we'd render the CTA as soon as host viability passes.
• Tokenize URL.pathname and the registrable domain only.
• Filter tokens: grammar stopwords, URL-structure stopwords, opaque-token filter.
• If surviving tokens >= minKeywords then we return action=keywords and reason=ok. Otherwise use the host fallback: action=host and reason=no-path or no-meaningful-keywords.

Acceptance criteria

• If a host doesn't pass our host-viability check, then we do not show the CTA and we don't derive the search query.
• Produces the documented {action, query, reason} for the worked examples (blocked hosts, descriptive path, typo'd bare host, secret-laden path).
• URL search query string / #fragment are never tokenized.
• Host fallback uses the registrable domain only.
• minKeywords parameterized (defaults to 1).

Comment 1

[Jack Brown]

Attachment: Bug 2055619 - Add URLKeywordAnalyzer query-derivation module for the search CTA r?fchasen,jaws

Comment 2

[Gabriel H [:gharp, :gabriel]]

[:jbrown] Following discussion yesterday:

• agreed to keep the path-preference (path terms fill the query first; the registrable-domain (host) label is included only if a slot remains)
• cap the query at 8 keywords (not 6)

Comment 3

[Gabriel H [:gharp, :gabriel]]

[:jbrown] as discussed yesterday, below is the final list of ERRORPAGE_STOPWORDS. Methodology is captured here: https://docs.google.com/document/d/1c8ZPioin4Bnj_5u9_su7vsfv6L3FSgnmORz7EM0ucAI/edit?tab=t.r8kmvh7a363b

export const ERRORPAGE_STOPWORDS = new Set([
  // ---- function words (262) ----
  "about", "above", "across", "after", "afterwards", "again", "against",
  "all", "almost", "alone", "along", "already", "also", "although",
  "always", "am", "among", "amongst", "amoungst", "an", "and", "another",
  "any", "anyhow", "anyone", "anything", "anyway", "anywhere", "are",
  "around", "as", "at", "be", "became", "because", "become", "becomes",
  "becoming", "been", "before", "beforehand", "behind", "being", "below",
  "beside", "besides", "between", "beyond", "both", "but", "by", "can",
  "cannot", "cant", "co", "could", "couldnt", "de", "do", "done", "down",
  "due", "during", "each", "eg", "either", "else", "elsewhere", "enough",
  "etc", "even", "ever", "every", "everyone", "everything", "everywhere",
  "except", "few", "for", "former", "formerly", "from", "further", "get",
  "give", "go", "had", "has", "hasnt", "have", "he", "hence", "her",
  "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself",
  "him", "himself", "his", "how", "however", "ie", "if", "in", "inc",
  "indeed", "into", "is", "it", "its", "itself", "keep", "latter",
  "latterly", "least", "less", "ltd", "many", "may", "me", "meanwhile",
  "might", "more", "moreover", "most", "mostly", "much", "must", "my",
  "myself", "namely", "neither", "never", "nevertheless", "next", "no",
  "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere",
  "of", "off", "often", "on", "once", "only", "onto", "or", "other",
  "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own",
  "per", "perhaps", "please", "put", "rather", "re", "same", "search",
  "see", "seem", "seemed", "seeming", "seems", "several", "she", "should",
  "since", "so", "some", "somehow", "someone", "something", "sometime",
  "sometimes", "somewhere", "still", "such", "take", "than", "that", "the",
  "their", "them", "themselves", "then", "thence", "there", "thereafter",
  "thereby", "therefore", "therein", "thereupon", "these", "they", "this",
  "those", "though", "through", "throughout", "thru", "thus", "to",
  "together", "too", "toward", "towards", "under", "until", "up", "upon",
  "very", "via", "was", "we", "were", "what", "whatever", "when", "whence",
  "whenever", "where", "whereafter", "whereas", "whereby", "wherein",
  "whereupon", "wherever", "whether", "which", "while", "whither", "who",
  "whoever", "whom", "whose", "why", "will", "with", "within", "without",
  "would", "yet", "you", "your", "yours", "yourself", "yourselves",

  // ---- structure words (33) ----
  // existing 10
  "article", "blog", "category", "home", "http", "https", "index", "page",
  "tag", "www",
  // site plumbing
  "archive", "content", "details", "files", "images", "uploads", "wiki",
  "wp",
  // asset / format extensions
  "css", "html", "png",
  // api versioning
  "v2",
  // locale
  "en",
  // month abbreviations
  "apr", "aug", "dec", "feb", "jan", "jul", "jun", "nov", "oct", "sep",
]);

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/0ac28bf1cbb6
https://hg.mozilla.org/integration/autoland/rev/3f817da1c030
Add URLKeywordAnalyzer query-derivation module for the search CTA r=mconley

Comment 5

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/3f817da1c030