Create a QuantumBar Tokenizer module

RESOLVED FIXED in Firefox 64



Last year
Last year


(Reporter: mak, Assigned: mak)


(Blocks 1 bug)

Firefox 64
Dependency tree / graph

Firefox Tracking Flags

(firefox63 wontfix, firefox64 fixed)


(Whiteboard: [fxsearch])


(1 attachment)

We can move the input string tokenizer to a separate module, usable by both the old and new implementation
Whiteboard: [fxsearch]
This is a first stab at the new tokenizer.
It's not expected to be perfect yet, but good enough to be modified and replace the existing code in unifiedComplete with just a few modifications.
It's mostly intended to start setting up a code and tests structure.
Drew, I'd like to land something, I may not be able to do further work on it before PTO, anyway nothing is using this yet, we can look into replaceing the tokenization in unifiedComplete.js with this, likely with a few modifications or changes (like support for combined restriction chars).
It's likely we'll have to modify it anyway when we have more pieces of the puzzle.
But the naming and tests structure should be fine regardless.
Comment on attachment 8998138 [details]
Bug 14785825 - Move address bar model tokenization to a module.

Drew Willcoxon :adw has approved the revision.
Attachment #8998138 - Flags: review+
Pushed by
Bug 14785825 - Move address bar model tokenization to a module. r=adw
Backed out changeset c291d6d95022 (bug 1478582) for browser-chome failures at browser/base/content/test/static/browser_all_files_referenced.js


Failure push:

Failure log:

[task 2018-09-06T13:47:48.976Z] 13:47:48     INFO - indirectly whitelisted file: chrome://passwordmgr/content/recipes.json used from resource://gre/greprefs.js
[task 2018-09-06T13:47:48.977Z] 13:47:48     INFO - indirectly whitelisted file: resource://formautofill/FormAutofillSync.jsm used from resource://services-sync/service.js
[task 2018-09-06T13:47:48.978Z] 13:47:48     INFO - indirectly whitelisted file: chrome://global/locale/ used from resource://gre/modules/accessibility/Utils.jsm
[task 2018-09-06T13:47:48.979Z] 13:47:48     INFO - indirectly whitelisted file: resource://services-sync/bookmark_repair.js used from resource://services-sync/collection_repair.js
[task 2018-09-06T13:47:48.980Z] 13:47:48     INFO - indirectly whitelisted file: resource://services-sync/doctor.js used from resource://services-sync/bookmark_repair.js,resource://services-sync/stages/enginesync.js
[task 2018-09-06T13:47:48.983Z] 13:47:48     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Traversal.jsm used from resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-09-06T13:47:48.984Z] 13:47:48     INFO - Buffered messages finished
[task 2018-09-06T13:47:48.985Z] 13:47:48     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | there should be no unreferenced files - Got 1, expected 0
[task 2018-09-06T13:47:48.986Z] 13:47:48     INFO - Stack trace:
[task 2018-09-06T13:47:48.987Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:test_is:1304
[task 2018-09-06T13:47:48.989Z] 13:47:48     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:761
[task 2018-09-06T13:47:48.990Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-06T13:47:48.995Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-06T13:47:48.995Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-06T13:47:48.997Z] 13:47:48     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-09-06T13:47:48.998Z] 13:47:48     INFO - Not taking screenshot here: see the one that was previously logged
[task 2018-09-06T13:47:48.999Z] 13:47:48     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | unreferenced file: resource://app/modules/UrlbarTokenizer.jsm - 
[task 2018-09-06T13:47:49.000Z] 13:47:49     INFO - Stack trace:
[task 2018-09-06T13:47:49.001Z] 13:47:49     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:765
[task 2018-09-06T13:47:49.002Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-06T13:47:49.005Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-06T13:47:49.006Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-06T13:47:49.006Z] 13:47:49     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-09-06T13:47:49.008Z] 13:47:49     INFO - ignored unused whitelist entry: resource://app/blocklist.xml
[task 2018-09-06T13:47:49.010Z] 13:47:49     INFO - ignored unused whitelist entry: resource://gre/gmp-clearkey/0.1/manifest.json
[task 2018-09-06T13:47:49.012Z] 13:47:49     INFO - ignored unused whitelist entry: resource://gre/res/
[task 2018-09-06T13:47:49.017Z] 13:47:49     INFO - GECKO(2871) | [Parent 2871, Main Thread] WARNING: Error loading theme icon 'dialog-warning' for stock: Icon 'dialog-warning' not present in theme ubuntu-mono-dark: 'glib warning', file /builds/worker/workspace/build/src/toolkit/xre/nsSigHandlers.cpp, line 141
[task 2018-09-06T13:47:49.019Z] 13:47:49     INFO - GECKO(2871) | (firefox:2871): Gtk-WARNING **: Error loading theme icon 'dialog-warning' for stock: Icon 'dialog-warning' not present in theme ubuntu-mono-dark
[task 2018-09-06T13:47:49.020Z] 13:47:49     INFO - missing file: resource://gre/modules/commonjs/toolkit/loader.js
[task 2018-09-06T13:47:49.021Z] 13:47:49     INFO - missing file: chrome://help/content/help.js referenced from chrome://pippki/content/protectedAuth.xul
[task 2018-09-06T13:47:49.022Z] 13:47:49     INFO - missing file: resource://gre/modules/GeckoViewTelemetryController.jsm referenced from resource://gre/components/ContentProcessSingleton.js
[task 2018-09-06T13:47:49.023Z] 13:47:49     INFO - missing file: resource://services-sync/)%20==%200)%20%7B referenced from resource://services-sync/addonutils.js
Flags: needinfo?(mak77)
we'll just add to the whitelist for now.
Flags: needinfo?(mak77)
Pushed by
Bug 14785825 - Move address bar model tokenization to a module. r=adw
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → Firefox 64
You need to log in before you can comment on or make changes to this bug.