1478582 - Create a QuantumBar Tokenizer module

Status: RESOLVED FIXED

(Firefox :: Address Bar, enhancement, P1)

Description

[Marco Bonardo [:mak]]

We can move the input string tokenizer to a separate module, usable by both the old and new implementation

Comment 1

[Marco Bonardo [:mak]]

Attachment: Bug 14785825 - Move address bar model tokenization to a module.

This is a first stab at the new tokenizer.
It's not expected to be perfect yet, but good enough to be modified and replace the existing code in unifiedComplete with just a few modifications.
It's mostly intended to start setting up a code and tests structure.

Comment 2

[Marco Bonardo [:mak]]

Drew, I'd like to land something, I may not be able to do further work on it before PTO, anyway nothing is using this yet, we can look into replaceing the tokenization in unifiedComplete.js with this, likely with a few modifications or changes (like support for combined restriction chars).
It's likely we'll have to modify it anyway when we have more pieces of the puzzle.
But the naming and tests structure should be fine regardless.

Comment 3

[Drew Willcoxon :adw]

Comment on attachment 8998138 [details]
Bug 14785825 - Move address bar model tokenization to a module.

Drew Willcoxon :adw has approved the revision.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c291d6d95022
Bug 14785825 - Move address bar model tokenization to a module. r=adw

Comment 5

[Cristina Coroiu [:ccoroiu]]

Backed out changeset c291d6d95022 (bug 1478582) for browser-chome failures at browser/base/content/test/static/browser_all_files_referenced.js

Backout: https://hg.mozilla.org/integration/autoland/rev/84d78f630a12c74824f7e0a3cca92136128f2d40

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c291d6d950222a55e9c717993c95daad52c02bd8

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=197833811&repo=autoland&lineNumber=2376

[task 2018-09-06T13:47:48.976Z] 13:47:48     INFO - indirectly whitelisted file: chrome://passwordmgr/content/recipes.json used from resource://gre/greprefs.js
[task 2018-09-06T13:47:48.977Z] 13:47:48     INFO - indirectly whitelisted file: resource://formautofill/FormAutofillSync.jsm used from resource://services-sync/service.js
[task 2018-09-06T13:47:48.978Z] 13:47:48     INFO - indirectly whitelisted file: chrome://global/locale/AccessFu.properties used from resource://gre/modules/accessibility/Utils.jsm
[task 2018-09-06T13:47:48.979Z] 13:47:48     INFO - indirectly whitelisted file: resource://services-sync/bookmark_repair.js used from resource://services-sync/collection_repair.js
[task 2018-09-06T13:47:48.980Z] 13:47:48     INFO - indirectly whitelisted file: resource://services-sync/doctor.js used from resource://services-sync/bookmark_repair.js,resource://services-sync/stages/enginesync.js
[task 2018-09-06T13:47:48.983Z] 13:47:48     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Traversal.jsm used from resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-09-06T13:47:48.984Z] 13:47:48     INFO - Buffered messages finished
[task 2018-09-06T13:47:48.985Z] 13:47:48     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | there should be no unreferenced files - Got 1, expected 0
[task 2018-09-06T13:47:48.986Z] 13:47:48     INFO - Stack trace:
[task 2018-09-06T13:47:48.987Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:test_is:1304
[task 2018-09-06T13:47:48.989Z] 13:47:48     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:761
[task 2018-09-06T13:47:48.990Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-06T13:47:48.995Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-06T13:47:48.995Z] 13:47:48     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-06T13:47:48.997Z] 13:47:48     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-09-06T13:47:48.998Z] 13:47:48     INFO - Not taking screenshot here: see the one that was previously logged
[task 2018-09-06T13:47:48.999Z] 13:47:48     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | unreferenced file: resource://app/modules/UrlbarTokenizer.jsm - 
[task 2018-09-06T13:47:49.000Z] 13:47:49     INFO - Stack trace:
[task 2018-09-06T13:47:49.001Z] 13:47:49     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:765
[task 2018-09-06T13:47:49.002Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-06T13:47:49.005Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-06T13:47:49.006Z] 13:47:49     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-06T13:47:49.006Z] 13:47:49     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-09-06T13:47:49.008Z] 13:47:49     INFO - ignored unused whitelist entry: resource://app/blocklist.xml
[task 2018-09-06T13:47:49.010Z] 13:47:49     INFO - ignored unused whitelist entry: resource://gre/gmp-clearkey/0.1/manifest.json
[task 2018-09-06T13:47:49.012Z] 13:47:49     INFO - ignored unused whitelist entry: resource://gre/res/test.properties
[task 2018-09-06T13:47:49.017Z] 13:47:49     INFO - GECKO(2871) | [Parent 2871, Main Thread] WARNING: Error loading theme icon 'dialog-warning' for stock: Icon 'dialog-warning' not present in theme ubuntu-mono-dark: 'glib warning', file /builds/worker/workspace/build/src/toolkit/xre/nsSigHandlers.cpp, line 141
[task 2018-09-06T13:47:49.019Z] 13:47:49     INFO - GECKO(2871) | (firefox:2871): Gtk-WARNING **: Error loading theme icon 'dialog-warning' for stock: Icon 'dialog-warning' not present in theme ubuntu-mono-dark
[task 2018-09-06T13:47:49.020Z] 13:47:49     INFO - missing file: resource://gre/modules/commonjs/toolkit/loader.js
[task 2018-09-06T13:47:49.021Z] 13:47:49     INFO - missing file: chrome://help/content/help.js referenced from chrome://pippki/content/protectedAuth.xul
[task 2018-09-06T13:47:49.022Z] 13:47:49     INFO - missing file: resource://gre/modules/GeckoViewTelemetryController.jsm referenced from resource://gre/components/ContentProcessSingleton.js
[task 2018-09-06T13:47:49.023Z] 13:47:49     INFO - missing file: resource://services-sync/)%20==%200)%20%7B referenced from resource://services-sync/addonutils.js

Comment 6

[Marco Bonardo [:mak]]

we'll just add to the whitelist for now.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/77f552e1eae4
Bug 14785825 - Move address bar model tokenization to a module. r=adw

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/77f552e1eae4