Closed Bug 1464154 Opened 6 years ago Closed 6 years ago

Most of origins I use very often are not being autofilled

Categories

(Firefox :: Address Bar, defect, P1)

Product:
Firefox
Component:
Address Bar
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox61 --- unaffected
firefox62 + fixed
firefox63 --- fixed

People

(Reporter: marco, Assigned: adw)

References

Details

(Keywords: regression, Whiteboard: [fxsearch])

Attachments

(1 file)

log
1.35 KB, text/plain
Details 
E.g. wordreference.com, web.telegram.org (I use this one multiple times per day), corrieredelmezzogiorno.corriere.it.
tracking-firefox62: --- → ?
Keywords: regression
Thanks for the report, we'll check and ask further questions (we may have to ask you to run a few queries in scratchpad)

You are on the latest Nightly, I suppose.
Blocks: 1239708
one initial question, could you please check if you have a pref named places.database.migrateV48Frecencies?
Flags: needinfo?(mcastelluccio)
Also, do you reuse this profile across different channels of Firefox?
I'm on the latest Nightly, I don't have a pref named "places.database.migrateV48Frecencies", and I only use this profile for Nightly.
Flags: needinfo?(mcastelluccio)
Priority: -- → P1
Whiteboard: [fxsearch]
I think I'm seeing the same, in the last few weeks typing stuff in the awesomebar defaults to "search with google" instead of picking things up from browsing history.
Depends on: 1463132
So far, the provided urls seem to fall into bug 1463132, we'll check again once that is gone.
It doesn't look like web.telegram.org or corrieredelmezzogiorno.corriere.it (comment 0) redirect, though.  web.telegram.org may do some pushState() thing since you end up at web.telegram.org/#/login, but not an HTTP redirect.  So maybe something else in addition to bug 14631320 is going on.
(In reply to Drew Willcoxon :adw from comment #7)
> It doesn't look like web.telegram.org or corrieredelmezzogiorno.corriere.it
> (comment 0) redirect, though. 

http => https?
Other examples are "ilmanifesto.it" or "mozilla.slack.com". Not sure if they fall in the same bucket (maybe http -> https as well).
I had an idea some time ago, but then I forgot to write it down. Adaptive history (moz_inputhistory) has data that could be used to autofill. For example
SELECT input, url
FROM moz_inputhistory i
JOIN moz_places h ON h.id = i.place_id
JOIN moz_origins o ON o.id = h.origin_id
WHERE host BETWEEN input AND input || X'FFFF'
Some of the cases I mentioned are fixed in the latest Nightly (maybe by bug 1463132?).
"wordreference.com", "web.telegram.org", "mozilla.slack.com" are still broken.
Thanks, Marco.  If you don't mind, could you please try the following in your browser console and report the output?  It will print some frecency stats and the origins in your places database that match those three domains you mentioned.

(async function () {
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(`
    WITH s(count, sum, squares) AS (
      SELECT
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_count") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum_of_squares") AS REAL)
    )
    SELECT count, sum, squares, CASE count
    WHEN 0 THEN 0.0
    WHEN 1 THEN sum
    ELSE (sum / count) + sqrt((squares - ((sum * sum) / count)) / count)
    END
    FROM s
  `);
  for (let i = 0; i < 4; i++) {
    console.log(rows[0].getResultByIndex(i));
  }
  rows = await db.execute(`
    SELECT frecency, prefix, host
    FROM moz_origins
    WHERE like("%wordreference.com%", host)
          OR like("%telegram.org%", host)
          OR like("%slack.com%", host)
  `);
  for (let row of rows) {
    console.log(row.getResultByIndex(0),
                row.getResultByIndex(1),
                row.getResultByIndex(2));
  }
})();
Flags: needinfo?(mcastelluccio)
(In reply to Marco Bonardo [::mak] from comment #8)
> (In reply to Drew Willcoxon :adw from comment #7)
> > It doesn't look like web.telegram.org or corrieredelmezzogiorno.corriere.it
> > (comment 0) redirect, though. 
> 
> http => https?

That's not the HTTP redirection recognized by the frecency calculation code though right (301, 308)?  It's HTTPS upgrade or whatever.  If you visit those sites and check the network panel in devtools, there's no redirection going on.
(In reply to Drew Willcoxon :adw from comment #13)
> though right (301, 308)?

Er, those are only permanent redirects and we detect temp too, but my point still stands (I think)
In my case it happens with bugzilla.mozilla.org.  I used to be able to type "bu" or "bug" and have that autocompleted to "bugzilla.mozilla.org/".  That works in beta, but in nightly it only does the "search with google" thing which is horribly annoying.
132609
49676152
142541933430916
33158.193338713514
8280000 https:// web.telegram.org
4317 http:// www.wordreference.com
553 http:// wordreference.com
64296 https:// uplift-bugs.slack.com
126 https:// coveralls.slack.com
54 https:// forum.wordreference.com
11161 https:// mozilla.slack.com
20 https:// slack.com
20 https:// telegram.org
20 https:// instantview.telegram.org
20 https:// join.slack.com
466 https:// dev-productivity.slack.com
20 http:// dev-productivity.slack.com
20 https:// developersitalia.slack.com
20 http:// developersitalia.slack.com
26208 https:// zest-uzh.slack.com
Flags: needinfo?(mcastelluccio)
(In reply to Julien Cristau [:jcristau] from comment #15)
> In my case it happens with bugzilla.mozilla.org.  I used to be able to type
> "bu" or "bug" and have that autocompleted to "bugzilla.mozilla.org/".  That
> works in beta, but in nightly it only does the "search with google" thing
> which is horribly annoying.

Please adapt the script in comment 12 and post the results.
Flags: needinfo?(jcristau)
Drew, what if due to bug 1463512 (or bug 1463017) we didn't actually update frecency for some origins? We could have submitted the transaction and flipped the pref, but then merging the wal failed.
Also, could we calculate the threshold out of moz_origins instead of moz_places? that would probably be more efficient and reliable, since we select from that table.
Though, we seem to do a full recalculation of the origin frecency when a frecency in moz_places changes (we could probably be more efficient by keeping the new frecency if > currently stored one, recalculate otherwise). That would override any non-calculated value. So, it looks like the only thing that may be wrong long term (until the first maintenance) is stats.
44495
8113574
18394036735570
20513.66222311852
21294 https:// bugzilla.mozilla.org
4904 http:// bugzilla.mozilla.org
Flags: needinfo?(jcristau)
missed the %s the first time around (doesn't seem to make much difference, though).

44495
8113674
18394036865570
20513.66452225664
21294 https:// bugzilla.mozilla.org
4904 http:// bugzilla.mozilla.org
20 https:// bug623317.bugzilla.mozilla.org
Marco, could you please run this one?

(async function () {
  let hosts = [
    "wordreference.com",
    "web.telegram.org",
    "mozilla.slack.com",
  ];
  let sqlFrags = [];
  for (let host of hosts) {
    for (let prefix of ["http://", "https://"]) {
      sqlFrags.push(`
        SELECT "${prefix}", "${host}" AS host,
        (
          SELECT MAX(frecency)
          FROM moz_places
          WHERE like("${prefix}${host}%", url)
        ),
        (
          SELECT frecency
          FROM moz_origins
          WHERE prefix = "${prefix}" AND host = "${host}"
        )
      `);
    }
  }
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(
    sqlFrags.join(" UNION ") + " ORDER BY host ASC"
  );
  for (let row of rows) {
    console.log(row.getResultByIndex(0),
                row.getResultByIndex(1),
                row.getResultByIndex(2),
                row.getResultByIndex(3));
  }
})();
Flags: needinfo?(mcastelluccio)
http:// mozilla.slack.com null null
https:// mozilla.slack.com 10882 10882
http:// web.telegram.org null null
https:// web.telegram.org 8298000 8298000
http:// wordreference.com 539 539
https:// wordreference.com null null
Flags: needinfo?(mcastelluccio)
(In reply to Marco Castelluccio [:marco] from comment #16)
> 33158.193338713514
> 8280000 https:// web.telegram.org

(In reply to Julien Cristau [:jcristau] from comment #21)
> 20513.66452225664
> 21294 https:// bugzilla.mozilla.org

According to this, web.telegram.org and bugzilla.mozilla.org should be autofilling for each of you respectively.  They both clear your respective frecency thresholds.

If you start typing "web.telegram.org" or "bugzilla.mozilla.org" character by character, is it really not autofilling at some point?  There may be a similar origin that shares some prefix with it that has a higher frecency, and if that's the case, that other origin will win and be autofilled first, up to the shared prefix.

If these really aren't autofilling, we have some problem unrelated to frecencies.  I'll ruminate on this.

(In reply to Marco Castelluccio [:marco] from comment #23)

Thanks for the quick reply!  All the frecency data here looks in order.  According to this, web.telegram.org should be autofilling as I mentioned (but I understand it's not), but mozilla.slack.com and wordreference.com don't meet the autofill threshold, so it's "expected" that they don't autofill.

We may need to tweak what "expected" means.  If you do actually visit mozilla.slack.com and wordreference.com often/recently, their frecencies should naturally increase enough to meet the threshold at some point.  We may need to take a closer look at how frecencies are boosted, or exclude frecency outliers when calculating the autofill threshold (web.telegram.org has a much larger frecency than mozilla.slack.com and wordreference.com for example), or something else...
Could you please report the values of these prefs?

browser.urlbar.suggest.history
browser.urlbar.suggest.history.onlyTyped
browser.urlbar.suggest.bookmark
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(jcristau)
It may also be interesting to know how you actually visit those pages, because for example session-restore and app-tabs may register visits differently from typing in the urlbar or clicking a bookmark. As well as maybe you often access wordreference through a keyword, rather than from its homepage?
The more we know about the way you access those pages, the better we can tweak things.
One more snippet to please try.  It prints all the origins that meet the autofill frecency threshold.  These are all the origins that should autofill when you start typing them.

You don't need to paste the whole output here, but please do paste the last line, which is a count of the origins that meet the threshold.

Marco, are there domains in the list other than web.telegram.org that don't autofill?

Julien, are there domains other than bugzilla.mozilla.org that don't autofill?

(async function () {
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(`
    WITH
    frecency_stats(count, sum, squares) AS (
      SELECT
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_count") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum_of_squares") AS REAL)
    ),
    autofill_frecency_threshold(value) AS (
      SELECT MAX(1,
        CASE count
        WHEN 0 THEN 0.0
        WHEN 1 THEN sum
        ELSE (sum / count) + sqrt((squares - ((sum * sum) / count)) / count)
        END
      ) FROM frecency_stats
    )
    SELECT frecency, prefix, host
    FROM moz_origins
    WHERE frecency >= (SELECT value FROM autofill_frecency_threshold)
    ORDER BY frecency DESC
  `);
  for (let row of rows) {
    console.log(row.getResultByIndex(0),
                row.getResultByIndex(1),
                row.getResultByIndex(2));
  }
  let meetThresholdCount = rows.length;
  rows = await db.execute(`
    SELECT count(*) FROM moz_origins
  `);
  let allCount = rows[0].getResultByIndex(0);
  console.log(meetThresholdCount,
              allCount,
              (meetThresholdCount / allCount * 100) + "%");
})();
Assignee: nobody → adw
Status: NEW → ASSIGNED
(In reply to Drew Willcoxon :adw from comment #24)
> (In reply to Marco Castelluccio [:marco] from comment #16)
> > 33158.193338713514
> > 8280000 https:// web.telegram.org
> 
> (In reply to Julien Cristau [:jcristau] from comment #21)
> > 20513.66452225664
> > 21294 https:// bugzilla.mozilla.org
> 
> According to this, web.telegram.org and bugzilla.mozilla.org should be
> autofilling for each of you respectively.  They both clear your respective
> frecency thresholds.
> 
> If you start typing "web.telegram.org" or "bugzilla.mozilla.org" character
> by character, is it really not autofilling at some point?  There may be a
> similar origin that shares some prefix with it that has a higher frecency,
> and if that's the case, that other origin will win and be autofilled first,
> up to the shared prefix.
> 
> If these really aren't autofilling, we have some problem unrelated to
> frecencies.  I'll ruminate on this.

"web.telegram.org" is the one with the highest frecency value.
When I typed "we", nothing was being autofilled.

> (In reply to Marco Castelluccio [:marco] from comment #23)
> 
> Thanks for the quick reply!  All the frecency data here looks in order. 
> According to this, web.telegram.org should be autofilling as I mentioned
> (but I understand it's not), but mozilla.slack.com and wordreference.com
> don't meet the autofill threshold, so it's "expected" that they don't
> autofill.
> 
> We may need to tweak what "expected" means.  If you do actually visit
> mozilla.slack.com and wordreference.com often/recently, their frecencies
> should naturally increase enough to meet the threshold at some point.  We
> may need to take a closer look at how frecencies are boosted, or exclude
> frecency outliers when calculating the autofill threshold (web.telegram.org
> has a much larger frecency than mozilla.slack.com and wordreference.com for
> example), or something else...

web.telegram.org is now being autofilled in the latest Nightly (it wasn't working earlier today).
wordreference.com still doesn't work.

I don't visit mozilla.slack.com often, but I do visit wordreference.com often.

(In reply to Drew Willcoxon :adw from comment #25)
> Could you please report the values of these prefs?
> 
> browser.urlbar.suggest.history
> browser.urlbar.suggest.history.onlyTyped
> browser.urlbar.suggest.bookmark

browser.urlbar.suggest.history -> true
browser.urlbar.suggest.history.onlyTyped -> false
browser.urlbar.suggest.bookmark -> true

(In reply to Marco Bonardo [::mak] from comment #26)
> It may also be interesting to know how you actually visit those pages,
> because for example session-restore and app-tabs may register visits
> differently from typing in the urlbar or clicking a bookmark. As well as
> maybe you often access wordreference through a keyword, rather than from its
> homepage?
> The more we know about the way you access those pages, the better we can
> tweak things.

For "wordreference.com" (which is now the last one not working for me which definitely regressed), I have a weird way to use it.
I type "wo" in the address bar, then select one of the entries (e.g. "http://www.wordreference.com/iten/denunciato") then remove the word (e.g. "denunciato") and replace it with the word I want to lookup.
Given the way I use it, I don't really care if this particular case is fixed, but it is a regression.

(In reply to Drew Willcoxon :adw from comment #27)
> One more snippet to please try.  It prints all the origins that meet the
> autofill frecency threshold.  These are all the origins that should autofill
> when you start typing them.
> 
> You don't need to paste the whole output here, but please do paste the last
> line, which is a count of the origins that meet the threshold.

44 7687 0.5723949525172369%

> Marco, are there domains in the list other than web.telegram.org that don't
> autofill?

Not anymore, as far as I can see.
Flags: needinfo?(mcastelluccio)
(In reply to Marco Castelluccio [:marco] from comment #28)
> web.telegram.org is now being autofilled in the latest Nightly (it wasn't
> working earlier today).

I'm not sure what to make of that.  It's possible that the update was coincidental with the frecency stats recalculation that happens on idle, and the recalculation fixed the threshold, which fixed the autofill.  But that doesn't really make sense because the stats you pasted in comment 16 already indicated that web.telegram.org should have been autofilled.
(In reply to Marco Castelluccio [:marco] from comment #28)
> For "wordreference.com" (which is now the last one not working for me which
> definitely regressed), I have a weird way to use it.
> I type "wo" in the address bar, then select one of the entries (e.g.
> "http://www.wordreference.com/iten/denunciato") then remove the word (e.g.
> "denunciato") and replace it with the word I want to lookup.
> Given the way I use it, I don't really care if this particular case is
> fixed, but it is a regression.

I see, so you have a lot of _different_ typed urls that point to this origin. IIRC the frecency we pick for the origins is the max among urls, it may not be the best choice for this case.
(In reply to Drew Willcoxon :adw from comment #25)
> Could you please report the values of these prefs?
> 
browser.urlbar.suggest.history=true
browser.urlbar.suggest.history.onlyTyped=false
browser.urlbar.suggest.bookmark=true
I'm seeing bugzilla.mozilla.org get autofilled after updating today (just typing "b" is enough).
Flags: needinfo?(jcristau)
So, we may investigate a new way to calculate origins frecency, rather than just picking the max, and then calculate a threshold out of moz_origins, rather than moz_places. If we'd land this in nightly, nightly users would get another migration, while other channels would just use the latest algorithm. Thus, this is something to figure out soon.
(In reply to Marco Bonardo [::mak] from comment #33)
> So, we may investigate a new way to calculate origins frecency, rather than
> just picking the max, and then calculate a threshold out of moz_origins,
> rather than moz_places. If we'd land this in nightly, nightly users would
> get another migration, while other channels would just use the latest
> algorithm. Thus, this is something to figure out soon.

Thinking about it, I actually do the same with bugzilla.mozilla.org, but this one is being autofilled.
See Also: → 1464270
Autofill should take adaptive history into account.

(In reply to Marco Bonardo [::mak] from comment #10)
> I had an idea some time ago, but then I forgot to write it down. Adaptive
> history (moz_inputhistory) has data that could be used to autofill.

I filed bug 1466233.  See also bug 1464270.
Looks like something changed over the weekend and bugzilla stopped being autocompleted for me yesterday.
Do you have one or more bookmark keywords on bugzilla? For example if you have a "bug" keyword and you type "bug", it's NOT supposed to autofill, since it's a keyword.
I do not.
Julien, could you please re-run the script in comment 12, replacing wordreference.com with bugzilla.mozilla.org as you did before?  Right before you do that, please verify again that bugzilla still does not autofill.

And then please run the following.  Thanks so much for helping us with this.

(async function () {
  const SQL_AUTOFILL_WITH = `
    WITH
    frecency_stats(count, sum, squares) AS (
      SELECT
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_count") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "frecency_sum_of_squares") AS REAL)
    ),
    autofill_frecency_threshold(value) AS (
      SELECT MAX(1,
        CASE count
        WHEN 0 THEN 0.0
        WHEN 1 THEN sum
        ELSE (sum / count) + sqrt((squares - ((sum * sum) / count)) / count)
        END
      ) FROM frecency_stats
    )
  `;
  const SQL_AUTOFILL_FRECENCY_THRESHOLD = `(
    SELECT value FROM autofill_frecency_threshold
  )`;
  function originQuery(conditions = "", bookmarkedFragment = "NULL") {
    return `${SQL_AUTOFILL_WITH}
            SELECT :query_type,
                   host || '/',
                   prefix || host || '/',
                   frecency,
                   ${bookmarkedFragment} AS bookmarked,
                   id
            FROM moz_origins
            WHERE host BETWEEN :searchString AND :searchString || X'FFFF'
                  AND frecency >= ${SQL_AUTOFILL_FRECENCY_THRESHOLD}
                  ${conditions}
            UNION ALL
            SELECT :query_type,
                   fixup_url(host) || '/',
                   prefix || host || '/',
                   frecency,
                   ${bookmarkedFragment} AS bookmarked,
                   id
            FROM moz_origins
            WHERE host BETWEEN 'www.' || :searchString AND 'www.' || :searchString || X'FFFF'
                  AND frecency >= ${SQL_AUTOFILL_FRECENCY_THRESHOLD}
                  ${conditions}
            ORDER BY frecency DESC, id DESC
            LIMIT 1 `;
  }
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(originQuery(), {
    query_type: 1,
    searchString: "bug",
  });
  console.log("Row count:", rows.length);
  for (let row of rows) {
    console.log(row.getResultByIndex(1),
                row.getResultByIndex(2),
                row.getResultByIndex(3));
  }
})();
Flags: needinfo?(jcristau)
Depends on: 1467627
On nightly 20180606220131

From the script in comment 12:

45235
8977829
22581355232563
22540.394785785466
19244 https:// bugzilla.mozilla.org
4319 http:// bugzilla.mozilla.org
20 https:// bug623317.bugzilla.mozilla.org

From the script in comment 40:

Row count: 0
Flags: needinfo?(jcristau)
This output makes more sense than the earlier one.  The frecency threshold is 22540, but bugzilla falls below that at 19244, so it's not being autofilled.  It's working as expected right now (but of course that doesn't mean it's optimal). Bug 1467631 may improve the situation for you so that if you're often reaching bugzilla by typing it into the urlbar, or by starting to type it and then selecting a result, its frecency will get boosted enough to clear the threshold.

What I'm really interested in seeing is when bugzilla is above the threshold yet not being autofilled, as your earlier comments indicated.  If you happen to see that happening -- and running that first script is the only way you'd see that, so I don't really expect that you'll see it -- then running the second script and letting me know the output would be helpful.

One interesting thing about that ouput though is that bugzilla does clear the threshold when you sum the frecencies of its http and https entries.  Maybe we should consider http + https as a single entry when comparing against the threshold, but then take the higher entry when actually autofilling, in this case https.
I'm curious why bugzilla would ever fall below the threshold, given it's probably by far the origin I visit most (though maybe not type, these days, due to this bug).
(In reply to Julien Cristau [:jcristau] from comment #43)
> I'm curious why bugzilla would ever fall below the threshold, given it's
> probably by far the origin I visit most (though maybe not type, these days,
> due to this bug).

It's not about the fact you visit it a lot, but it's about how you visit it. For example you may not always enter from the same page, and since we currently use the max frecency in the origin, it won't account for those multiple entry points.
Bug 1467627 will likely solve this.
The bugzilla case (from comment 41) is kind of interesting.  bugzilla has a big frecency, but it's split across two origins, one for http and another for https, and each has a frecency that's too low to cross the threshold.  Ideally, what's supposed to happen is that you consistently visit one over the other, so its frecency keeps going up, and eventually it crosses the threshold for autofill, leading to a positive cycle where it's cemented into place because it keeps getting autofilled and you keep visiting it.  If instead you tend to visit both, then it might happen that neither crosses the threshold soon.  (http://bugzilla redirects to https, so by "visit" I mean the one that you choose at first.  Ultimately you end up on https regardless.)

When I say that that's what's "supposed" to happen, I make it sound like you're doing something wrong, but of course that's not true.  Firefox should offer one over the other -- in your case, the https one since it has a higher frecency (or maybe it should offer https over http).

I'm not sure I agree with Marco that bug 1467627 will fix this.  With that bug fixed, you could still end up in this situation if you consistently visit both http and https.  Marco and I talked about this case in particular, and there are some other strategies we can try that all involve combining the frecencies of all origins of a domain instead of treating them separately.  Doing that would make bugzilla clear the threshold in this case.
(In reply to Drew Willcoxon :adw from comment #45)
> (or maybe it should offer https over http).

*always offer https over http
(In reply to Drew Willcoxon :adw from comment #45)
> I'm not sure I agree with Marco that bug 1467627 will fix this.

I filed bug 1469651 specifically for this.
Depends on: 1469651
Trying to understand the latest status of this bug -- do we still consider it worth blocking 62 over?
Flags: needinfo?(mak77)
Marco C. and Julien, I expect that this bug is now significantly better for you now that bug 1467627 and bug 1469651 have landed on Nightly.  Could you try your problematic cases from before and let us know whether things are better?

The only thing I'm still wondering about is the case where a domain clears the autofill threshold but still isn't autofilled.  That's just a straight up bug, and I'm still not sure why that would happen, but maybe one of these two bugs fixed it.
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(jcristau)
wordreference.com is still not being autofilled.
Flags: needinfo?(mcastelluccio)
(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #48)
> Trying to understand the latest status of this bug -- do we still consider
> it worth blocking 62 over?

Bug 1467627 and bug 1469651 should have made this better, but obviously I'm still trying to figure out what's going on in Marco C.'s case.  I don't think it makes sense to track this bug.  It makes more sense to track actionable bugs that we spun off from this one, like bug 1467627 and bug 1469651, which are both tracking 62.
Flags: needinfo?(mak77)
Flags: needinfo?(mcastelluccio)
Sorry, Marco, please try this one instead:

(async function () {
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(`
    WITH s(count, sum, squares) AS (
      SELECT
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "origin_frecency_count") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "origin_frecency_sum") AS REAL),
        CAST((SELECT IFNULL(value, 0.0) FROM moz_meta WHERE key = "origin_frecency_sum_of_squares") AS REAL)
    )
    SELECT count, sum, squares, CASE count
    WHEN 0 THEN 0.0
    WHEN 1 THEN sum
    ELSE (sum / count) + sqrt((squares - ((sum * sum) / count)) / count)
    END
    FROM s
  `);
  for (let i = 0; i < 4; i++) {
    console.log(rows[0].getResultByIndex(i));
  }
  console.log("*** origins:");
  rows = await db.execute(`
    SELECT frecency, prefix, host
    FROM moz_origins
    WHERE like("%wordreference.com%", host)
  `);
  for (let row of rows) {
    console.log(row.getResultByIndex(0),
                row.getResultByIndex(1),
                row.getResultByIndex(2));
  }
  let origins = [
    "http://wordreference.com",
    "http://www.wordreference.com",
    "https://wordreference.com",
    "https://www.wordreference.com",
  ];
  for (let origin of origins) {
    console.log(`*** ${origin} places:`);
    rows = await db.execute(`
      SELECT frecency
      FROM moz_places
      WHERE like("${origin}%", url)
    `);
    for (let row of rows) {
      console.log(row.getResultByIndex(0));
    }
    rows = await db.execute(`
      SELECT TOTAL(frecency)
      FROM moz_places
      WHERE frecency > 0 AND like("${origin}%", url)
    `);
    console.log("total:", rows[0].getResultByIndex(0));
  }
})();
Flags: needinfo?(mcastelluccio)
Attached file log — 
Flags: needinfo?(mcastelluccio)

    
    

    
  
bmo seems to get autocompleted for me today. It wasn't last week.
Flags: needinfo?(jcristau)

    
    

    
  
(In reply to Drew Willcoxon :adw from comment #49)
> Bug 1467627 and bug 1469651 should have made this better, but obviously I'm
> still trying to figure out what's going on in Marco C.'s case.  I don't
> think it makes sense to track this bug.  It makes more sense to track
> actionable bugs that we spun off from this one, like bug 1467627 and bug
> 1469651, which are both tracking 62.

Makes sense to me. Let's let relman flip the tracking flag.

          status-firefox62:
          affected → fix-optional

    
    

    
  
(In reply to Marco Castelluccio [:marco] from comment #54)
> Created attachment 8990683 [details]
> log

Thanks Marco.  I'm sorry to say autofill is working as designed.  It looks like bug 1467627 and bug 1469651 fixed the other sites you've mentioned in this bug (and Julien's bmo case), but wordreference.com just doesn't have a big enough frecency to clear the autofill threshold.

I'm starting to think we should lower the threshold.  It's better to err on the side of autofilling too much rather than too little.  We're erring on the side of too little right now, it seems clear to me.  That's especially frustrating since the previous autofill algorithm filled too much, so we're basically regressing people's autofill right now.

Lower it to maybe just the average frecency, instead of the average + one standard deviation.

    
    

    
  
(In reply to Drew Willcoxon :adw from comment #57)
> I'm starting to think we should lower the threshold.
>
> ...
> 
> Lower it to maybe just the average frecency, instead of the average + one
> standard deviation.

I filed bug 1474755 for this

    
  
Depends on: 1474755

    
    

    
  
(In reply to Marco Castelluccio [:marco] from comment #54)
> Created attachment 8990683 [details]
> log

> *** origins:
> 96174 http:// www.wordreference.com
> 15037 http:// wordreference.com
> ...
> 49564 https:// www.wordreference.com

One other thing that comes to mind looking at this data is that we could collapse www and non-www versions of the same domain, the way we now do with http and https versions.  That still would not cause wordreference.com to autofill in this particular data (and ignoring bug 1474755), but it very well could if the threshold in similar data happened to be lower.

    
    

    
  
Personally I'm not thrilled of lowering the threshold, until we collect more feedback from users.
Autofilling more would probably go towards the goal of covering the old behavior, but we're not completely sure the old behavior was a good example. Autofilling has also disadvantages when it does the wrong thing, and that was the main reason for the algorithm change.
I'd rather prefer to spend time improving frecency in general, it's possible the problem is not the threshold, but the way you visit the page is not properly accounted in frecency, or alternatively, we are bumping too much frecency of all the rest, watering down average results.

I think we should look into things like bug 1467631 before touching the threshold.
This is just my opinion clearly, not intended to be a stop-energy.

Anyway, imo this bug is fixed, and edge cases should now be filed and tracked apart.

    
    

    
  
Also, for the wordreference case I'd suggesting setting up a keyword or a search engine alias (if wordreference has an opensearch definition), it doesn't sound like a good example for autofill.

    
    

    
  
OK based on comment 60 I'm marking this fixed. (And fixed for 62 as well since we landed work from bug 1467627 and bug 1469651 in beta 62).

Marco, would you mind filing a followup for the wordreference case?
Status: ASSIGNED → RESOLVED
Closed: 6 years ago

          status-firefox62:
          fix-optional → fixed

          status-firefox63:
          --- → fixed
Flags: needinfo?(mcastelluccio)
Resolution: --- → FIXED

    
  
See Also:  → 1476522

    
    

    
  
I've filed bug 1476522.
Flags: needinfo?(mcastelluccio)

    
    

    
  
Bug 1474755 has landed and should help this, too.

    
    

    
  
Can we merge fragments?

In my history, I have a bunch of things from "webaudio API", but when I try to visit them my awesomebar is polluted with the same link over and over again with different fragment specifiers. https://user-images.githubusercontent.com/1617736/43021777-caf94462-8c19-11e8-9877-a3620002c118.png

    
    

    
  
That's a different component from this bug and "autofill," which means that the urlbar textbox automatically completes the domain or URL that you're typing.  There's probably already a bug on file somewhere, but feel free to file one for that.  (FWIW I agree with you)

    
    

    
  
I'm not typing a URL, I'm typing a title. I have things on github.com, reviewable.io, and web-audio-api.github.io with the HTML title "webaudio API", but i mostly get the latter two.

    
    

    
  
I understand, I'm saying that's different from what this bug is about and what autofill is.
See Also:  → 1477411

    
    

    
  
Is manual testing from QE needed here? If yes, is there any suggested STR you can provide us in order to replicate/verify this particular bug?
Flags: qe-verify?
Flags: needinfo?(adw)

    
    

    
  
This is covered by other bugs you've already verified, so I'll set this one to qe-verify-.  Thanks!
Flags: qe-verify?
Flags: qe-verify-
Flags: needinfo?(adw)

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: