1204361 - New Tab Page doesn't honor my "x" clicks on Suggested Tiles; instead of removing the tile, it swaps in a very similar one [until I exhaust all the options?]

Status: VERIFIED FIXED

(Firefox :: New Tab Page, defect, P1)

Description

[Daniel Holbert [:dholbert]]

I just returned from vacation and updated Firefox Nightly and opened it for the first time in a few weeks.

Now, the first tile on my New Tab Page is for some random pop-culture news blurb which I'm not interested in. The caption below it is "An Entertainment Site Suggestion".

I clicked its "x" to dismiss it, but it was simply replaced with a different news blurb (still showing "An Entertainment Site Suggestion"). I had to repeat this (clicking the "X", watch it be replaced by some other suggestion) about 5 times.

EXPECTED RESULTS: When I click the "X" on a suggested tile, it should be dismissed, rather than being replaced by a similar-but-subtly-different tile. If I click the "x", it's a pretty good signal that I don't want to see that sort of thing.

Comment 1

[Daniel Holbert [:dholbert]]

Is this the intended flow? i.e. do we actually *intend* to replace suggested tiles with other similar suggested tiles? If so, I think we should seriously reconsider this strategy, because that's a *really* good way to confirm users' bad feelings about "boo, Firefox is putting ads on the new tab page".

All along, we've been saying "Don't worry, this will be high-quality stuff, and if you don't like it, you can remove it just by clicking the x".  But if clicking the "x" [asking for the tile to go away] ends up simply making us swap in some extremely-similar alternative tile, then that seems like a violation of that promise to me.

Comment 2

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #1)
> Is this the intended flow? i.e. do we actually *intend* to replace suggested
> tiles with other similar suggested tiles?

[...when the user clicks the "x", I mean]

Comment 3

[Daniel Holbert [:dholbert]]

This was particularly bothersome because I had the exact same experience on two different (synced) computers, BTW. (And I expect I'll have the exact same experience again tomorrow when I go into the office and open Firefox Nightly on my work desktop. :-/  )

Comment 4

[Ed Lee :Mardak]

What's the expected behavior with trying to remove suggestions?

Comment 5

[Daniel Holbert [:dholbert]]

(Oddly, just now -- opening my browser this morning on one machine where I'd x'd out An Entertainment Site Suggestion yesterday -- I saw this tile on my new tab page again, for a few seconds. I opened 3 new tabs, each of them had either "An Entertainment Site Suggestion" or "A Food & Lifestyle Site Suggestion" [no idea how I was categorized as being interested in reading about those things].  BUT, then when I switched back to those tabs a second later, the suggested tile had disappeared. Very odd. Might be its own separate bug, but I have no idea how to reproduce reliably, and it seems like it may be related to the "persistence" of these tiles mentioned above, so I'm just noting it here for now.)

Comment 6

[Daniel Holbert [:dholbert]]

OK, I just saw the comment 5 behavior on my work Desktop machine, where I had not yet X'ed anything out yet.  That is: a few New Tabs that I opened each showed an Entertainment / Food & Lifestyle suggestion [in the prime first-unpinned-tile slot], and then a few seconds later (after a reload) the suggestion was gone.

Maybe that's what I was seeing in comment 0 as well, and the suggested tile appearance was untintended and brief [albeit and unresponsive to "x" actions while it was there]? This is all seeming very mysterious.

Comment 7

[Kevin Ghim]

Daniel, enhanced blocking part of our roadmap which has dependencies on entities like advertiser/brand, campaign being created and managed during the tile serving process. When the user clicks on the X, we can block by brand, campaign, category or individual tile level. If the user isn't turning off suggestions as a whole, we'd need to ask what the user would like to not see. Eventually, we could have a user maintained preferences stored on the client or associated with accounts to manage or dynamically update these preferences. 

What was your expectations when you clicked on the X - to not see the brand, campaign, category or tile? And for how long?

Comment 8

[Daniel Holbert [:dholbert]]

(In reply to Kevin Ghim from comment #7)
> What was your expectations when you clicked on the X - to not see the brand,
> campaign, category or tile?

For the tile to disappear (just like with any other tile) -- not for its content to be replaced (which is what happened).

> And for how long?

Not sure about that part. But at least a few minutes, maybe at least a day (maybe longer).

Comment 9

[Daniel Holbert [:dholbert]]

...and by when I say I expect the tile to disappear, I of course mean "disappear and have the other visible tiles slide leftwards/upwards to fill its now-vacant place [leaving a blank spot at the bottom-right, which will now get populated with some other tile].

I think part of the issue here might be that we might (?) be "reserving" this #1 slot for suggested tiles. If that's the case, it means that the "x" button works differently for suggested tiles in this slot, which is sub-optimal from a user-experience perspective.

Comment 10

[Daniel Holbert [:dholbert]]

[Tracking Requested - why for this release]: Annoying UI interaction ("x" button seemingly not working to remove a suggested tile) in prominent piece of UI (new tab page)

Comment 11

[Daniel Holbert [:dholbert]]

Attachment: screencast with suggested tile being replaced instead of removed when I click the "x". [sometimes this happens over and over and over]

Just saw a suggested tile come up, and took a screencast to demonstrate this bug.

As you can see in the attached screencast:
 (1) The suggested tile shows up at prominent "#1 unpinned tab" spot.
 (2) When I click the "x" on the suggested tile, it doesn't go away -- it's just replaced with different content.
 (3) When I click the "x" *again*, it finally disappears. (But as noted in comment 0, sometimes I have to x out 4 or 5 different pieces of content before this happens.)

Comment 12

[Kevin Ghim]

(In reply to Daniel Holbert [:dholbert] from comment #11)
> Created attachment 8661307 [details]
> screencast
> 
> Just saw a suggested tile come up, and took a screencast to demonstrate this
> bug.
> 
> As you can see in the attached screencast:
>  (1) The suggested tile shows up at prominent "#1 unpinned tab" spot.
>  (2) When I click the "x" on the suggested tile, it doesn't go away -- it's
> just replaced with different content.
>  (3) When I click the "x" *again*, it finally disappears. (But as noted in
> comment 0, sometimes I have to x out 4 or 5 different pieces of content
> before this happens.)

Hi Daniel, thanks for documenting behavior. We're prioritizing on working on a solution. Will keep you updated.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Not sure I need to track this as it sounds like the team isn't certain if it's intended behavior. Or, we may be treating the behavior as something to be tweaked as we go along. If so, it would be good to write up changes in the specs as we go, so that QE (and users) will understand.  

I'll go ahead and track for now.

Comment 14

[Daniel Holbert [:dholbert]]

(Not to prod, but any thoughts on a solution here?  We should try our best fix this before it hits beta, IMO.  We need to respect users' "x" actions by removing the suggested tile slot for a period of time, instead of just swapping in some different suggested content over and over and over again.

As noted in previous comments, the current workflow ends up feeling like:
 "I don't want to see this right now -- 'x', go away. Grr, I said 'x', go away.  Grrrrr, 'x' go away!!!" [repeated N times, where N is the number of suggested tiles we have queued]

There's a pretty high potential for user frustration here.)

Comment 15

[Kevin Ghim]

Daniel, unfortunately because of the Remote New Tab project (https://bugzilla.mozilla.org/showdependencytree.cgi?id=1176429&hide_resolved=1), any work on new tab needs to be frozen. We have a solution for it, but it's delayed by RNT. Oyiptong can provide more info.

Comment 16

[maxim zhilyaev]

I suggest half-ass solution for now.  As soon as a user blocks a suggested tile, we stop showing suggested tiles for the next 24 hours. This change will require only mild code modification, could be ported in RNT easily, and will avoid constant user irritation.  Since blocks are rare (at most 2% of the impressions), this fix should not reduce inventory in any material way.

Comment 17

[Kevin Ghim]

Tim, above.

Comment 18

[Olivier Yiptong [:oyiptong]]

I think blocking a suggested tile should block the suggested tiles. The port to Remote New Tab is fine as this provides a bad UX.
The solution max proposes sounds fine for now, until we have more blocking options.

To give some perspective on the frequency of this event, our max-monthly block-through-rate is actually at ~0.7% of impressions so far this year. It's a fairly rare event, however, it is also our most common user action other than impressions.

Comment 19

[Olivier Yiptong [:oyiptong]]

We have a feature freeze, but I think a bug fix is fine

Comment 20

[Tim Spurway [:tspurway]]

I am fine with this fix, though I wonder if we could also look into fixing this feature properly by utilizing a menu instead of trying to guess what the user is trying to do:

- stop showing this tile
- stop showing tiles from this content provider
- stop showing all suggested content

Comment 21

[maxim zhilyaev]

Agree with Tim, however we only have a month to duck tape before 43 becomes beta.
I suggest going with the simplest fix and figuring out a canonical way later.

Comment 22

[Daniel Holbert [:dholbert]]

Attachment: screencast #2 (with 4 "movie content suggestions")

In case it's helpful to illustrate the current user experience here, here's a better screencast of me running into this, with a larger backlog of suggested tiles than in my previous screencast.  This one's got 4 that I have to churn through.

All 4 tiles are in the same category ("A movie content suggestion").

Comment 23

[maxim zhilyaev]

Attachment: v1. postponing suggested tile recomendation after a block

the fix is to stop recommending suggested tile for 1 days after it was blocked

Comment 24

[Marcos Caceres [:marcosc]]

Comment on attachment 8675219 [details] [diff] [review]
v1. postponing suggested tile recomendation after a block

Review of attachment 8675219 [details] [diff] [review]:
-----------------------------------------------------------------

Generally, this looks ok to me (so f+ from me), but I'm not very knowledgeable of DirectoryLinksProvider.jsm. Marina is probably the best person to provide the actual review.

Comment 25

[Marina Samuel [:emtwo]]

Comment on attachment 8675219 [details] [diff] [review]
v1. postponing suggested tile recomendation after a block

Review of attachment 8675219 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good.

Just one comment. Can we add a specific test to mimic Daniel's experience? i.e. there is more than 1 suggested tile but blocking one of them will block all others from showing up for 24 hours.

Comment 26

[maxim zhilyaev]

Attachment: v2. fixed reviewer comments

Comment 27

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/117b2d69c650

Comment 28

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/117b2d69c650

Comment 29

[maxim zhilyaev]

Attachment: v2. aurora patch

Approval Request Comment
[Feature/regressing bug #]:
Bug 1204361 - New Tab Page doesn't honor my "x" clicks on Suggested Tiles; instead of removing the tile, it swaps in a very similar one.

[User impact if declined]:

Significant user impact - irritating behavior of newtab insisting on showing suggested tile even though a user blocked it. 

[Describe test coverage new/current, TreeHerder]:
- https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8ccc93ce4ea
- manual tests
- also added xpcshell unit test

[Risks and why]: 

Unlikely to affect user experience in any negative way.
Could reduce total number of suggested tile views by a very small fraction. 
Regression risks are likely minor - changes are small, tested and unit-tested. 

[String/UUID change made/needed]:
None

Comment 30

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8678479 [details] [diff] [review]
v2. aurora patch

Has new tests, ok on m-c. Approved for uplift to aurora.

Comment 31

[Liz Henry (:lizzard) (relman/hg->git project)]

It would be good to give this a little extra testing once this lands in aurora (or in early beta)

Comment 32

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/828cf57b59e6

Comment 33

[Paul Silaghi, QA [:pauly]]

I'm trying to reproduce the initial issue. How can I get some "suggested" tiles on my New Tab Page as in your screencasts?

Comment 34

[Daniel Holbert [:dholbert]]

You'd have to use a profile that's built up a good bit of history, I think, and then you'd have to wait for Firefox to discover that there are suggested tiles available.

Punting needinfo to maxim to see if he's got any more concrete suggestions for testing this on-demand.

Comment 35

[maxim zhilyaev]

paste these urls into url bar and reload them a few times

air.tv
activebeat.com
wired.com
gizmodo.com
bloomberg.com
beamly.com
maxim.com
budgettravel.com
abc.com
cbsnews.com
nytimes.com

this should do it :)

Comment 36

[maxim zhilyaev]

Paul,

Ignore air.tv - it does not seem to work
you should load at least 8 sites from the list
and you should push reload button a few times on each page to bump up its frecency above directory tiles.
I just tested it on Nightly and Beta and seems to work.
Ping me if you still have trouble to repro - I am maksik on IRC

Comment 37

[Daniel Holbert [:dholbert]]

(Note also that Suggested Tiles (the tiles in question here) will have a "Suggested" label on them, and usually appear at the top-left. Don't be fooled by the tiles with a "Sponsored" (not "Suggested") label that show up in a completely-fresh profile -- those are different.)

Comment 38

[Paul Silaghi, QA [:pauly]]

(In reply to maxim zhilyaev from comment #35)
> paste these urls into url bar and reload them a few times
I've loaded in a new profile all the urls and reloaded 20 times each -> still no sign of "suggested" tiles on the NTP.
Tested on Nightly 2015-09-18 and latest nightly, Win 7 and Ubuntu 14.04.
What could be the problem?

Comment 39

[maxim zhilyaev]

Paul,

I see suggested tile on NTP on 45.0a1 (2015-11-06) on Mac.
Ping me on irc - maksik, and let's just debug together

Comment 40

[Paul Silaghi, QA [:pauly]]

I didn't find you on IRC, I've sent you an email.
Thanks.

Comment 41

[Paul Silaghi, QA [:pauly]]

So, I finally managed to make suggested tiles appear using a US proxy.
The problem is I still need to press 'X' 2 times to make a suggested tile disappear (before the fix I had to press X 5-6 times).
Moreover, the "Undo" button doesn't seem to work for suggested tiles.
Here is a screencast using latest Nightly - http://screencast.com/t/G0mEszAxu8R
What do you think?

Comment 42

[maxim zhilyaev]

tested on multiple versions of FX to figure what's going with either double clicks and unfo button.

1.  The need to double block suggested tile

I managed to reproduce this behavior on nightly. I do not get consistent behavior though, sometime I can get rid of suggested tile with one block, and other times it does take two blocks.  Which means that the problem is caused by some caching issue that I did not observe while working on the bug initially. 

The nature of this specific fix is to stop recommending suggested tiles for 24 hours if a user blocks a suggested tile. I looked at current 42 release (that doesn't have the fix): and it's very difficult to get rid of suggested tiles (as Paul mentioned), since blocking one suggested tile does not stop others to be served. 

I propose to file the double block issue as a separate bug and let this fix roll into release to let users get rid of suggested tiles with at most two blocks - right now it takes quite an effort to remove suggested tiles, and even then they come back to live again when FX updates tile distribution and it contains new suggested tiles that haven't being blocked before. 

2. Undo button

Undo is seriously broken for suggested tiles in 42 already.  When a suggested tile is blocked, another suggested tile pops up. When a user clicks undo, a user does not go back to the tile he has blocked - a user sees a random suggested tile instead. 

With this fix, we stop recommending suggested tiles as soon as a user blocks one. Which surfaces the problem in a different way: since there's no suggested tiles to show, "undo" simply does nothing, which is probably why Paul noticed it. 

I am not sure if the this is a regression since the functionality was already broken, and the fix surfaced this brokenness in a different way. But we still need to file a separate bug for it.

---------------

Perhaps, we could verify the current fix with caveat of the need to fix "double block issue" (filed as a separate bug) and scope the "undo" problem.  Daniel, Paul, your thoughts?

Comment 43

[Daniel Holbert [:dholbert]]

The need for 2 blocks is concerning.  (I haven't been able to reproduce that locally, FWIW. Thanks Paul for catching it on video.  And, I'm glad maxim can reproduce even if I can't, since hopefully that means he can debug/fix it.)

From my perspective, I'm happy to have that tracked in a followup bug, though we should consider having that bug prioritized in the same way that this bug was prioritized (unless it's extremely-unlikely that users will hit it, or something).

RE the undo button, I'm pretty sure that was already broken w/ suggested tiles, so that should also be tracked separately (if there isn't already a bug on it).

Comment 44

[maxim zhilyaev]

Filed follow up bugs:

Bug 1224310 - Suggested tiles requires two blocks to stop showing in newtab
Bug 1224311 - Undo button is broken for suggested tiles

Will try to root-cause Bug 1224310 asap.

Comment 45

[Paul Silaghi, QA [:pauly]]

(In reply to maxim zhilyaev from comment #42)
> Perhaps, we could verify the current fix with caveat of the need to fix
> "double block issue" (filed as a separate bug) and scope the "undo" problem.
Agreed. Marking this bug as verified fixed on FF 43b2, 44.0a2 (2015-11-11), considering the follow-up bugs from comment 44. Thanks.