657958 - Default snippet content showing

Status: RESOLVED FIXED

(Snippets Graveyard :: General, defect, P1)

Description

[Laura Forrest]

I'm getting reports that the default snippets are showing. Is the snippet service down? 

Chelsea and one other person from the Toronto office reported this and I saw the default snippets showing on Chelsea's machine. Chelsea is running 4.0.1. She did mention she switches back and forth from different builds at times.

Assigning to Chris for now. Chris - let me know if there's someone else I should assign this to. I know that Les is transitioning away from the service and want to be sensitive to that.

Comment 1

[Les Orchard [:lorchard]]

Not sure what a next step would be on this bug... can't reproduce. As far as I can tell, the snippet service is working fine. I can fetch snippets manually from the service, and I see them in my browser. 

Default snippets may appear if the machine running the browser is offline at the time of an attempted fetch, or if it's a build that doesn't yet have snippets. (eg. I don't think there are any for Beta channel yet)

Comment 2

[Chris More [:cmore]]

I actually saw this yesterday. I loaded about:home and the snippet box was blank. I refreshed and a snippet showed up. If I see this again I will use Firebug to see what is contained in the div. 

@lorchard: Could it be something in the server cache? I've had a similar problem with a caching server at a previous job and occasionally we would flush the server cache and it would resolve the problem. Not sure if that is feasible or would have any affect on it.

Comment 3

[Les Orchard [:lorchard]]

(In reply to comment #2)
> I actually saw this yesterday. I loaded about:home and the snippet box was
> blank. I refreshed and a snippet showed up. If I see this again I will use
> Firebug to see what is contained in the div. 

Blank content is not the same as default content. Default content consists of two snippets built into Firefox. I don't remember what they are, but they don't have icons, FWIW.

Blank content probably means it was in the process of fetching content, or was taking awhile to display content it already had. If a refresh displayed content, then one of those two is probably what happened - because a refresh of about:home does not cause another request to the server.

> @lorchard: Could it be something in the server cache? I've had a similar
> problem with a caching server at a previous job and occasionally we would
> flush the server cache and it would resolve the problem. Not sure if that is
> feasible or would have any affect on it.

Unless IT knows of a cache outage, I can't think of what issue there'd be with the server cache beyond stale content if changes to snippets were pushed very recently. I disabled a set of snippets earlier this week, but that shouldn't have caused an issue.

Comment 4

[Chris More [:cmore]]

@lorchard - true -- I thought she was saying the default snippets are NOT showing.

@Laura - one method to determine if there has been a problem is to look at the webtrends for the snippet clicks and see if they fell off. If the analytics look fine and we are unable to replicate the issue, it is more likely a connectivity issue with the person's browser. 

Without additional debug info, it is nearly impossible to determine the root cause. Given that the snippets are viewed billions of times a month, if there was a problem on our side, we should see a dramatic drop in clicks.

Comment 5

[Chelsea Novak [:chelsea]]

(In reply to comment #3)

> Blank content is not the same as default content. Default content consists
> of two snippets built into Firefox. I don't remember what they are, but they
> don't have icons, FWIW.
> 

One is a thank you for using Firefox message and the other is an invitation to start using addons. 

FWIW, I have restarted, cleared cache and looked at this page from two different IP addresses (here at Mozilla and at the hotel) and I am still only seeing the two default snippets. I am using Firefox 4.01.

Comment 6

[Chris More [:cmore]]

@lorchard -- I got the problem resolved, but I don't have a clear indication on why it was happening. Initially, she was unable to connect to snippets.mozilla.com:443 from any browser thus it had to be a network/routing issue at the system level. I walked away and suddenly she was able to connect to the snippets server, but the default snippets were still showing up even after clearing the local cache. I then installed your snippets switcher .xpi file and it immediately started working. I uninstalled the plugin and it is now working fine for Chelsea.

If it happens again, I will first check the snippets URL in the about:config setting. There is another person who is having the problem and I will investigate those too.

Thanks,
Chris

Comment 7

[Les Orchard [:lorchard]]

It sounds like there was a connectivity problem at some point. Since there were 2 IPs and a hotel involved, we're probably talking about a moving laptop that had the bad luck of being disconnected when Firefox checked for snippets.

But, just to be clear about snippet caching: Neither refreshing the about:home page nor clearing the browser cache will have any effect on snippets.

The about:home page has its own built-in cache that expires only once every 24 hours. Clearing out localstorage for about:home is only thing that clears that cache early and forces a fetch of fresh content. That's one of the things the snippet switcher add-on does.

Comment 8

[Chris More [:cmore]]

@lorchard -- do you think the install of the snippet switcher add on just aided on clearing up the localstorage cache sooner after the connectivity was restored?

Comment 9

[Les Orchard [:lorchard]]

(In reply to comment #8)
> @lorchard -- do you think the install of the snippet switcher add on just
> aided on clearing up the localstorage cache sooner after the connectivity
> was restored?

Yeah, that's basically what I was saying.

If there was a connectivity problem when Firefox tried fetching snippets, it won't try again for 24 hours. Doesn't matter if connectivity is restored in the meantime. Only clearing localstorage with something like the add-on will force an early fetch.

Comment 10

[Laura Forrest]

Hey Chris - any updates here? I'm getting other reports that default content is showing.

Comment 11

[Les Orchard [:lorchard]]

(In reply to comment #10)
> Hey Chris - any updates here? I'm getting other reports that default content
> is showing.

I don't think that this is a bug that has a fix, so there will likely be no updates.

Showing default snippets is a normal fallback feature, triggered whenever the browser loses contact with the service. As far as I can tell, there are no known issues with the service, so this is probably a client connectivity issue.

More details about when it does might be helpful to uncover a problem if there really is one.

Comment 12

[Les Orchard [:lorchard]]

(In reply to comment #10)
> Hey Chris - any updates here? I'm getting other reports that default content
> is showing.

In other words: There will always be cases when default content shows. That's why it's built into the browser, so that there's at least something to fall back on.

Comment 13

[Chris More [:cmore]]

Laura:

I'm still waiting for Ryan Merkley to respond to my last inquiry into his problem with his home computer. If you could give me another person that is having problems, I can try my process to fix. This does not seem to be a server issue, but a local client/browser issue. A possible FF bug, but I will know more if I can test my theory on other people.

Les:

What I am seeing is that people are able to get to the snippets server and they have been experiencing the default snippet showing for more then 24 hours (usually much longer). Sometimes one or both of the default snippets are showing up or the same exact non-default snippet is being displayed over and over. What it seems like is doing on is that the LocalStorage is not expiring after 24 hours and also not being updated every 24 hours. The only thing that seems to resolve the problem is flushing the LocalStorage, which your Snippet Switcher just happens to do. I haven't tested this theory on enough people, but I'm leaning toward LocalStorage being the culprit here. If it was a true server issue, it wouldn't random as it seems to be.

Thanks,
Chris

Comment 14

[Chris More [:cmore]]

(In reply to comment #12)
> In other words: There will always be cases when default content shows.
> That's why it's built into the browser, so that there's at least something
> to fall back on.

Completely agree and I understand, but it seems like some people are having problems coming back "online" after default Snippets are displayed or after their LocalStorage cache should expires. Regardless if it was a connectivity problem or not, the problem should not continue for more then 24 hours after connectivity is restored, right?

Comment 15

[Les Orchard [:lorchard]]

(In reply to comment #14)
 
> Completely agree and I understand, but it seems like some people are having
> problems coming back "online" after default Snippets are displayed or after
> their LocalStorage cache should expires. Regardless if it was a connectivity
> problem or not, the problem should not continue for more then 24 hours after
> connectivity is restored, right?

Really, there's just a need for more information here, but I'm not sure how we'd get it other than anecdotally. The service seems fine, and I've not been able to reproduce the problem. Just guessing, otherwise.

Since this isn't stuff we can fix in the service, here are some ideas for improvements in about:home itself for a future Firefox version:

* File a bug to make about:home retain snippet content, if there's a failure or even just blank content from the service. I'm not sure what exactly is happening here, but the browser should at least hold onto what it last fetched and not revert to default content.

* File a bug to make the about:home page fetch more frequently after a failure (eg. once every 3 hours) until the next success. This would probably be veto'd by IT, though, since the code intentionally errs on the side of caution. If the service does actually get overloaded, having clients batter it harder for updates is the last thing we want to see happen.

Comment 16

[Chris Beard]

(In reply to comment #12)

> In other words: There will always be cases when default content shows.
> That's why it's built into the browser, so that there's at least something
> to fall back on.

Actually, I believe the original design was that it would only do this if it had never loaded external content or if the local store had been erased for some reason.  The idea being that the load of new content would be asynchronous to the disply and that it would only fall back to the default if there was no cached data.  From the comments here it sounds like we're wiping the local store when we fail to grab new content?

Comment 17

[Les Orchard [:lorchard]]

(In reply to comment #16)

> Actually, I believe the original design was that it would only do this if it
> had never loaded external content or if the local store had been erased for
> some reason.  The idea being that the load of new content would be
> asynchronous to the disply and that it would only fall back to the default
> if there was no cached data.  From the comments here it sounds like we're
> wiping the local store when we fail to grab new content?

Yeah, that sounds like a bug, per comment 15

Comment 18

[Chris More [:cmore]]

(In reply to comment #17)
> (In reply to comment #16)
> 
> > Actually, I believe the original design was that it would only do this if it
> > had never loaded external content or if the local store had been erased for
> > some reason.  The idea being that the load of new content would be
> > asynchronous to the disply and that it would only fall back to the default
> > if there was no cached data.  From the comments here it sounds like we're
> > wiping the local store when we fail to grab new content?
> 
> Yeah, that sounds like a bug, per comment 15

Yeah, that seems like it is part of the problem too. The issue here is that we don't have enough people that are experiencing the problem to test this bug and to come to a real conclusion. The two people who experienced default content being loaded saw this problem for much longer (days) then 24 hours. They were able to access the Snippets server, had access to the internet since the default snippets were loaded, and they still were only displayed the default snippets. Only when LocalStorage was purged, did they see the real live snippets. If all of this was within 24 hours, I would say this is by default, but it wasn't.

Shall I file a bug to investigate this further per comment 15?

Comment 19

[Les Orchard [:lorchard]]

(In reply to comment #18)
> ... had access to the internet since the default snippets were loaded ...

FWIW, this is incorrect: The default snippets are embedded in the browser itself. They are not loaded from the internet.

> Shall I file a bug to investigate this further per comment 15?

Yeah, we should have bugs for both of those issues in comment 15

Comment 20

[Laura Forrest]

Additionally - default snippets are also showing to Beta 5 users, per Chris B. 

Expected content is: "Get Firefox Beta for Android. </a>  Experience cutting edge features and provide your feedback."

Does this permit a new bug, or does this bug cover the root cause?

Comment 21

[Chris More [:cmore]]

(In reply to comment #19)
> (In reply to comment #18)
> > ... had access to the internet since the default snippets were loaded ...
> 
> FWIW, this is incorrect: The default snippets are embedded in the browser
> itself. They are not loaded from the internet.
> 

I know that the Snippets are embedded into the browser build. I meant that the default Snippets were loaded even though the person had connectivity and more then 24 hours has passed. That was my point... the problem is longer then the expected 24 hours.

Comment 22

[Chris Beard]

I forced a server update by setting localStorage["snippets-last-update"] to 0 using the web console and reloading the page.  

[14:05:00.125] GET https://snippets.mozilla.com/1/Firefox/5.0/20110517192056/Darwin_Universal-gcc3/en-US/beta/Darwin%2010.4.0/default/default/ [HTTP/1.1 200 OK 275ms]

Which appeared to be successful.  Looking at what was returned though, it looks like there are no snippets in the snippetContainer for this client version, and so the wrapper code is setting localStorage["snippets"] to "", removing all evidence and then forcing the display of the default snippets. 

[..]
<script type="text/javascript">
var snippet_container = document.getElementById('snippetContainer');
var snippets = snippet_container.getElementsByClassName('snippet');
if (snippets.length > 0) {
    var show_snippet = snippets[Math.floor(Math.random()*snippets.length)];
    show_snippet.style.display = 'block';
    activateSnippetsButtonClick(show_snippet);
} else {
    localStorage['snippets'] = '';
    showSnippets();
}
</script>

Whereas, I see snippetContainer content if I force the version to 4.0.1 by fetching a URL like:
https://snippets.mozilla.com/1/Firefox/4.0.1/20110517192056/Darwin_Universal-gcc3/en-US/beta/Darwin%2010.4.0/default/default/

I suppose one hack to "fix" this would be to have the server return null content if there are no snippets (and thus not overwriting localStorage["snippets"]), so that it would display a previously cached valid snippet set until such time as the new release snippet set is updated.

I'll poke around some more to see if it's the case in all instances that we're seeing this that the snippetContainer is returning no content.

Comment 23

[Chris More [:cmore]]

@Chris -- What you described above seen to be true. Even though there is no object named "snippets" when no snippets returned from the server, document.getElementsByClassName('snippet').length will still return 0 and wipe localStorage['snippets'].  This still doesn't explain what Chelsea and Ryan experienced. 

Ryan got back to me and the last test fixed the problem he was having.

Ryan's problem:

He is running 4.0.1 full and only sees the Aura channel switcher snippet over and over again. This has been happening for weeks and nothing resolves the issue. He has not had any connectivity issues that he knows if when the issue started. He should not even be seeing the Aurora snippets since he is running 4.0.1 of FF.

Here are the steps that I tested with Ryan:

1. Checked browser.aboutHomeSnippets.updateUrl and verified it was set to: https://snippets.mozilla.com/%STARTPAGE_VERSION%/%NAME%/%VERSION%/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

2. He visited the snippets URL (https://snippets.mozilla.com/) to verify he is able to connect to the address. All good.

3. Clear localStorage["snippets"].

After step 3, he refreshed about:home and the 4 typical snippets started showing up fine. This is the first time that it worked in weeks for him. I did nearly the same steps with Chelsea and got the same results and she has been fine since then.

Here is the caveat that could be the root cause:

Both have installed Aurora, Beta, and full versions, and uninstalled and went back to 4.0.1 full around the time that they started to notice the issue. Chelsea wasn't seeing the Aurora channel switcher snippet, but just the default built-in snippets. 

Here are the things in common with the two people:

1. Both users were running 4.0.1 full at the time they noticed the problem.
2. They were NOT seeing the expected snippets.
3. This problem exists for weeks with no change.
5. Both were able to connect fine to snippets.mozilla.com.
4. Both have previously installed aurora and beta copies of FF.
5. Both users had their problems go away as soon as localStorage["snippets"] was manually flushed.

As Les mentioned earlier, I don't think this is a server issue. It seems to be a localStorage hiccup and something with switching between releases of FF.

Comment 24

[Les Orchard [:lorchard]]

(In reply to comment #22)
> I forced a server update by setting localStorage["snippets-last-update"] to
> 0 using the web console and reloading the page.  

That's what the snippet switcher add-on does, by the way.

> [14:05:00.125] GET
> https://snippets.mozilla.com/1/Firefox/5.0/20110517192056/Darwin_Universal-
> gcc3/en-US/beta/Darwin%2010.4.0/default/default/ [HTTP/1.1 200 OK 275ms]
> 
> Which appeared to be successful.  Looking at what was returned though, it
> looks like there are no snippets in the snippetContainer for this client
> version, and so the wrapper code is setting localStorage["snippets"] to "",
> removing all evidence and then forcing the display of the default snippets. 

What you're seeing is this case is a successful request to the snippet service. But, it just doesn't have anything that matches that set of user agent attributes.

Bug 654355 is probably an answer to this, since more/better rules will cover a broader range of versions.

> Whereas, I see snippetContainer content if I force the version to 4.0.1 by
> fetching a URL like:
> https://snippets.mozilla.com/1/Firefox/4.0.1/20110517192056/Darwin_Universal-
> gcc3/en-US/beta/Darwin%2010.4.0/default/default/

The switcher add-on will let you do that in the browser, too. And, what you're seeing there is a successful request where the service *does* have content for the user agent attributes.

> I suppose one hack to "fix" this would be to have the server return null
> content if there are no snippets (and thus not overwriting
> localStorage["snippets"]), so that it would display a previously cached
> valid snippet set until such time as the new release snippet set is updated.

The situation here is that a successful request was made to the service, but it just came up empty-handed for that particular client. 

If we return null, then there's no easy way to ensure that old / expired snippets stop displaying in the browser. Sometimes, there really are no snippets to show and the previous cached set is not actually valid, just stale.

If the display logic doesn't clear out localStorage["snippets"], then *nothing* will be shown - because, there are no matching snippets. So, it clears out localStorage so that at least default snippets will be displayed if the service genuinely has no content for that browser.

> I'll poke around some more to see if it's the case in all instances that
> we're seeing this that the snippetContainer is returning no content.

That would be useful... I'm really not sure what the circumstances are for people seeing default content. It could just be that they're running something for which we don't have matching snippets. I don't think we have snippets for every combination of version and channel yet.

Comment 25

[Les Orchard [:lorchard]]

(In reply to comment #23)

> As Les mentioned earlier, I don't think this is a server issue. It seems to
> be a localStorage hiccup and something with switching between releases of FF.

FWIW, this is the code within Firefox for loading snippets:

http://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/aboutHome.js#l203

Just eyeballing it, I couldn't say what might cause a hiccup and sticky snippets. Not sure how to find it, but a reproducible test case and maybe pulling in someone working on the Firefox side of things could help. Not sure if there's anything showing up in the Error Console for people affected, or if maybe something's throwing a suppressed warning/error on load?

Comment 26

[Chris More [:cmore]]

I am going to close this bug since we don't have any other people to test if this a widespread problem. Once we get webtrends tracking on default Snippets, it will probably give us some better insight into times when default Snippets are being displayed.

Both Chelsea and Ryan are working fine now.

Comment 27

[Marco Bonardo [:mak]]

Sounds like I have a similar problem, it tries to download this page:

https://snippets.mozilla.com/1/Firefox/7.0a1/20110628030753/WINNT_x86-msvc/en-US/nightly/Windows_NT%206.1/default/default/

and it has no snippets, so default snippets are always shown

Comment 28

[Chris More [:cmore]]

Marco: This is expected results. Currently, if there are no Snippets defined for a combination of Browser+OS+locale+channel, you will get the default Snippets. One improvement to the Snippets Service that should be done this week will allow the engagement team to be able to easily copy rules from previous versions of Firefox to newer/future ones.

The problem this bug was trying to solve was people who had a browser version+OS+locale+channel that should have been being served up real snippets (not default) and they were not for more then 24 hours after firefox tried again. The two people that were said to have had this issue were resolved when we flushed localStore["snippets"]. We have not heard of other people with this exact issue to determine a root cause or if this was just local issue with their browsers.

Comment 29

[Marco Bonardo [:mak]]

(In reply to comment #28)
> Marco: This is expected results. Currently, if there are no Snippets defined
> for a combination of Browser+OS+locale+channel, you will get the default
> Snippets.

Yes, my question was mostly if it's expected that we have no snippets in Nightly, since I'm sure I was seeing some weeks ago. Looks like the answer is yes, but I wanted to ask, just in case.

> The two people that were said to have had this issue
> were resolved when we flushed localStore["snippets"].

Any idea what was inside it before the flush? it's strange that helped, since it is being overwritten every time from what I see.

Comment 30

[Chris More [:cmore]]

(In reply to comment #29) 
> Any idea what was inside it before the flush? it's strange that helped,
> since it is being overwritten every time from what I see.

Unfortunately, no. We need additional people that are experiencing the problem to do more investigation work. Given the billions of people that see about:home, if this was a widespread issue, we would have a lot of people to test. The issue is that some people may not know the difference between default and production Snippet and never know there is a problem.

This will be resolved in future versions of Firefox, because we are including webtrends tracking on default Snippet so we have an idea how often they are being loaded directly about the browser.

Thanks,
Chris