601563 - Twitter stream updates are way too infrequent

Status: RESOLVED INVALID

(support.mozilla.org :: Army of Awesome, task)

Description

[David Tenser [:djst]]

Attachment: Extremely old tweets

I was constantly hitting Refresh to see how often the actual twitter stream in the backend and I discovered that the update interval is at least 12 minutes long. This is way too infrequently in my opinion -- it means that a lot of users might be looking at and responding to the same tweets, while there are hundreds of other tweets that are ignored. 

1) Can we shorten the interval to the minimum allowed by the Twitter API? (I'm assuming it's shorter than 12 minutes.)

2) *Ideally*, we should be querying the last 100 tweets every 2 minutes or so, and then use local javascript logic to randomize which 20 of those 100 tweets is shown in the list. That way, we can minimize the risk of having different people seeing and responding to the same tweet even further. It would also make it possible for one single user to hit Refresh to see a new list of 20 tweets even if the server-side list of tweets hasn't been updated. (This might be a separate bug for a future push though.)

Comment 1

[Fred Wenzel [:wenzel]]

Unless I am mistaken, we are collecting the latest tweets every minute. The infrequency of actual updates when clicking might come from caching. Maybe we can shorten the cache times there?

Randomizing between the last few tweets might be a good idea as well. If we relay the entire list of 100 to the client and do the randomizing in JS, we can help concurrent users to see different tweets, reducing the likelihood of duplicate answers.

Comment 2

[David Tenser [:djst]]

(In reply to comment #1)
> Unless I am mistaken, we are collecting the latest tweets every minute. The
> infrequency of actual updates when clicking might come from caching. Maybe we
> can shorten the cache times there?

Yes, I suspect this is an IT issue.

> Randomizing between the last few tweets might be a good idea as well. If we
> relay the entire list of 100 to the client and do the randomizing in JS, we can
> help concurrent users to see different tweets, reducing the likelihood of
> duplicate answers.

Agreed. Let's file a separate bug for that.

Comment 3

[James Socol [:jsocol, :james]]

Before we do a ton of work, let's double check that the cron jobs on both staging servers are running as frequently as they should. If that's the only issue it's a very simple solution.

Comment 4

[James Socol [:jsocol, :james]]

(In reply to comment #2)
> (In reply to comment #1)
> > Unless I am mistaken, we are collecting the latest tweets every minute. The
> > infrequency of actual updates when clicking might come from caching. Maybe we
> > can shorten the cache times there?
> 
> Yes, I suspect this is an IT issue.

Just to clarify, cache expiration times are not an IT issue. The cron not running often enough is.

Comment 5

[James Socol [:jsocol, :james]]

David: Are you 100% sure this was happening on master.support and not support-stage?

Comment 6

[David Tenser [:djst]]

Yes, 100% sure. It's the only "Army of Awesome" tab I have had open the last week.

Comment 7

[James Socol [:jsocol, :james]]

Double checked the cron rates with Shyam. We can move this back to server-ops if there's anything for them to do.

Comment 8

[David Tenser [:djst]]

Attachment: Some tweets on our Customer Care page

Here's a list of tweets from our Customer Care page. I'll attach the next screenshot which shows the corresponding search.twitter.com page.

Comment 9

[David Tenser [:djst]]

Attachment: The search.twitter.com results

Some questions about a few of these tweets. In this query, I specified that I did *not* want to include retweets, but still, there are tweets in here that should be part of the Customer Care page. 

Legend:
* Yellow: should be on our Customer Care page -- why are they not?
* Red: replies, so shouldn't be part of our Customer Care page

Comment 10

[David Tenser [:djst]]

After talking to James on IM, it looks like we're filtering out any tweets that contain links. Not clear on why we're doing that, but if that should change, we'll file a new bug for that.

This bug is INVALID. Sorry for wasting people's time (but thanks for clearing this up)!