Closed Bug 586511 Opened 15 years ago Closed 14 years ago

build dashboard based on twitter data

Categories

(Mozilla Metrics :: Data/Backend Reports, enhancement)

Product:
Mozilla Metrics
Component:
Data/Backend Reports
Platform:
x86
macOS
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Unreviewed

People

(Reporter: aphadke, Assigned: nelson.sousa)

References

Details

(Whiteboard: ETA 1st dashboard: 10/01; ETA 2nd dashboard: 10/08)

Attachments

(6 files, 2 obsolete files)

Screen 1
630.64 KB, image/png
Details
Screen 2
769.37 KB, image/png
Details
Screen 2 - word selection
717.10 KB, image/png
Details
screen1
763.96 KB, image/png
Details
screen2
846.40 KB, image/png
Details
screen2_keyword_drill
750.30 KB, image/png
Details
We are currently searching twitter for bunch of keywords related to Mozilla's products. This data is currently stored inside HBase. Requirements below: 1 - Number of replies that include either the "Firefox" or "#fxhelp" keywords by day. A reply is defined as a tweet that starts with '@'. 2 - Percentage of "Firefox" tweets that are being replied to by day. Defined as the number of replies from above divided by the number of tweets with a "Firefox" or "Mozilla" keyword. 3 - Top 'x' users whose tweets include keyword "#fxhelp" by day. A top user is someone who has the most tweets containing "#fxhelp" for that time period. 4 - Top 'x' users whose tweets include keyword "#fxhelp" for the whole dataset 5 - Top 'x' languages for certain "terms" by day (based on language code)
Just out of curiosity, where / how will this dashboard live?
Perfect. I was going to kill you if you gave me any other answer. Then we have a somewhat challenging and insanely rewarding road ahead of us! We need to talk.
there's a lot of people waiting to kill me, i'll add u to the list.. we can talk whenever you want.... is now a good time?
Lemme check my agenda: 12/Aug/2010, 1am - Sleep! Nop, sorry, busy ;) (to whoever's on this bugmail list, sorry for the spam, but it's stronger than me!)
I for one support this bugmail spam. I am very, very excited about this dashboard. Anurag and Pedro, let me know how I can help with reviewing the dashboard.
forgot to add some requirements: 1 - All Tweets for certain "terms" by day. 2 - Top 'x' keywords for a given day. 3 - Top 'x' keywords for the whole dataset. 4 - Ability to search the tweets.
Blocks: 584886
Target Milestone: --- → 2010-08.2
Next steps (based on conversation between Pedro and Anurag): 1 - Pedro/team is going to go ahead and build a CDA interface for HBase. 2 - Anurag is going to add new table/column family that would essentially put the business logic for the requirements listed by William inside HBase thereby making it easier/faster for CDA to query HBase and build charts.
Group: mozilla-stats
Whiteboard: building additional columns inside HBase (Anurag) and CDA connector (Pedro) - ETA - 8/19
(In reply to comment #0) > We are currently searching twitter for bunch of keywords related to Mozilla's > products. This data is currently stored inside HBase. Requirements below: > 1 - Number of replies that include either the "Firefox" or "#fxhelp" keywords > by day. A reply is defined as a tweet that starts with '@'. > 2 - Percentage of "Firefox" tweets that are being replied to by day. Defined as > the number of replies from above divided by the number of tweets with a One of the requirements in comment #0 doesn't make sense to me. If we define requirement 1 as replies that include a keyword, that eliminates replies to requirement 2 that don't contain the keyword. i.e. with these two tweets: deinspanjer: I love the new Firefox beta! bgates: @deinspanjer I agree, I'm going to switch over now We wouldn't count the reply in requirement 2. Also, it seems to me that retweets/vias are a pretty important measure that isn't discussed here yet. Thoughts?
Using Hive in place of HBase for data storage. Based on requirements listed in description & comment #7, hive might be a better datastore than hbase.
Daniel - Fair questions related to #9. I'll request William to handle the replies as he is the main driver for this project.
Good questions Daniel. Based on the data we are collecting in bug 586511, I didn't think it was possible to track the use case you mention in comment 9. If we can include this reply tracking in our requirements, let's do it. In terms of measuring retweets/vias, I'll post my comments later today. Also thinking about tracking some basic metrics for the @firefox accounts for the dashboard.
I believe you are right that we couldn't currently track the use case mentioned in comment 9. That said, we need to make sure that we word and understand the requirement #2 properly so that it is known that the use case is not counted.
Sorry for not following up earlier to comment 12 - thought I already posted my follow up comments. Tracking the use case from comment 9 may be possible but it is costly in requests because it requires do a reverse lookup. Therefore, I think we should not track replies in that use case. I'm including some links I used to come to that conclusion below. Would also be nice to include basic tracking for the @firefox account on this dashboard. Number of followers by day and number of tweets from the account by day are probably the most important. Being able to visualize how our tweets for a given day impacted our follower count would be very interesting as well. For example, if I could see that tweeting about a marketing campaign on Monday led to +500 followers for the day but tweeting about Firefox Panorama created a spike of +1200 followers would help us with crafting our @firefox tweet strategy. Facebook Insights' Interactions view does this nicely by showing a line graph of Likes, Comments, and Unsubscribes for the past 30 days with a list of messages/updates posted below the graph and the date the message was posted. Could a similar view for the @firefox account be included in this dashboard? Links about twitter replies using API: http://dev.twitter.com/pages/streaming_api http://thecodetrain.co.uk/2009/05/following-twitter-conversations-using-yahoo-pipes/ http://code.google.com/p/twitter-api/issues/detail?id=273 http://code.google.com/p/twitter-api/issues/detail?id=142
Assignee: aphadke → pedro.alves
Whiteboard: building additional columns inside HBase (Anurag) and CDA connector (Pedro) - ETA - 8/19 → ETA: 9/1, prototype dashboard will be ready by then
Target Milestone: 2010-08.2 → 2010-09.1
Anurag and Pedro, will we be able to include the @firefox account tracking from comment 14 in the dashboard?
William - We can't commit on having @firefox ready by 9/1, but we will do the best we can, at worst, you should have it ready by 9/8
By 9/8 would be great, thanks!
Hello everyone. We're currently working on the data and we should have it ready very soon. Next step is building the dashboards. Nuno created the mockups for it (I'm attaching them to the bug in a minute), and before we implement it we'd like your feedback on them to see if that's what you expect. We have 2 sections, one for the main info / charts and another one for keywords. In the last one, choosing a keywork will allow us to search through the actual tweets. Does this cover all you'd like to see here? Would you like to change/add/remove something? About data update frequency, what are your needs? I'd say that for the main chart table, daily updates are enough as the charts have that granularity. For the tag cloud we can update it more frequently if you want. Comments? Thanks
Attached image Screen 1 —
Attached image Screen 2 —
Attachments added
No feedback? :( Does that mean "do whatever you want"? :p On a more serious note (and you won't have many from me), we have a very short deadline to implement this, so would be good to know if this is what you're looking for
Here are my initial thoughts: - Love the design and clean layout - Change "Twitter Survey" tab name to "Overview" - Total replies (daily): can we include the day of the week under the dates? - Total replies (overall): I'm having trouble understanding this one. Is "overall" a cumulative number going back to when we started collecting data? - Top 25 users tweets include Firefox (daily): was envisioning this to show a list of usernames for the top 25 users who are tweeting about Firefox. Could we change this to a list of usernames and how many tweets with #fxhelp they have posted that day, in the prior week, and overall (all time)? I don't understand what this graph is showing right now either. - Top 25 users tweets include Firefox (daily): same as previous. I'm still not sure what "overall" means. - Top 10 tweet locations (daily): didn't think of this metric before, but it's really cool to see where tweets are coming from. I see the select box is currently showing @FirefoxCares. What are the other options for what this can show? Can it show locations for all tweets that mention Firefox? Could we also show a historical metric to compare how this changes for today? For example, show in an orange bar next to each gray bar how many tweets that locale had either yesterday or a week ago. This helps provide context and show changes. - Keyword tweets table: What does the up/down arrow indicate? That the number of tweets is higher/lower today than average? - @Firefox graph: can we change the legend to say "# of tweets" and "# of of followers"? The time period select box currently says "Prior week." What are the other options? Again, it would be awesome if we can include the day of the week under the dates on the x-axis. - Firefox tweet replies (overall): This is just for the @Firefox account, correct? If so, we should add @ in front of Firefox in the title to make that more clear. I think this metric would look good on a timeline as well, and I think we could probably move this metric to the @Firefox graph on the left so the two are combined. That way, we're adding two lines to that graph for number of tweets that include @Firefox (to @Firefox) and how many replies are sent from the @Firefox account (from @Firefox). Let me know what you think and if I can clarify any of those.
A few more comments on the other two pages: Top 50 keywords (daily): think this looks great. Would also be helpful to include a smaller version of this on the Overview tab. Could we include the top 15 or 20 keywords in this table format on that page? Keyword drilldown (list of tweets for a specific keyword): I think this could use an improved layout. I would suggest alternative a background color for each line (giving it a striped table look) and also adding the relative time for each tweet on the right of the message (ex: 5 minutes ago, 4 hours ago, 1 day ago). Lastly, the layout might look better if width of the messages are could be narrowed and each message took up two lines. The Customer Care page design does this well. Here's what it looks like: http://people.mozilla.com/~williamr/communitycare/1-landingpage.jpg
cc'ing David, Kadir, and Michael from SUMO who are helping with the Customer Care program as well and can offer feedback
The overall is exactly what you thought it was - we can rephrase it for "total" or other designation, and add a * on the bottom describing the data is the total numbers since date x - the day we started collecting info. I think the comparison between a daily/weekly/monthly analysis and the total numbers can bring some interesting perspectives, but if I'm wrong, please correct me and we will surely get to something better. Top 25 users_ can be on a table, no problem. Or we can name the X axis with the user names, didn't include it because I thought it was indifferent if it was user X or Y, but the important thing was to know there was someone that tweeted x times a day/time frame. Also I wasn't absolutely sure if using the user names was absolutely correct, I leave that discussion to you. I have to agree with you this can be a little vague at least in comparison terms. Top 10 Tweet locations _ It's a selected list that was sent to us. Will post it here once I find it again. Keywords tweets table - up and down arrows indicate the prior relative position ( updated daily on the daily table and weekly on the "total" side, if possible) like on a Music Top list. Top 50 keywords - agree and improving atm. I was looking for a solution that was not too expensive in terms of implementation time, given we need a quick answer to this request. I have to agree I went a bit too minimal. Your example works very well as a reference, thank you. Other than this, I will start "injecting" your thoughts on the mock-up. Thank you for your feedback.
Thanks for the quick reply nuno. Overall: Sounds good. Having a comparison between daily/weekly/monthly is very helpful as well. Top 25 users: A table would be perfect. Having the names is helpful for us since we plan on following up with these top users to thank them for their time and help. Top 10 Tweet locations: let us know what you find Keywords tweets table: Love that this shows prior relative position. I think this table would be even more helpful if we showed day-over-day or week-over-week percentage changes for that word and sorted the table descending by this percentage. We'd still show the number of tweets with that keyword, but I think sorting this way help us easily spot trends in what is being talked about (or not talked about it). With the current view, I have a feeling that that top 10 words would be pretty consistent over time. This alternative approach shows more about what's being said now compared to in the past.
Attached image screen1 —
Attached image screen2 —
Attached image screen2 - keyword drill (obsolete) —
Dashboard updated.(sorry, couldn't find a way to update/delete the old ones) Some notes: Inserted a global date selector - Options proposed : prior day; prior 7 days; last month, 3 months; 6 months and one year. At the moment I believe data from last 7 days = last month, we should consider hiding / graying out the options that have no relevance at the present day and start showing them when they become useful. On the tables, I suggested the ability to sort (by clicking on the column header) this allows us to check info on the terms you suggested with added flexibility. On the locales table and interpreting your words I suggest a time-line other than the double bar chart. I believe a line chart, with the ability to change the time range offers the context you where looking for, tell me what you think.
Attached image screen2-keyword_drill (obsolete) —
Attachment #471681 - Attachment is obsolete: true
Attached image screen2_keyword_drill —
Attachment #471790 - Attachment is obsolete: true
(In reply to comment #27) > > Top 25 users: A table would be perfect. Having the names is helpful for us > since we plan on following up with these top users to thank them for their time > and help. It would be good to have a filter for that. The top 'users' will be bots otherwise.
Not sure if it's already planned, but can we also have a line chart for the number of tweets we show on the customer care page and the number of tweets replied to from that page?
Top 10 Tweet locations keyword list as it is now: @FirefoxCares #fxhelp mozilla firefox "fox fire" "fire fox" foxfire "internet explorer" chrome ie9 ie8 webkit gecko internetexplorer It's configurable. We can add/remove keywords, based on your feedback and needs.
@atopal: this would require some code snippets on the page to track this, which isn't a current requirement. I'll ask WebDev if this is something we can easily do. @nuno: Looking much better! :) Here are my very lengthy comments. Let me know what you think. Total tweets: Graph looks good, especially with day names at the bottom. This is very picky, but could we remove the leading zero for the month (eg: 03-31 becomes 3-31)? Also, I just noticed that the orange bars are labeled as @Firefox, which I'm guessing is the number of tweets that include "@Firefox." We want the orange bars to show number of tweets that include the keyword "firefox" (case insensitive) and not just the ones that are directed to the @Firefox account. Right now I think there are 1000-2000 tweets with "Firefox" each day so that gives you an idea of how high those bars would be. It would be very neat to show the response rate (fxhelp tweets / firefox tweets) as a line on this graph, using the right axis to show the response percentage (0-100%). This would allow us to track our response percentage each day. Total tweets (overall): Still a bit confused on this one. Overall is making me think this shows the all-time (total for all data collected) response rate. I don't think this is what we want to show here. Maybe I'm misreading it though. Can you explain this graph more? I'm thinking we could use this top right space to show a summary of the dashboard, similar to what metrics.mozilla.com shows in the sidebar. This would include 1) Yesterday's response rate with the DoD percentage change in parenthesis; 2) # of unique users using #fxhelp (we'll call this # of Community Care contributors) with a DoD percentage change; 3) # of responses with #fxhelp (we'll call this # of responses from yesterday) and the DoD percentage change; 4) number of tweets about Firefox and the DoD percentage change; 5) table of top 5 keywords that saw the biggest DoD change (+ or -) with the same table headers as the Top 50 keywords table; 6) DoD change for keyword mentions of firefox, chrome, ie (including ie8 and ie9), and opera. I'm probably asking for a LOT to create this summary area, but I think it could be the most useful way to see a snapshot what the current state of our twitter presence is and how things are changing. What do you think? :) Top 25 users with #fxhelp tweets: This looks fantastic! I would change the title to "Top 25 Community Care contributors" to make it more clear. Could we change "daily" to "today"? Trying to understand what d-7d and d-30d mean. Is that the number of #fxhelp tweets that user has posted in the last 7 days and last 30 days? I would change "overall" to "total" as well. Can the usernames link to their twitter profiles? Ex: AbbeyMead links to twitter.com/AbbeyMead. Top 25 users with Firefox tweets: Same comments as above except I would change the title to "Top 25 users tweeting about Firefox". Since this takes up more space, I would change "Showing 1st to 10th" to just "Showing 1-10" so that everything can still fit on that line. This applies to the #fxhelp tweets table on the left as well. @Firefox analysis: this looks great. I think we just need to make the legend labels a bit more clear. Let's try this: "# of tweets" to "tweets from @Firefox"; "to @Firefox" to "tweets to @Firefox"; "from @Firefox" to "replies from @Firefox". Top 5 tweet locales: looks great. What are the other options in the "@FirefoxCares" dropdown box? Comments page: Thanks for adding the usernames and linking to their twitter pages. I'm sure that will be very helpful. I would change the header to "Tweets" instead of "Comments" so it's more clear what is on this page. I think of comments as what is posted on blogs.
William, you have a very constructive way of saying "Looking much better! :)" ;) What do I think: My main concern is that this dashboard fits exactly on your needs (yours and everyone who uses it as a work tool at mozilla). That being so, I have to thank you for your attention and detailed feedback. We are already evaluating and working on it. Thank you.
Also, you can find the list on comment 36
The first prototype of the dashboard is now available at the staging metrics server: http://tinyurl.com/2wst2qn It's available from MPT (login credentials are those on LDAP) Please beware that this is just a prototype, a lot of work remains to be done. Performance isn't very good, the "Twitter overview #2" isn't done yet, the keyword list if fixed, charts need some tweaking and the date range is fixed (last 7 days only). But the data is real and live (data for "today" is updated every 5 minutes and reflects real tweets as they're being published). Work on this dashboard is suspended while the new bugzilla dashboards are being built and will resume during next week. Feedback is more than welcome.
Assignee: pedro.alves → nelson.sousa
There seems to be a problem with the site. The tinurl resolves to http://www.cm-metricsapp01.com:8080/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=%2Ftwitter&file=twitter_data.wcdf and the server at that address does not repsond.
are you connected to the VPN? The tinyurl resolves to cm-metricsapp01 (without www or com), but if you're not connected the url doesn't resolve and the browser adds www and com and tries that one.
Thanks for the update Nelson. I'm also having trouble viewing the site and it's resolving to the same url as comment 42. Tried using the VPN and got the same error message. What is the non-tinyurl? Also, could we update the whiteboard with a new ETA?
Full URL: http://cm-metricsapp01:8080/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=/twitter&file=twitter_data.wcdf
Whiteboard: ETA: 9/1, prototype dashboard will be ready by then → Will resume work on 9/16; ETA: 9/23
Target Milestone: 2010-09.1 → 2010-09.2
Thanks Nelson. I was using the MV vpn and Anurag told me to use the MPT vpn. I can view it now :) Overall, it's looking great. Just a few comments: - Total tweets pie chart: I suggested in comment 37 that we replace this area with summary text instead of a chart. Is that possible? - Can we make the clickable arrows in the Top 25 users tables larger? They are difficult to click right now. - Anurag suggested making the usernames in the Top 25 users cables clickable. Clicking on "rimozas" would then show recent tweets by rimozas that include "#fxhelp" (for the #fxhelp table) or "firefox" (for the Firefox table). - Could we replace the word "entries" with "users" at the bottom of the user tables? :) Also, I'm noticing some changes from the mockup in comment 28 and want to make sure we're implementing the mockup correctly. Specifically: - Total tweets bar chart: Gray bar is supposed to be #fxhelp, not @FirefoxCares. This shows how many tweets included "#fxhelp" for that day. Adding a response rate line graph (mentioned in comment 37, 4th paragraph) would help a lot too. - @FirefoxCares analysis: This should be analysis for the @Firefox account. We could display both by adding a dropdown field to select the account. However, we're mostly interested in the @Firefox account. - Top 25 users tables. As shown in the mockup, these should be Top 25 users on 1) #fxhelp tweets and 2) Firefox tweets. Also, some of the column headers in the prototype (basedate and period) are difficult to understand. Could we make this more clear or use the headers from the mockup in comment 28? Anurag might have some ideas here. Thanks!
Another thing we talked about before was to rename @FirefoxCares to @FirefoxHelp, to make the connection to the website stronger. I'm assuming we aren't hardcoding "FirefoxCares" anywhere in the app, but if we are, we should probably move forward with the change asap to avoid additional work later down the road.
David, thanks for mentioning that. I don't think we had mentioned in this bug or to metrics before. The @FirefoxCares twitter account will be renamed to @FirefoxHelp very soon - probably in the next few days. Anurag, can existing data collected about @FirefoxCares be merged with @FirefoxHelp data once we change the account name?
> can existing data collected about @FirefoxCares be merged with > @FirefoxHelp data once we change the account name? Yep. Just tell when
Thanks Pedro - we'll comment in this bug once the change is made.
Checking on the status here. Has work resumed on this? Are we still on track for finishing on 9/23? Thanks
Pedro asked me point out the Summary text area mentioned in comment 37 that will replace the existing pie chart. This summary section would be *very very* helpful for us to have if we can include it :)
We expect to finish the first dashboard by the 23rd, with most (if not all) of your suggestions implemented.
An updated version of the dashboard was just uploaded. http://cm-metricsapp01:8080/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=/twitter&file=twitter_data.wcdf Most of the comments above were followed. There's still work to be done: - summary data: will include totals for the reference date, DoD and WoW variation for both #fxhelp and @Firefox - styling of selectors (both the date input and the period selector will turn out very different - see mockup above); - clickable usernames: when clicking on a user a pop-up will show the user's recent tweets with that keyword - Keyword analysis: another dashboard (the Twitter overview #2 on the top menu) that will include a word cloud for the different keywords for which we're capturing data. Sorry about the delay, a lot of the components of the dashboard were developped specifically for this purpose and we ran into (more than) a few unexpected bugs.
Whiteboard: Will resume work on 9/16; ETA: 9/23 → ETA 1st dashboard: 9/27; ETA 2nd dashboard: 9/30
Looking much, much better. I'd like to see the 3 coding changes below, and they're a higher priority than the keyword analysis. Coding changes... Overall tweets: - Would like to change our metric for the gray bars. Instead of showing daily totals for the "@firefox" keyword, I'd like to show the daily totals for what we are considering tweets that could be responded to for our Community Care program. Let's label these bars as "Firefox tweets". - "Firefox tweets" includes all tweets for the "firefox" and "@firefox" keywords **unless** they include one or more of these conditions: 1. a url (contain "http://"), 2. a reweet (starts with "rt" or "RT"), 3. mentions not to the @firefox account (starts with "@" and does not start with "@firefox"). Examples: - Included: "@firefox how do I stop crashes?"; "my firefox is broken"; "i want to contribute to firefox" -Excluded: "@joesmith firefox rocks", "firefox 3.6.10 update: http://mozilla.com"; "RT @sally I love firefox" Twitter Summary: - Mockup (what I'd like to see): http://people.mozilla.com/~williamr/communitycare/summary-mockup.png - "Army of Awesome" Response % for yesterday and for last 7 days overall. Response % = # of tweets with "#fxhelp" keyword / # of "Firefox tweets" that meet the conditions above (eg: # of tweets that could be responded to). Let's also include DoD and WoW variance for the response % - Number of unique twitter users for "#fxhelp" keyword for yesterday and last 7 days. Also show DoD and WoW variance. We'll call this line "# of Contributors" - Response %, # of tweets from @firefox, # of tweet to @firefox for yesterday and last 7 days. Also show DoD and WoW variance. @Firefox analysis: - Let's remove the "# of tweets" gray line from this graph. It doesn't appear to add any value since it just adds up the from @firefox and to @firefox values. Minor text changes + a question… Overall Tweets: - Change "Overall Tweets" header to "Community Care." This makes more sense because this graph shows how many tweets could be answered and how many tweets our community is answering. - Change orange "#fxhelp" legend label to "Army of Awesome". That's the nickname for the people tweeting the #fxhelp hashtag, so it's easier for us to understand. Both "Top 25 Users" table headers: - Still confused about what "base date" means. Please explain. - Change "Over period" to be the actual period name that's in the dropdown box at the top (ex: "Last 7 days") "@Firefox analysis" header: change to "@Firefox account" All graphs that have a date x-axis: Can we display the abbreviated day of the week under the date? I think we had this before. Ex: "9/23 Thurs"
About the tweets to include/exclude: are you sure you want to exclude ALL tweets with "firefox" and an URL? There are many tweets that have that and are not from @Firefox, they're from other users; do you really want to exclude these? Examples of tweets that would be excluded: "@Firefox: my firefox is crashing when I go to http://www......", "my firefox displays site http://.... all wrong" or "check out my awesome add-on on http://addons.mozilla.org/...." Exclude/include list: the remaining items can be implemented. May take a few days as they require a redesign of parts of the data extraction scripts and a data warehouse redesign to accommodate the necessary extra tags. About the top 25 tables: the first column refers to the date selected on the upper right corner (that specific day only); the second column refers to the date range selected (using the reference date and period selectors from upper right corner); And overall is, of course, since we started gathering data. This was our goal with the three column layout. But I totally agree, the names of the columns are too confusing. We can change the titles, but I just didn't come up with anything better. What would you like to see on the tables? About date formats: the current chart component we're using doesn't have date formats implemented. We're working on it and expect to have that feature available sometime this week.
Removing the block on bug 584886 since these can launch independently
No longer blocks: 584886
(In reply to comment #56) > About the tweets to include/exclude: are you sure you want to exclude ALL > tweets with "firefox" and an URL? Let's show both. One bar for "All Firefox tweets" and another for "Needs Help" (that meet conditions in comment 55). The third bar is for #fxhelp tweets, called "Army of Awesome". > > Exclude/include list: the remaining items can be implemented. May take a few > days as they require a redesign of parts of the data extraction scripts and a > data warehouse redesign to accommodate the necessary extra tags. That's okay. Please update the whiteboard with the new ETA. > This was our goal with the three column layout. But I totally agree, the names > of the columns are too confusing. We can change the titles, but I just didn't > come up with anything better. What would you like to see on the tables? Let's use these titles: Column 1: [actual reference date (eg: 9/30)] Column 2: [selected period name (eg: Last 7 days, Last 30 days)] Column 3: Overall > About date formats: the current chart component we're using doesn't have date > formats implemented. We're working on it and expect to have that feature > available sometime this week. Great Top 5 languages: - Are these daily totals for all keywords? Or just the firefox keyword?
All the changes are being implemented, except the clickable usernames on the tables, which we'll leave for after developping the word cloud dashboard, so as to have a fully functioning site asap. The data is currently being reprocessed to account for the url exclusion mentioned above and should be finished by tomorrow morning at which time we resume normal operation. As for the top locales: yes, the chart is for the firefox keyword only.
Thanks Nelson. Can you update the whiteboard with the new ETA dates when you get a chance?
Done
Whiteboard: ETA 1st dashboard: 9/27; ETA 2nd dashboard: 9/30 → ETA 1st dashboard: 10/01; ETA 2nd dashboard: 10/08
The firtst dashboard is complete. You can see it in http://cm-metricsapp01:8080/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=/twitter&file=twitter_data.wcdf I did promise it on October 1st and... well, it's still October 1st PDT ;) Performance is bad. All data is on mysql database which doesn't behave well with this type of queries and no optimizations were done yet. The first time the dashboard is rendered it will take about 1 minute to finish. However, the second time you hit the dashboard within 1hour it will display cached results and will load in just a few seconds.
This might be a bug: hovering over the gray and orange bars in the Community Care graph shows NaN instead of the number of tweets. Exporting the data shows the numbers fine though.
Also, clicking on the Choose Date dropdown shows Chinese characters instead of English.
Thanks for reporting it - we'll fix it asap
Moving project level "bugs" (i.e. new features, new data analysis requests) to enhancement rather than bug.
Severity: normal → enhancement
The two dashboards are ready and running on the production server. There are, however, some issues, which one must bear in mind (they'll be addressed as soon as possible): - Data issues: the twitter API is not very reliable in providing us with complete sets of data and our data extraction scripts struggle with errors and missing data somewhat often. Last week, because we needed to figure out a way to correctly parse tweets in asian alphabets (most notably japanese), we had to shut down data collection on October 7th. The search should have recovered those tweets, but did not. We're trying to figure out a solution that would allow us to fetch older data when something goes wrong, but for the time being we don't have one; - Speed: because data is (for the time being) on mysql and no optimizations were made, the queries take a long time. We're going to move everything to a faster database, create agregation tables with pre-calculated values and pre-warm the caching system with the most common queries, but this is on hold until all issues with data collection have been sorted out; - Tag cloud: although we added a lot of stop words in several languages (namely, articles, prepositions, pronouns), there are a lot of words that must be added to this list and removed from the tag cloud data extraction scripts. This will, of course, take time and also help from native speakers of languages nobody in our team speaks fluently (namely, every language but Portuguese, English, Spanish and French). - Functionality not yet implemented: both usernames on the Twitter awesomeness dashboard and Tags on the Trending topics dashboard will be clickable. However, this will only be implemented on a later stage, when we have enough resources to tackle new features and improvements. So, without further delay, here's the url for the dashboard: https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=%2Ftwitter&file=twitter_data.wcdf
Since this dashboard is now on a production server, can we add it to the main Metrics dashboard page? There's a nice place for it under the More section :) If so, I'm happy to file a bug to have it added. Thanks
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Target Milestone: 2010-09.2 → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: