384408 - Firefox productization for Hindi (hi-IN)

Status: RESOLVED FIXED

(Mozilla Localizations :: hi-IN / hindi, defect)

Description

[Axel Hecht [:Pike]]

We'll need to find a set of good search engines for India, in particular Hindi.
Same for RSS readers, and do we have a good news feed for India? Is http://www.bbc.co.uk/hindi/ any good?

Rajesh, any suggestions?

Mic, I guess there should be more people on CC here, can you find out who?

Comment 1

[Rajesh Ranjan]

As far as I know google gives best result for all langs and particularly for Hindi.


You are right :)
http://www.bbc.co.uk/hindi/ is ths best news feed for India in Hindi.

Comment 2

[Mic Berman]

hi
working on cc list for others to comment, will post something shortly
mic

Comment 3

[Axel Hecht [:Pike]]

Seth, this is the bug for the search and feeds changes for the Hindi localization. Could you CC the people you wanted to comment here, too?

Comment 4

[Seth Bindernagel]

Axel, the people who are doing research on this are management school students.  Once we get deeper into the project, which should be sometime in late August '07, I will introduce them to the bug tracking system.

Comment 5

[Axel Hecht [:Pike]]

Putting this bug on Mic's radar a bit more prominently.

Comment 6

[Vijay Haryal]

I am not able to attach the report as it is 522 KB and the maximum attachment allowed is 300 KB. I have mailed the report to Seth.The references of page numbers given below are from that report. Request Seth to please upload it.

The document attached gives the top of the mind recall and the most used web portals by the Indian user
1)For the news feed, as mentioned on Page 11 , Times of India is the favorite news paper for the internet users , followed by Hindu. The top news websites is indiatimes, ndtv and rediff( Page 10) . The feeds of these can be added
2)India Today is the most preferred magazine of the internet users ( Page 11)
3)Preferred websites for top financial Info are Money Control &Yahoo ( Page 10)
4)Google leaves everyone far behind in information search (Page 9)
5)Top online shopping websites are Ebay, Rediff and Indiatimes(Page 3)
Thanks
Vijayendra Haryal
Team Lead,Team 21
IIM Ahmedabad

Comment 7

[Rajesh Ranjan]

Things mentioned in comment #6 are mostly of English language,but this bug wants it for Hindi. I also want to read the full report. 

Comment 8

[Vijay Haryal]

There are no clear winners in Hindi as of now.
In a survey that was conducted, 18% of the net users highlight Hindi as their preferred language of reading, 17% also indicate that they visit Hindi websites already.This phenomena over language preference is beginning to emerge.
We would do more research and let you know but as of now. Chances are bleak that we will find any reliable source of information which will give us clear preferences for Hindi websites. 
The only way to do this is to conduct some primary research .

Comment 9

[Vijay Haryal]

Some facts about regional portals-
1)It is still a market in its infancy - industry estimates that out of 20 billion pages globally on the Net, there are only one million Hindi pages (Tamil would have half-a-million).
2)Over 90 per cent of Indian portals are in English. A recent study shows that the top search engines together (includes Google and AOL, MSN and Yahoo) were able to identify over two lakh Hindi website links on the portal bbc.co.uk/Hindi - again reflecting the limited nature of Hindi content available on the Net. 
3)Delhi based Indicus Analytics  found that the biggest problem for users in Hindi was how to access pages in Hindi.You could not use existing popular search engines, as you could not query in Hindi. The company launched Raftaar.com - a search engine with a soft key, so that you can type in Hindi alphabets (which are on the screen) with a click of the PC button.
4)Microsoft as part of its special initiative launched "Bhasa India" a community portal in various Indian languages which provides users a platform to express their views on regional language computing, attract developers to post queries and even use the knowledge of users to create technical glossary of software and hardware technology. The result: it is getting over 300,000 hits a month and has over 20,000 odd registered users.
5)"Only five per cent of internet advertising goes to regional languages - the rest is all with English portals.
Source: Rediff.com

Comment 10

[Vijay Haryal]

Attachment: Search Engines in India

Here is a report on the various search engines in India. Again , there are no clear winners as of now.

Comment 11

[Rajesh Ranjan]

Thanks Vijay!
I have some different views on the stats shown.

Things started changing very fast in last few years and particulary last two years. Earlier there were several technical problems that were barrier in the path of language preference. You just remember the television early days when dominance of english channel existed for few years. But when vernacular language television revolution started, market share for English became very thin. So its hard to believe the statistics. Anyway there is also a saying about stats...lies, damn lies and statistics. What happened in television field already will be replicated on Internet also. Among 10 most circulated newspaper probably one one english newspaper is coming. Please go through the following sites. You can see the changes and inclination of the major player of web. 

http://in.jagran.yahoo.com/
http://in.hindi.yahoo.com/
http://content.msn.co.in/hindi/Default
http://navbharattimes.indiatimes.com/
http://news.google.co.in/news?ned=hi_in

You correctly mentioned that the biggest problem for users in
Hindi was how to access pages in Hindi. That was a problem of past. So I am telling in past few years 
things changed entirely after unicode came into the scene. So its was a problem of past that we were unable to query in Hindi in search engine. So we have to think according to new outlook. And again google seems ahead of all. The seach engine you mentioned are of very limited users though these are made for India search.

Comment 12

[Vijay Haryal]

As far as the statistics go, I already mentioned that secondary sources would not be able to give a clear picture. 
Things are changing agreed, but the market is still in the infant stage. Both MSN and Yahoo have launched their regional portals in 2006. The results have not been very clear about the success.
However, one interesting statistic is that out of the 18% of the net users highlight Hindi as their preferred language of reading, 17% also indicate that they visit Hindi websites already. The trend is good as u rightly mentioned and the potential is there no doubt.
Also as far as the search engines are concerned, we mentioned before that there are no clear winners and google is the right choice.
The optimism with the regional languages is good, but we strongly recommend a reality check.
We did conduct some indepth interviews and found out that an interface in Hindi is important. We would be sharing more insights tomorrow in the meeting with Seth

Comment 13

[Seth Bindernagel]

Attachment: Online Trends in India, created by student team at Indian Institute of Management, Ahmedabad

Online Trends in India, created by student team at Indian Institute of Management, Ahmedabad

Comment 14

[Seth Bindernagel]

Attachment: (part 2) Online Trends in India, created by student team at Indian Institute of Management, Ahmedabad

(part 2) Online Trends in India, created by student team at Indian Institute of Management, Ahmedabad

Comment 15

[Mic Berman]

Hey everyone
reading the posts above and the work Team 21 I'd like some reactions to a proposed list of search engines:
Google
Yahoo
eBay
Rediff (?? should we use this, I'm not sure if they have their own search engine built for Hindi, if not Google might be just as good?)
Wikipedia
Others?? e.g., is there a better local Amazon type provider?

I'm not sure about Amazon as it doesn't seem to even register as a site well known or used by users - having said that I'm not sure if this data is based only on english users vs hindi
Also what about 
Answers? I don't see that ranked highly either

I look forward to your thoughts

Comment 16

[Mic Berman]

on RSS feeds is BBC sufficient or should it be a local provider that specializes in Hindi language news?
thanks

Comment 17

[Rajesh Ranjan]

List of search engines are good enough. 
The reputation of BBC is very good in the Indian audience and generally people of India think that service of BBC is unbiased. But BBC is anable to provide all the remote news that all are coming from villages and small towns of India.
So for RSS feeds we should take some sites that have good reach all over Hindi belt. For example,
http://in.jagran.yahoo.com/news/

Comment 18

[Mic Berman]

Rajesh/Vijay

on RSS Feeds - earlier in the bug Comment #1 it seemed BBC was good but in comment #17 you're suggesting in.jargran.yahoo.com/news. Please advise which is your preference.

on Search engine's 
thanks for your comments Rajesh. Could you offer more specific comments on my questions about Rediff and a local version of Amazon see comment #15
Vijay - any thoughts in reaction to comments #15 or #17?

thanks
mic

Comment 19

[Vijay Haryal]

For search Engines, Google should do the needful. 
For newspapers , here is the trend. I will cover more in a short report later. 
Top ten list (present readership and decline/growth figures):
1. Dainik Jagran (Hindi) – readership of 1.65 crore (1 crore = 10 million). Decline by 3.5 per cent.
2. Dainik Bhaskar (Hindi)- readership of 1.28 crore. A growth of 2.4 per cent.
3. Malayala Manorama (Malyalam) - readership of 86.62 lakh (one lakh is one hundred thousand) A decline of almost 2 lakhs.
4. Hindustan Times (English) - readership of 85.51 lakh. A decline of 5.5 per cent.
5. Amar Ujala (Hindi) - readership of 80.75 lakh. A decline of 2.1 per cent.
6..Daily Thanthi (Tamil) - readership of 79.1 lakh. A decline of 5.2 per cent.
7. Rajasthan Patrika (Hindi) – readerships of 74.02 lakh. And increase of 6.5 per cent, highest increase amongst the top 10 dailies.
8. Eenadu (Telugu) - readership of 69.19 lakh. A decline 4.34 per cent.
9. Ananda Bazar Patrika (Bengali) - readership of 68.55 lakh. An increase of 4.84 per cent.
10. Times of India (English): readership of 68.28 lakh. An increase of 0.69 per cent.
( 2007 survey: Source Rediff.com)
One more thing which we need to look into very seriously is compatibility of fonts .
For example , please have a a look at http://www.amarujala.com/today/default.asp
it's not able to show the Hindi Font. This is very disturbing.
While the same page is opening up in Internet Explorer very well in Hindi. 
I would request someone from the technical team to please look into it and offer us their comments
Based on this our approach will be determined as to which provider to suggest

Comment 20

[Vijay Haryal]

The trend is that the print readership is declining in India which is similar to the Global Trend.Some points to note are
1)Only ET ( Economic Times), an English Business Newspaper, could maintain it's readership and even grown in the Delhi and Mumbai. 
2)Among the top ten listed, six out of ten show declining trend ( including English).Times of India and Anand Bazar Patrika ( No 10 and 9 respectively) were not there in the top 10 last year. They have replaced Lokmat, which is the largest read Marathi daily
3)Amongst all the southern newspapers, Malayalam is doing the best
If we take the success of  print readership of a news paper as a proxy for the online readership as well, Dainik Jagran and Dainik Bhaskar seem to be the obvious choices

Comment 21

[Rajesh Ranjan]

in.jargran.yahoo.com/news is still in Beta...So BBC we can choose. 
Rediff is okay. I can not say more about local amazon type provider.

Comment 22

[Vijay Haryal]

We think that, we can rely on print readership as a proxy for online and therefore Dainik Jagran and Dainik Bhaskar seem to be appropriate choices.
As Dainik Jagran is still in Beta we may like to go with http://www.bhaskar.com/
Apart from being the second most read daily, it has "localized news" for every major town in the Hindi speaking states and secondly as the Hindi script is compatible with Firefox.

Comment 23

[Rajesh Ranjan]

if we decide going by this logic better go in.jargran.yahoo.com/news
Jagran has base in two most Hindi speaking populated states ie Uttar Pradesh and Bihar where Dianik Bhaskar has no base.

Comment 24

[Vijay Haryal]

If we are fine with the Beta version, then in.jargran.yahoo.com/news is the obvious choice as I mentioned before.
As far as Dainik Bhaskar is concerned, it is the second most read daily in India.Dainik Bhaskar is a Hindi-language daily newspaper of India. It first started in the state of Madhya Pradesh. Currently it is published from many cities of North India.
The Dainik Bhaskar Group has soared to the top of the print media industry with its flagship Hindi daily,  Dainik Bhaskar  and the Gujarati frontrunner,  Divya Bhaskar. Dainik Bhaskar has a large presence in Madhya Pradesh, Rajasthan, Punjab, Haryana, Himachal Pradesh, Chhatisgarh and Uttar Pradesh; while  Divya Bhaskaris the largest circulated regional daily in Gujarat.

Comment 25

[Mic Berman]

Hi
actually I talked with Chofmann about this to get more clarification. 
We default ship with BBC as its the best coverage and our rationale was to include it such that the user might go "what's this" and realize that they can add additional RSS feeds into the browser. If BBC provides sufficient coverage in Hindi than we should keep it as is. If BBC does NOT provide sufficient coverage then lets look at other options. 
Please LMK your thoughts on which we should ship with in light of this further clarification.
Thanks
mic

Comment 26

[Vijay Haryal]

BBC won't give the local flavor ( Rajesh, please correct me if I am wrong). By this I mean the localized news of the city won't be there. This is there in case of The hindi dailies which we have mentioned above.
The local news of the city is an important differentiator.

Comment 27

[Rajesh Ranjan]

I think Dainik Bhaskar is not coming from Uttar Pradesh and also bihar jharkhand etc.
So population and area wise jagran is leading Hindi Daily. Yes certainly it is more localized. If for feed we need all major news and views then its BBC that we should choose. If we need more local flavour then jagran is okay.

I have a question, whether we are going to adopt this choice forever or we can change from time to time?
 
I am continous travelling these days. So delay in replying. Sorry for this problem.

Comment 28

[Axel Hecht [:Pike]]

Back to search for a moment, looking at the stats on wikipedia, the Hindi section there seems to be growing nicely, I'd think we should add it. I guess that rediff is on our agenda, too?

I'd like to see a patch adding the two, you can take a sneak peek at what runa did in bug 398992, and the files are at http://mxr.mozilla.org/l10n/source/bn-IN/browser/searchplugins/ (or in cvs). The change from bn to hi in wikipedia should be fairly trivial.

As for the feed:

We can change that, but I'd rather not do that often. We're essentially changing other peoples bookmarks each time we do that, as the bookmark goes to a redirect on our server, and we just change the redirect. And messing with other peoples profile isn't nice :-)

I think that having the jagran feed would give a more 'Indian' touch to the Hindi localization. Given how many folks used the English builds at foss.in, I'm not sure if that's a pro or a contra, though.

It might be that the BBC feed for India is actually a good thing for en-IN, and the jagran feed is a good thing for hi-IN.

Just guessing, Mic had more discussions on the choice of feeds since her last comment, maybe there's newer insight from her now.

Comment 29

[Mic Berman]

comment #28 thanks Axel - yes there is new guidelines:
http://wiki.mozilla.org/Firefox_web_services_guidelines

comment #27
the choice can change from time to time

for RSS feed best choice is in language and popular for users. from comments #23 and #24 it seems that in.jargran.yahoo.com/news. So let's go with jargran for hi-IN

for search please confirm the list is:
google.in
yahoo.in
amazon.in
ebay.in
wikipedia.in

Rediff is only a suggestion IF it provides good search in Hindi language. If not then we can leave it with Google and Yahoo. These two are supposed to provide good Hindi language search for Global results and local results.

For RSS Readers: currently the default is Bloglines, My Yahoo, Google
Bloglines is NOT available in Hindi so please recommend whether you think it is useful to include (if users can't read english and it's not popular than I would not recommend including it). I'm not sure about My Yahoo or Google. If they are also not available in Hindi let's see if we can get a Hindi reader. Any ideas?

Comment 30

[Rajesh Ranjan]

Thank you so much. Its all okay!

For RSS Readers:  We can skip Bloglines because its not popular here. So better to keep yahoo and google.

Comment 31

[Axel Hecht [:Pike]]

Hrm. I don't find no amazon.in, and wikipedia.in should be hi.wikipedia.org, right? google.in should just be google.com, AFAICT.

Comment 32

[Mic Berman]

apologies Axel, there is not Amazon India - right Runa? so it would have to be amazon.com
and for wikipedia the url you have is correct.
google is also correct
mic

Comment 33

[Rajesh Ranjan]

Mic, 

I am Rajesh...not Runa...
Seems Bengali work is also in progress. :-)


http://india.amazon.com/

is available.

Comment 34

[Mic Berman]

Rajesh :) I'm sorry about that. (yes, I am working on the Bengali build as well)

India.Amazon.com - I had seen that site, it seems to be more of a more of a corporate information (for developers) site than a commerce site, so, shall we go back to using the default en-US search plug in for amazon?

Comment 35

[Rajesh Ranjan]

For Amazon using default en-US is better. you are right site is having only corporate info only.

Comment 36

[Axel Hecht [:Pike]]

Rajesh, can you create a patch to implement the remaining changes for search? Create a cvs diff -uN and attach the output for review before committing to CVS, please?

Changes should include:

cvs removal of google.xml, that's not needed, it's picked up from en-US at build time.
addition of search plugin for hi.wikipedia.org, likely wikipedia-hi.xml. I prefer http://mxr.mozilla.org/l10n/source/lv/browser/searchplugins/wikipedia-en.xml as example.
Changes to list.txt, notably adding wikipedia-hi and amazondotcom, so that the build actually picks up the wikipedia plugin that you created, and the amazondotcom one from en-US. The latter works like google.xml, thus there are no changes required for the google in list.txt.

Comment 37

[Mic Berman]

ping?

Comment 38

[Rajesh Ranjan]

Attachment: patch for searchplugin

pike, can you please review it and let me know if something more is needed from my side.

Comment 39

[Axel Hecht [:Pike]]

Comment on attachment 296489 [details] [diff] [review]
patch for searchplugin

Sorry for the lag.

r-, two things: You don't need amazon.xml, that's picked up from mozilla/browser/locales/en-US/searchplugins by the build process. The changed to list.txt is right, though.

You need to 

cvs add wikipedia-hi.xml

You might want to look at http://mxr.mozilla.org/l10n/source/it/browser/searchplugins/wikipedia-it.xml again, we improved the current ones a bit just in the last few days, most of that is copy and paste. To guide you before, the <ShortName> should be "विकिपीडिया (हिन्दी)", that's what hi.wikipedia.org uses. The template should be either http://hi.wikipedia.org/wiki/Special:Search or the translated one, though I don't even find the translated name mediawiki uses for Search. Looking at other languages, there might not be one, so I'd just use Special:Search, that works.

Anyway, then do a 

cvs -z3 diff -uN

The 'N' makes new and removed files show up in the diff, so that I can actually review those.

Comment 40

[Rajesh Ranjan]

was not well last week, so will do this week.

Comment 41

[Rajesh Ranjan]

Hi Pike,

I just commited the changes... Can you please review...

Comment 42

[:Gavin Sharp [email: [e-mail]]]

(In reply to comment #41)
> Hi Pike,
> 
> I just commited the changes... Can you please review...

Hi Rajesh,

It's generally best to generate a diff, using |cvs diff -up8N|, and attach that here *before* committing to CVS. Much easier to review that way. I will post comments on the file you've committed to CVS, though.

Comment 43

[:Gavin Sharp [email: [e-mail]]]

><SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">

The "xmlns:os" part isn't needed, just remove it.

><ShortName>Wikipedia (Hindi)</ShortName>
><Description>Hindi Wikipedia</Description>

You should use the values from the Wikipedia plugin for these, unless there is some reason not to. This will ensure that users don't get prompted to install the engine again when they visit Wikipedia.

<ShortName>विकिपीडिया (हिन्दी)</ShortName>
<Description>विकिपीडिया (हिन्दी)</Description>

Could you make these changes, then generate a diff using |cvs diff -up8N|, and attach the output here?

Comment 44

[Rajesh Ranjan]

Attachment: searchplugin patch

Hi Gavin,

Can you please review the latest patch ?

Comment 45

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 302989 [details] [diff] [review]
searchplugin patch

Looks good, thanks!

Comment 46

[Mic Berman]

are all changes complete then? can we close this bug?

Comment 47

[Axel Hecht [:Pike]]

Attachment 302989 [details] [diff] didn't land yet.

Rajesh, would you land that change, referencing this bug, describing the change, and mentioning gavin's review? Something like 
"bug 384408, updating wikipedia plugin to match wikipedia's site, r=gavin" 
would work in this case.

Comment 48

[Rajesh Ranjan]

Comment on attachment 302989 [details] [diff] [review]
searchplugin patch

bug 384408, updating wikipedia plugin to match wikipedia's site, r=gavin

Comment 49

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 302989 [details] [diff] [review]
searchplugin patch

I'm assuming this was just a mistake...

Comment 50

[Rajesh Ranjan]

sorry. I did mistake.

I landed the change. please chk if everything is right :)

Comment 51

[:Gavin Sharp [email: [e-mail]]]

(In reply to comment #50)
> I landed the change. please chk if everything is right :)

Yeah, looks good. Thank you!

Comment 52

[Axel Hecht [:Pike]]

Attachment: update wikipedia plugin to match hi.wikipedia.org

The title in the search link on hi.wikipedia.org changed, let's match up with that. I switched the suggest Url over to use Params, too, as I was already on it.

Rajesh, can you review the patch? And land it, if you like it. Again with a comment referencing this bug, something like "bug 384408, update wikipedia plugin to match site, patch by [e-mail], r=me" or so ("me" in this case would mean you, as you land).

Comment 53

[Axel Hecht [:Pike]]

Comment on attachment 309075 [details] [diff] [review]
update wikipedia plugin to match hi.wikipedia.org

Rajesh, you landed the first chunk of this as part of bug 425688, could we still get the suggestions part fixed?

Comment 54

[Rajesh Ranjan]

Attachment: searchplugin_suggestionpartdiff

Please chk it for the suggestion part.

Comment 55

[Axel Hecht [:Pike]]

Comment on attachment 314311 [details] [diff] [review]
searchplugin_suggestionpartdiff

r=me, please check in with a good comment again.

Comment 56

[Rajesh Ranjan]

committed the same.

Comment 57

[Axel Hecht [:Pike]]

http://in.jagran.yahoo.com/news/ doesn't seem to have a feed, AFAICT.

Comment 58

[Rajesh Ranjan]

At this page http://in.jagran.yahoo.com/news/national.html  we can find feed for different catagories like 

http://in.jagran.yahoo.com/news/rss/national/politics.xml
http://in.jagran.yahoo.com/news/rss/national/crime.xml

and similarly for other catagories also. But it seems there is not a single place to get all. better we should go according to the description and comment #1 of this bug :-).

Comment 59

[Axel Hecht [:Pike]]

Mic, http://www.bbc.co.uk/hindi/index.xml good for you, too?

Comment 60

[Mic Berman]

per comment #59 and comment #1 - find change. thanks

Comment 61

[Rajesh Ranjan]

can we move ahead to make it final and resolve?

Comment 62

[Mic Berman]

i'm ok with that. 
axel did you already create a it server side bug for this? 

Comment 63

[Axel Hecht [:Pike]]

Not yet. I'll need to do some source verification before closing this bug, too.

Comment 64

[Mic Berman]

here's what needs to happen to close this bug which Axel will do
bloglines needs to be removed from list per comment #30
amazon.com needs to be added again per comment #39
server side change for http://www.bbc.co.uk/hindi/index.xml has happened bug 440802

Comment 65

[Axel Hecht [:Pike]]

Attachment: add amazon to search, drop bloglines from rss readers

Here's the patch to fix the remaining issues, gonna land that in a minute.

Comment 66

[Axel Hecht [:Pike]]

Checking in searchplugins/list.txt;
/l10n/l10n/hi-IN/browser/searchplugins/list.txt,v  <--  list.txt
new revision: 1.5; previous revision: 1.4
done
Checking in chrome/browser-region/region.properties;
/l10n/l10n/hi-IN/browser/chrome/browser-region/region.properties,v  <--  region.properties
new revision: 1.4; previous revision: 1.3
done

Marking FIXED with reference http://hg.mozilla.org/users/axel_mozilla.com/l10n-src-verification/index.cgi/file/ccc442125f2c/reference/MOZILLA_1_8_BRANCH/hi-IN/.