98144 - Search date isn't system locale sensitive

Status: VERIFIED FIXED

(MailNews Core :: Filters, defect)

Description

[Jo Hermans]

I was looking for mails after July 1st, so I made a search filter like this :

'Date'     'is after'     1/7/2001

(I'm in Europe, and I'm using day-month-year)

To my surprise it also found all my mails since January 7th. Ok, so it was using
M/D/Y for its input field. But then I saw that the 'Date' field in the table
below, was using the D/M/Y order. So that was localized correctly.

Please localize the input-field too, according to the users prefs (it comes from
the OS, right ?). Since the expected order wasn't indicated at all, and no
errors are generated when you supply an incorect date (like 20/20/2001), the
user will never know in what order to supply the date

Comment 1

[Jo Hermans]

Build 2001080110 (regular 0.9.3) on NT. My regional settings (NT) are 'd/MM/yy'
and 'dddd d MMMM yyyy'

Comment 2

[Chase Tingley]

-> filters

Comment 3

[Christopher Hoess (gone)]

Adding keyword.

Comment 4

[ji]

[e-mail], what's the date format on the thread pane (the mail 
list pane on the mail window)? It should be consistent with your system setting.
I can't reproduce the problem with 09/05 build on my Japanese win98. My system 
date format is YY/MM/DD, and the thread pane has the same format.

Comment 5

[ji]

Switched to Japanese NT. I can see the problem, actually it's impossible to 
create a filter with the date format consistent with thread pane. For example, 
the thread pane is in YY/MM/DD format, which is  same as system date. After I 
create a filter with criteria set to "Date" "is after" "01/09/04", then open 
this filter again from Edit button, the date in the criteria has changed to 
1/9/1904. Confirmed the bug.

Comment 6

[Jo Hermans]

thread pane: DD/MM/YY
date-filter: MM/DD/YYYY (select 'date is', and it automatically shows you the
current date)
search-result: DD/MM/YY

Comment 7

[Frank Tang]

the problem is the date widget itself. Reassign to ftang . 

Comment 8

[Frank Tang]

ok, the problem exist in mail searching, mail filtering (and what else). 
The problem code is in the following three functions. we need to change them 
together:

convertPRTimeToString
convertDateToString
convertStringToPRTime
 
they are defined in 
/mailnews/base/search/resources/content/searchTermOverlay.js

they are called by /mailnews/base/resources/content/mailWidgets.xml

they are implemented by sspitzer
cc sspitzer

Comment 9

[Frank Tang]

which mean we need a i18n enabled date widget

Comment 10

[Frank Tang]

shanjian-
it seems we need to design a i18n enable date picker. do you want to look into 
it ?

Comment 11

[Shanjian Li]

accepting.

Comment 12

[ji]

Nominating for nsbeta1, since this breaks search by date function.

Comment 13

[nhottanscp]

nsbeta1+

Comment 14

[laurel]

*** Bug 117551 has been marked as a duplicate of this bug. ***

Comment 15

[Shanjian Li]

We are currently using javascript's Date.Parse to parse date string, and it 
only accept  IETF standard date syntax and continental US time-zone abbreviations.
To accept date by parsing a string is never a good idea, the locale difference 
will confuse many users. We should really implement a date picker. 

I am not sure if I could fix this in mozilla1.0 time frame. 

Comment 16

[ji]

*** Bug 126795 has been marked as a duplicate of this bug. ***

Comment 17

[Shanjian Li]

Date input is always confusing to international user, and simplily using locale
sensitive parsing format will only make things more confusing to user, since
"locale" is hidden from user. The typical approach is to implement a date
picker. There is some date picker sample implemented with JS.
"http://www.google.com/search?hl=en&q=date+picker&btnG=Google+Search" will list
some of them. But to reimplement with XUL is not a simple work for people like
me that seldom touch xul programming before. I really don't want to distract my
focus at this time. Besides the implementation of date picker don't need i18n
knowledge. Reassign to don. 
Don, please reassign it to the right person, thanks!

Comment 18

[Shanjian Li]

Blaker seems to be the more appropriate person to own this. 

Comment 19

[Blake Ross]

Why am I the right owner for this? I'm confused.  Is this bug to implement a new
date picker?  I know that Joe had at some point an xbl date picker, but I don't
know how locale aware it was. I don't think using a new, untested widget is the
course we want to take here if there's an easier fix. But I'm pretty sure I
shouldn't own this bug.

Comment 20

[Peter Trudelle]

removing plus, we don't have authorization for any new feature work.  What is
the business need?  Who is paying for it? cc kmurry, evelyn

Comment 21

[ji]

This is not a new feature. The problem is that searching  mail by date is broken
for international users who use different date format as English. For example,
Japanese users use date format as YYYY/MM/DD, and searching mails by their own
date format won't work.

Comment 22

[ji]

If there is no time to come up with a good solution for this, as a temporary
solution or workaround maybe we should indicate what date format be used for
search by adding string "MM/DD/YYYY" following "Date" in the drop-down list. So
the search criteria becomes something like "Date (MM/DD/YYYY)" "is/isn't/is
before/is after" <user input date>. So the user will know what date format
should be used for search. And then we need to add localization notes to ask
localization team not translate this to language format in localized builds.

Comment 23

[nhottanscp]

I agree with the string change.
Reassign to nhotta.

Comment 24

[Alexander Zink :alex]

If we agree to not care about any locales, why not use the international 
standard then?
According to ISO (as well as EN, DIN, and many other standardization 
organizations) the universal date format shall be YYYY-MM-DD.

Since the US standard for the date format is really confusing and inconvenient 
(especially for European users, since just day and month positions are 
swapped: 5/2/2001 or 2/5/2001 ?), I vote for the one universal date format: 
2001-02-05 (although 2001-2-5 and maybe even 2001/2/5... could be accepted as 
well (for typing convenience)).

The exact ISO standard is ISO-8601. An impressive list of worldwide adoptions of 
this standard can be found here: http://www.qsl.net/g1smd/isoimp.htm

The URL http://www.saqqara.demon.co.uk/datefmt.htm lists some good reasons why 
to use this format worldwide.

Any opinions on that?

Comment 25

[nhottanscp]

The workaround is proposed because of the date format parsing issue (see comment
#15). 

Comment 26

[Frank Tang]

nsbeta1- per triage meeting/
ji will open a new bug for the word around and we will nsbeta1+ that one . ji,
please reassign the new one to rchen and nsbeta1+ that one.

Comment 27

[ji]

Filed bug 129547 for the workaround.

Comment 28

[Jo Hermans]

*** Bug 160641 has been marked as a duplicate of this bug. ***

Comment 29

[nhottanscp]

remove minus

Comment 30

[nhottanscp]

Attachment: Chagned to make search date format localizable.

Comment 31

[Jean-Francois Ducarroz]

Comment on attachment 105659 [details] [diff] [review]
Chagned to make search date format localizable.

R=ducarroz.

I have 2 comments:
1) should we have a fallback when gSearchDateFormat is out of range?
2) why the default format is not the Swiss one :-)

Comment 32

[nhottanscp]

Attachment: Added a validation code for the property value.

The default used for the error fallback is the current format before the change
which is mm/dd/yyyy.

Comment 33

[(not reading, please use [e-mail] instead)]

Comment on attachment 105665 [details] [diff] [review]
Added a validation code for the property value.

looks good, just a few nits.

1)

I'd prefer it if you did this:

+function initializeSearchDateFormat()
+{
+  if (gSearchDateFormat)
+    return;

so that you can do this:

function convertDateToString(time)
{
  var year, month, date;
  initializeSearchDateFormat();

and 

function convertStringToPRTime(str)
{
  initializeSearchDateFormat();

2)

why not use a switch statement?

+  if (gSearchDateFormat == 1)
+    dateStr = year + sep + month + sep + date;
+  else if (gSearchDateFormat == 2)
+    dateStr = year + sep + date + sep + month;
+  else if (gSearchDateFormat == 3)
+    dateStr = month + sep + date + sep + year;
+  else if (gSearchDateFormat == 4)
+    dateStr = month + sep + year + sep + date;
+  else if (gSearchDateFormat == 5)
+    dateStr = date + sep + month + sep + year;
+  else if (gSearchDateFormat == 6)
+    dateStr = date + sep + year + sep + month;
+

3)

same thing, use a switch

+  if (gSearchDateFormat == 1)
+  {
+    year = arrayOfStrings[0];
+    month = arrayOfStrings[1];
+    date = arrayOfStrings[2];
+  }
+  else if (gSearchDateFormat == 2)
+  {
+    year = arrayOfStrings[0];
+    month = arrayOfStrings[2];
+    date = arrayOfStrings[1];
+  }
+  else if (gSearchDateFormat == 3)
+  {
+    year = arrayOfStrings[2];
+    month = arrayOfStrings[0];
+    date = arrayOfStrings[1];
+  }
+  else if (gSearchDateFormat == 4)
+  {
+    year = arrayOfStrings[1];
+    month = arrayOfStrings[0];
+    date = arrayOfStrings[2];
+  }
+  else if (gSearchDateFormat == 5)
+  {
+    year = arrayOfStrings[2];
+    month = arrayOfStrings[1];
+    date = arrayOfStrings[0];
+  }
+  else if (gSearchDateFormat == 6)
+  {
+    year = arrayOfStrings[1];
+    month = arrayOfStrings[2];
+    date = arrayOfStrings[0];
+  }

Comment 34

[(not reading, please use [e-mail] instead)]

once you fix those nits, sr=sspitzer

Comment 35

[nhottanscp]

Attachment: Changed the init code, change to use switch.

Comment 36

[nhottanscp]

Comment on attachment 105985 [details] [diff] [review]
Changed the init code, change to use switch.

copy r=ducarroz, sr=sspitzer

Comment 37

[nhottanscp]

checked in to the trunk

cc to yxia
there are two values in messenger.properties can be localized
mailnews.search_date_format
mailnews.search_date_separator

Comment 38

[:deanis74]

Was it not possible for us to use the system date format?

Comment 39

[nhottanscp]

Unlike date formatting, parsing locale formatted string is not usually provided
by OS API. Implement it internally is not trivial. I made this specific place
localizable for now.

Comment 40

[nhottanscp]

Attachment: Additional change to auto-detect common short date format.

Comment 41

[nhottanscp]

Attachment: Additional change to auto-detect common short date format (ignore the last one).

Comment 42

[Jean-Francois Ducarroz]

Comment on attachment 106444 [details] [diff] [review]
Additional change to auto-detect common short date format (ignore the last one).

Looks good. R=ducarroz

Comment 43

[(not reading, please use [e-mail] instead)]

Comment on attachment 106444 [details] [diff] [review]
Additional change to auto-detect common short date format (ignore the last one).

sorry for the delay

sr=sspitzer

Comment 44

[nhottanscp]

checked in to the trunk

Comment 45

[ji]

Checked 12/03 trunk build on my JA system with system date set to yyyy/mm/dd,
when I search the mails with the criteria as "the date is after 2002/11/3", I
get some mails dated as 1969/12/31 in the search results pane. But the other
mails in the results pane meet the criteria.

Comment 46

[ji]

Above comment is also true for the builds w/o this fix.
So marked this as verified.

Comment 47

[Anje]

Any update on this...when is this going to be fixed?

Using Thunderbird 17.0.2
Problem still occurs.

Does not accept system date eg:
Date > is after > 28/01/2013
or
Date > is after > 28/01/13
returns everything from 'wed 4/07/12   07:42PM  to current date

Comment 48

[Anje]

Additional:
Using windows vista, thunderbird 17.0.2
RE: search Messages:
Note:
1. does not search by system date eg: 12/02/2013
2. does not allow search by date and time

if I use eg: 2013/01/12 (no one uses this in UK and my computer is not set to use it and the returned results display as per system eg: Sat 26/01/13 07:30PM)

3. parameter = is after - does not return 'after' but includes that date.
4. parameter = is before - works

Comment 49

[Wayne Mery (:wsmwk)]

(In reply to Anje from comment #47)
> Any update on this...when is this going to be fixed?
> 
> Using Thunderbird 17.0.2
> Problem still occurs.
> 
> Does not accept system date eg:
> Date > is after > 28/01/2013
> or
> Date > is after > 28/01/13
> returns everything from 'wed 4/07/12   07:42PM  to current date

Anje, if you still see an issue, please file a new bug report. Thanks