119857 - There is no way to set the reading/composing directionality via the UI

Status: NEW

(MailNews Core :: Internationalization, defect)

Description

[Shoshannah Forbes]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.7+) Gecko/20020111
BuildID:    20020211108

When editing or reading Email that contains mixed Hebrew/roman text, there is no
way to set the directionality of the window, making composing plain text
messages with mixed Hebrew english text or reading such messages very difficult
if not impossible

Reproducible: Always
Steps to Reproduce:
1. start a new email messege
2. try to compose mixed text. esp. things that have () or lists (even plain text)


Actual Results:   notice that there is no way to set the window as RTL to
compose the text proprly

OE has an option to change the directionality of a message locally. something
similar would be fine

Comment 1

[Shoshannah Forbes]

Can't set it for HTMl emai either. Should i open a separate bug for that (since
it affects sending and not just local viewing)?

Comment 2

[Simon Montagu :smontagu]

Changing platform/OS to ALL/ALL.

I think that between this bug and bug 96057 and bug 98160 we have covered all
the bases, and there is no need for another bug for HTML mail. On the other
hand, HTML mail is a more serious case than normal HTML composition, because
there doesn't seem to be a way to workaround by editing the HTML source
directly, unless I am missing it.

Comment 3

[Simon Montagu :smontagu]

*** Bug 120509 has been marked as a duplicate of this bug. ***

Comment 4

[Shoshannah Forbes]

Re: bug 98160 - it is a windows only bug (mac does not have that behavier at 
all).
Any progress going to be done on this bug here, which was originaly filed for 
the mac?
It is a real pain to compose plain text email properly on the mac, which kind 
of defeates the whole thing...

Comment 5

[Shoshannah Forbes]

ok, until this bug is fixed, there is a workaround published by Mooffie, which
can be fund at: http://www.typo.co.il/~mooffie/mozilla/hebmailpack.html (the
server is not stable so you might need to try a few times before you can get thruh)

Comment 6

[Tsahi Asher]

what's the differece between this bug and bug 96057?

Comment 7

[Simon Montagu :smontagu]

This bug is about composing plain text, and bug 96057 is about composing in HTML
format.

Comment 8

[Prognathous]

Shoshannah Forbes:
> "ok, until this bug is fixed, there is a workaround published by Mooffie"

Since this link is now dead, could you please post this workaround here?

Thanks,

Prog.

Comment 9

[Prognathous]

Well, I don't know about Mooffie's workaround, but Ilya Konstantinov's 
suggestion (in Bug 96057) seem to do the trick here as well:

  Format-> Page Colors and Background-> Advanced Edit-> Attribute=Dir;Value=RTL

Prog.

Comment 10

[Shoshannah Forbes]

> Format-> Page Colors and Background-> Advanced Edit-> Attribute=Dir;Value=RTL

1) this does not work with plain text email
2) These menus are not avalible when viewing a message- only when composing

Comment 11

[Prognathous]

It seems like there's another bug that hinders the use of my workaround:
If the body of the message only includes plain text, Mozilla defaults to 
Content-Type: text/plain, instead of Content-Type: text/html - regardless of 
the settings in the Send Format prefs.

Is this a known issue? should I open a new bug? this problem appears in both 
Mozilla 1.2.1 for Windows and 1.3a for BeOS.

BTW, one can bypass this bug by using some HTML formatting *within* the body of 
the message (setting a single white space as boldface is sufficient).

Prog.

Comment 12

[Prognathous]

The bug that I described in comment #11 was recently fixed by Scott MacGregor.
More details in Bug 228720 ("Don't down-convert message to plain text if dir
attribute is set").

This makes Moofie's XPI (see comment #5) a much more effective solution, it's a
shame that only hardcore Mozilla enthusiasts know about the existence of this
addon. Is there any opposition to including it by default in Mozilla?

Prog.

Comment 13

[Adin Aronson]

Since mooffies's site seems unstable I downloaded mooffie's fix. So if anyone 
want it. send an email to adinaronson at hotmail.com

Comment 14

[Adin Aronson]

Attachment: Mooffie's xpi fix

unzip the html page and xpi to a folder. Open the webpage and click the link to
install the xpi patch.

Comment 15

[Eyal Rozenberg]

Can anyone point out why Moofie's XPI, despite containing an 'overlay' for
composition as well as for viewing, only works for viewing messages, not for
composing them? I mean, when I right-click the content-frame in the composer
window and select 'RTL body', nothing happens. What's up with that?

Comment 16

[Prognathous]

(In reply to comment #15)
> Can anyone point out why Moofie's XPI, despite containing an 'overlay' for
> composition as well as for viewing, only works for viewing messages, not for
> composing them? 

It works very well for both.

> I mean, when I right-click the content-frame in the composer
> window and select 'RTL body', nothing happens. What's up with that?

This is probably just an isolated issue with your setup, but instead of spamming
bugzilla, let's continue this discussion in mozilla.org.il forum.

Prog.

Comment 17

[Eyal Rozenberg]

Attachment: Alternative hebmailpack

To solve my problem, I replaced the 'dirAlignComposer' function code (see
inside the zip; note you have multiple unzipping to do before you actually get
to any sources...) with alternative code; it now works for me. I also added a
submenu to the view menu for the composer window, same as for the messenger
main window, with the directionality control.

TODO list (well, my opinion of what a TODO list should be): 
1. this needs a toolbar button, not just a menu setting
2. the visibility of these controls needs to be controlled by a "show
directionality controls" option in Edit|Preference|Mail&Newsgroups|Composition
(this would make the fix more attractive as a checkin into the trunk code)
3. the submenu should show a checkmark for the current directionality
4. how about installation of the XPI into a user's profile rather than the
mozilla program directory?

Comment 18

[Simon Montagu :smontagu]

*** Bug 245551 has been marked as a duplicate of this bug. ***

Comment 19

[Eyal Rozenberg]

More items for the TODO list:

5. Need to improve the direction guessing policy, which is currently not at all
perfect.
6. Need to apply the direction guessing policy for composition of 'reply'
messages, that is, if a message is displayed RTL and I press 'reply' or 'reply
all' , the new message's initial direction should be set to RTL.

Comment 20

[Tsahi Asher]

(In reply to comment #17)

> 2. the visibility of these controls needs to be controlled by a "show
> directionality controls" option in Edit|Preference|Mail&Newsgroups|Composition
> (this would make the fix more attractive as a checkin into the trunk code)

actually, IMO it should go to composer|toolbars prefs panel.

> 4. how about installation of the XPI into a user's profile rather than the
> mozilla program directory?

that's a good thing for every extension. however, things which are only relevant
for an extension shouldn't be discussed here, IMHO.

Comment 21

[Eyal Rozenberg]

Over time I've noticed most failures of Moofie's auto-detection are in cases
where latin documents use accented letters within a word. The algorithm should
decide whether the encoding is Hebrew based on words which have 0xE0-and-higher
character values in all characters of the word, rather than based on single
characters with no regard of context (e.g. single 0xE0-and-higher character
within a mostly below-0xE0 word). This would reduce the false positives at
practically no cost of false negatives.

(0xE0 is where Hebrew letters start in windows codepage 1255)

Comment 22

[Eyal Rozenberg]

Ok, the 0xE0 is meaningless, apparently moofie checks for Hebrew by the unicode
values, which means after MailNews chooses the encoding. So the false positive
LTR alignment can be fixed by in the extension, but not the wrong selection of
the encoding (I thought the extension also reset the encoding, I was wrong).

Comment 23

[Eyal Rozenberg]

Attachment: hebmailpack 1.0.2

Implemented the "look for words instead of characters" logic I suggested, plus
changed the installation script so as to be able to install to the profile
directory. Note that if the extension has already been installed to the app
directory, it will be installed there, so remove it. Version bumped to 1.0.2.

Comment 24

[Eyal Rozenberg]

Attachment: hebmailpack 1.2.1

Well, it seems Moofie himself has made some modifications since 1.0.0 in his
own typo.co.il webpage (I thought he was inactive). Anyway, this latest
attachment incroporates my changes into his most recent version. This time no
zip, just the xpi. I promise not spam this bug any more until I sort things out
with him (or is it her?).

Eyal

PS - this now has he-IL support, plus context menu for viewed as well as
composed messages

Comment 25

[Mike Cowperthwaite]

*** Bug 231277 has been marked as a duplicate of this bug. ***

Comment 26

[Mike Cowperthwaite]

*** Bug 234894 has been marked as a duplicate of this bug. ***

Comment 27

[Eyal Rozenberg]

The BiDi UI project now offers an extension which is a further development of
Moofie's fix, above, with plenty of new functionality and other goodies. See:

http://bidiui.mozdev.org/mail/

Comment 28

[Prognathous]

The functionality provided by BiDi Browser UI extension was recently merged into
FF 1.0 and SM 1.8.x. Is there anything that prevents the Mail extension from
sharing the same happy ending?

Prog.

Comment 29

[Simon Montagu :smontagu]

*** Bug 342112 has been marked as a duplicate of this bug. ***

Comment 30

[Khaled Hosny]

Is there any work done on this in the context of Thunderbird 3?

Comment 31

[Eyal Rozenberg]

Not any that I'm aware of.

Comment 33

[Eyal Rozenberg]

Comment on attachment 139888 [details]
Mooffie's xpi fix

Obsoleting here to avoid confusion.

And I'll use this opportunity to mention that I'm always willing to collaborate to integrate some of this functionality into the Mozilla codebase...

Comment 34

[Tomer Cohen :tomer]

(In reply to comment #33)
> I'll use this opportunity to mention that I'm always willing to collaborate
> to integrate some of this functionality into the Mozilla codebase...

Which parts do you suggest to omit? I'm not sure if we need the directionality auto-detection now-days, but a RTL/LTR buttons in Composer and a 'Switch page detection' command should be enough.

Comment 35

[Eyal Rozenberg]

Well, it's not about omissions, but rather about different implementations. I'll remind everyone of our discussion here:

https://www.mozdev.org/bugs/show_bug.cgi?id=20370

As for directionality auto-detection - it's a must. So much so that I've made it a different, default mode to all-LTR and all-RTL, and users rarely bother to use any of the latter two.

Comment 36

[Tomer Cohen :tomer]

Paragraph auto detection is complicated algorithm, and no other software do this. Currently more users are aware of directionality in mail messages than few years ago, and even web mail sites such as gmail do include those control buttons. 

 While it could be a nice feature, please don't forget that most Thunderbird users won't ever need it. I'd suggest adding the basic functionality to the core, and keeping all the nice features in an optional add-on.

Comment 37

[(no longer active)]

Paragraph auto-detection will come in form of bug 548206.

Comment 38

[Tomer Cohen :tomer]

(In reply to comment #37)
> Paragraph auto-detection will come in form of bug 548206.
What's the status of that code? Is it going to land in the near future? If so, maybe we should make it blocking this bug.

Comment 39

[(no longer active)]

(In reply to comment #38)
> (In reply to comment #37)
> > Paragraph auto-detection will come in form of bug 548206.
> What's the status of that code? Is it going to land in the near future? If so,
> maybe we should make it blocking this bug.

We're still working on the specification in the W3C HTML bidi group.  It should then be submitted to the WHATWG group for submission.  I don't think that anyone has started to work on the implementation yet.

Comment 40

[Dotan Cohen]

I am not sure that I understand all the comments. This bug was opened 10 years ago. Working code exists in the form of the BiDiUi extension. This extension breaks when updating Thunderbird between version numbers, requiring Arabic, Hebrew, and Persian Thunderbird users to delay upgrading to the latest stable Thunderbird. What is the argument against including the BidiUi code in core Thunderbird? Is it just "nobody's done it yet"? Can I get SVN access to do it?

Comment 41

[Dotan Cohen]

(In reply to :ehsan akhgari from comment #39)

We're still working on the specification in the W3C HTML bidi group. It
should then be submitted to the WHATWG group for submission. I don't think
that anyone has started to work on the implementation yet.

Has the specification to the HTML bidi group been submitted yet?

Comment 42

[Eyal Rozenberg]

@DotanCohen:

Answering your second-to-last comment - from 2011 (!), here are some aspects of the reason:

• The BiDi Mail UI code is an ex-post-facto hack. What you want is the layout engine to implement something - which is not what my extension does. That means that integrating the BiDi Mail UI functionality will not involve integrating the code.
• ... actually, it's more complicated than that, because there's the question of what to apply direction to. That's another heuristic; but it also involves how messages are represented and rendered into (X)HTML (e.g. do multiple P elements get created for plain-text messages with \x10 \x10 sequences?) that means messing with MIME library code. The core of that code is from the early 1990s, and nobody will touch it. It will, at some point, be replaced by a Javascript-based MIME library (maybe this is even further along than I realize).
• There's also the matter of caching the results of applying the direction-detection heuristic. That could also possibly be handled by the layout engine, but maybe it doesn't/wouldn't do that.
• Not enough Arabic, Farsi, Hebrew and Urdu speakers/emailers in the userbase to push this.
• Zero or just-about-zero Arabic, Farsi, Hebrew and Urdu speakers/emailers among the core developers to push this.
• (A bit of a rant/slightly conspiratorial) A weird mentality that has took root overtime among some people recently, supporting over-simplification and slimming-down of the application as something the users supposedly want/need/must have. It's also a bit of a chicken-and-egg situation, since the users who need some feature are driven away, then apparently no users want that feature.

Comment 43

[Dotan Cohen]

Thank you Eyal!

mentality that has took root overtime among some people recently, supporting
over-simplification and slimming-down of the application as something the
users supposedly want/need/must have

Actually, I'm all for slimming down the application to just the core components needed to fetch, read, compose, and send email. And since setting the reading and composing directional of the text is core to reading and composing email, of course it should be implemented. Implementing this feature does support the goal of slimming down Thunderbird to the core functionality only.

Comment 44

[Eyal Rozenberg]

I meant over-simplification and slimming-down of the UI.

Also - what you or I believe is in the core is not what others believe is in the core. However - it is not inconceivable that some sort of a petition to the TB council could have a stronger effect than my occasional remark on the TB discussion mailing list. A petition/letter with speakers of several RTL languages, and it would be even better if it could include a few semi-well-known people.

Just a thought.

Comment 45

[Wayne Mery (:wsmwk)]

To what extent is this bug a duplicate of 464436? In other words, what parts of this bug are not covered in bug 464436?
Also, should bug 96057 not be marked as dependency of one of these?
And are other core bugs from https://mzl.la/31atXTD needed?

Comment 46

[Eyal Rozenberg]

Bug 464436 is more specific in that it regards a specific path to addressing directionality control, and at the same time more general in that it involves more missing functionality. So, not a dupe.

About bug 96057 - it's about paragraph direction, while this is about entire-message direction. Also, that one is about composition and this one is - at least in title - about both composition and reading. I suppose that's why they haven't been marked dupes of each other.

From the above you can also gather why 96057 and 464436 are not dupes of each other.

As for the longer list - I'll have a look tomorrow or later this week.

Is there any interest in getting some work done on this?

Comment 47

[Wayne Mery (:wsmwk)]

(In reply to Eyal Rozenberg from comment #46)

Bug 464436 is more specific in that it regards a specific path to addressing directionality control, and at the same time more general in that it involves more missing functionality.
About bug 96057 - it's about paragraph direction, while this is about entire-message direction. Also, that one is about composition and this one is - at least in title - about both composition and reading. I suppose that's why they haven't been marked dupes of each other.

Thank you for clarifying - very helpful. So should this bug focus on the reading functionality, and leave the others to focus on composition?

Is there any interest in getting some work done on this?

I don't know. I am asking around to see determine the interest and what is possible. If anything happens it would obviously depend heavily on what the user and add-on community is willing to contribute - you in particular since you clearly understand the issues very well.

Comment 48

[Dotan Cohen]

So should this bug focus on the reading functionality, and leave
the others to focus on composition?

The way this bug is worded, it addresses both. But I agree that they are two separate issues. Perhaps one of the dupes that addresses either should be reopened, and this bug focus on the other.

For me personally, received emails are decypherable with the (currently) incorrect directionality, but it is very important to me to send email with the proper directionality. So I would vote to prioritize composition. However, some people would have valid reasons to emphasize a priority on reading.

Comment 49

[Eyal Rozenberg]

@Dotan:

1. I am going to make a TB-78 compatible version of BiDi Mail UI soon. So are you referring to the immediate state of affairs; to the fact that the functionality is not in the core; or to the functionality that cannot be introduced through an extension?

2. I don't think the "bug configuration" matters all that much. If someone wants to have a clean-slate description of the state of affairs, I'd think some sort of Wiki page somewhere would be a good place for that.

@Wayne:

So should this bug focus on the reading functionality, and leave the others to focus on composition?

See point (2.) of my reply to Dotan. I don't think playing with the bug titles is worth the trouble.

I am asking around to see determine the interest and what is possible.

I was hoping you would say something like "this came up in a council discussion, people said that we need to be more accommodating of other languages and cultures, we decided we should put some developer time on it" ...

If anything happens it would obviously depend heavily on what the user and add-on community is willing to contribute you in particular since you clearly understand the issues very well.

See point (1.) of my reply to Dotan and the question there.

Comment 50

[Dotan Cohen]

1. I am going to make a TB-78 compatible version of BiDi Mail UI soon. So are you referring to the
   immediate state of affairs; to the fact that the functionality is not in the core; or to the functionality
   that cannot be introduced through an extension?

All of the above, really. From the end user perspective the result is the same.

I was hoping you would say something like "this came up in a council discussion, people said
that we need to be more accommodating of other languages and cultures, we decided we should
put some developer time on it" ...

Completely disregarding Eyal and my own language Hebrew, Arabic is also RTL and it is the fifth most widely spoken language in the world. Persians, Kurds, and Urdu are also significantly large populations with RTL languages.