Closed Bug 352842 Opened 18 years ago Closed 18 years ago

Bad import of UTF-8 ICS file with non-ASCII characters

Categories

(Calendar :: Import and Export, defect)

Product:
Calendar
Component:
Import and Export
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: mattwillis, Unassigned)

References

Details

(Keywords: dataloss, regression, Whiteboard: [patch in hand])

Attachments

(3 files)

ICS from iCal.app
3.77 KB, text/plain
Details
ICS exported from Sunbird after importing
5.18 KB, text/plain
Details
use octet arrays
2.94 KB, patch
mattwillis
: first-review+
dmosedale
: second-review+
Details | Diff | Splinter Review 
When importing iCal.app's Birthday's calendar, it uses a right curved single quote (’) in the SUMMARY (ex: "SUMMARY:Matthew Willis’s Birthday")

When this file (which opens properly in BBEdit as UTF-8, no BOM) is imported into Sunbird, the quote becomes mangled, and is displayed as "AcAA" where the c is a cents sign, the first A has a tilde above it and the other two have carats above them.

Something's wacky here.
Attached file ICS from iCal.app — 

    
    

    
  
The character in question, in HexEdit, from iCal's ICS:
E2 80 99

This matches the info at 
http://www.fileformat.info/info/unicode/char/2019/index.htm

The character in question, in HexEdit, from Sunbird's ICS:
C3 83 C2 A2 C3 82 C2 80 C3 82 C2 99 

We're adding C3 83 C2 to the first, and C3 82 C2 to the second and third, and th e first byte is becoming A2 rather than E2.

Flags: blocking0.3+
Keywords: dataloss

    
    

    
  
(In reply to comment #3)
> The character in question, in HexEdit, from iCal's ICS:
> E2 80 99
> The character in question, in HexEdit, from Sunbird's ICS:
> C3 83 C2 A2 C3 82 C2 80 C3 82 C2 99 

These bytes are getting misinterpreted as ISO-8859-1 and double-converted to UTF-8, not once but twice

ISO-8859-1              UTF-8
----------              -----
E2 80 99          -->   C3 A2 C2 80 C2 99
C3 A2 C2 80 C2 99 -->   C3 82 C2 A2 C3 82 C2 80 C3 82 C3 99

    
    

    
  

      
      
      
      

        Attached patch
          
          
        use octet arrays
        — 
          — Splinter Review
      

    


  
Somehow, using a string as array of didn't really work. This patch makes the importer really only use octet arrays until it has been decoded from utf8 into a string.

        Attachment #238908 -
        Flags: second-review?(dmose)

        Attachment #238908 -
        Flags: first-review?(mattwillis)
Whiteboard: [needs review dmose]

    
    

    
  
Comment on attachment 238908 [details] [diff] [review]
use octet arrays

This fixed it on my birthdays calendar with the smart quotes.

r=lilmatt

        Attachment #238908 -
        Flags: first-review?(mattwillis) → first-review+

    
    

    
  
regression from bug 315672
Keywords: regression
Whiteboard: [needs review dmose] → [patch in hand][needs review dmose]

    
    

    
  
*** Bug 352964 has been marked as a duplicate of this bug. ***

    
    

    
  
Updating summary to be more general
Blocks: UTF-import
Summary: Bad import of UTF-8 ICS file with smart-quotes from iCal.app → Bad import of UTF-8 ICS file with non-ASCII characters

    
    

    
  
Comment on attachment 238908 [details] [diff] [review]
use octet arrays

r=dmose

        Attachment #238908 -
        Flags: second-review?(dmose) → second-review+
Whiteboard: [patch in hand][needs review dmose] → [patch in hand][needs checkin]

    
    

    
  
patch checked in
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED

    
    

    
  
verified with
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060921 Calendar/0.3a2+
Status: RESOLVED → VERIFIED

    
  
Whiteboard: [patch in hand][needs checkin] → [patch in hand][litmus testcase wanted]

    
    

    
  
Litmus testcase 2693 created
Whiteboard: [patch in hand][litmus testcase wanted] → [patch in hand]

          You need to log in
          before you can comment on or make changes to this bug.