Closed Bug 458450 Opened 18 years ago Closed 18 years ago

[silme] Improve file read/write encoding experience

Categories

(Mozilla Localizations :: Infrastructure, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: zbraniecki, Assigned: adriank)

References

Details

Attachments

(1 file, 3 obsolete files)

Adrian is working on improving the way Silme is reading/writing files wrt encoding
Adrian: any status update?
Attached patch encoding fix by akalla (obsolete) — — Splinter Review
akalla approach to fix this bug
Attachment #343476 - Flags: review?(gandalf)
adrian, can you give me combined patch against the tip pls?
Attached patch both encoding patches merged (obsolete) — — Splinter Review
Attachment #343476 - Attachment is obsolete: true
Attachment #344803 - Attachment is obsolete: true
Attachment #345522 - Flags: review?(gandalf)
Attachment #343476 - Flags: review?(gandalf)
good job! Adrian: can you give me any estimations on a perf hit with this patch on loading two en-US packages (firefox 2.0 and 3.0) and diffing them with values? Thanks Also. I'm wondering if it would add yet more per hit to separate another function getSourceWithEncoding(path, encoding) that would be called from getSource() on each fallback pass. This way the code with try: codecs.open()/read() would be in one place. What do you think?
Attached patch patch v2 — — Splinter Review
addressed comments from comment #6
Attachment #345522 - Attachment is obsolete: true
Attachment #345522 - Flags: review?(gandalf)
Gandalf: with this patches applied I'm experiencing a slow down of about 0.5%
Attachment #345620 - Flags: review?(gandalf)
Comment on attachment 345620 [details] [diff] [review] patch v2 great patch! I like the approach and it gives an author a lot of flexibility. Checkin in
Attachment #345620 - Flags: review?(gandalf) → review+
I'm not closing this bug, because I believe we can try to reduce the performance hit. In my testcase, of 15 packages loaded and compared (as entityList), the hit is like this: without the patch: gandalf:silme zbraniecki$ time ./scripts/test.py real 0m5.375s (to 0m5.420s) user 0m4.864s sys 0m0.454s with the patch: gandalf:silme zbraniecki$ time ./scripts/test.py real 0m5.486s (to 0m5.520s) user 0m4.970s sys 0m0.449s I identified several potential improvements already and I'm working on them
So the two things I'm working on right now are: 1) Do not call toUnicode if the encoding is 'utf_8'. Saves 20% of what we lost with this patch 2) Do not use 'rU' in opening a file. Saves 80% of the hit. Surprisingly I don't see any difference with putting the code inline vs. leaving it in a separate function. Maybe it doesn't do a difference because they're in one file and in result it's compiled (to pyc). Adrian: can you tell me what you think about removing this 'rU'?
Universal line ending support (U) is pretty important. We have lots of files with mixed line endings, and you need to pay the price for that somewhere. Either, you do that on reading the file, or you do it while parsing. I found 'rU' to be the mentally cheap way, I didn't do perf tests on that though.
'rU' is the easiest way to not run into trouble with universal line ending, which is used in many Mozilla l10n files. If not using 'rU' then, like Axel said, we would need to solve this in the parsers.
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Blocks: 478225
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: