Closed
Bug 458450
Opened 18 years ago
Closed 18 years ago
[silme] Improve file read/write encoding experience
Categories
(Mozilla Localizations :: Infrastructure, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: zbraniecki, Assigned: adriank)
References
Details
Attachments
(1 file, 3 obsolete files)
|
8.88 KB,
patch
|
zbraniecki
:
review+
|
Details | Diff | Splinter Review |
Adrian is working on improving the way Silme is reading/writing files wrt encoding
| Reporter | ||
Comment 1•18 years ago
|
||
Adrian: any status update?
| Reporter | ||
Comment 2•18 years ago
|
||
akalla approach to fix this bug
| Assignee | ||
Updated•18 years ago
|
Attachment #343476 -
Flags: review?(gandalf)
| Reporter | ||
Comment 3•18 years ago
|
||
| Reporter | ||
Comment 4•18 years ago
|
||
adrian, can you give me combined patch against the tip pls?
| Assignee | ||
Comment 5•18 years ago
|
||
Attachment #343476 -
Attachment is obsolete: true
Attachment #344803 -
Attachment is obsolete: true
Attachment #345522 -
Flags: review?(gandalf)
Attachment #343476 -
Flags: review?(gandalf)
| Reporter | ||
Comment 6•18 years ago
|
||
good job!
Adrian: can you give me any estimations on a perf hit with this patch on loading two en-US packages (firefox 2.0 and 3.0) and diffing them with values?
Thanks
Also.
I'm wondering if it would add yet more per hit to separate another function
getSourceWithEncoding(path, encoding) that would be called from getSource() on each fallback pass. This way the code with try: codecs.open()/read() would be in one place.
What do you think?
| Assignee | ||
Comment 7•18 years ago
|
||
addressed comments from comment #6
Attachment #345522 -
Attachment is obsolete: true
Attachment #345522 -
Flags: review?(gandalf)
| Assignee | ||
Comment 8•18 years ago
|
||
Gandalf: with this patches applied I'm experiencing a slow down of about 0.5%
| Assignee | ||
Updated•18 years ago
|
Attachment #345620 -
Flags: review?(gandalf)
| Reporter | ||
Comment 9•18 years ago
|
||
Comment on attachment 345620 [details] [diff] [review]
patch v2
great patch!
I like the approach and it gives an author a lot of flexibility. Checkin in
Attachment #345620 -
Flags: review?(gandalf) → review+
| Reporter | ||
Comment 10•18 years ago
|
||
I'm not closing this bug, because I believe we can try to reduce the performance hit.
In my testcase, of 15 packages loaded and compared (as entityList), the hit is like this:
without the patch:
gandalf:silme zbraniecki$ time ./scripts/test.py
real 0m5.375s (to 0m5.420s)
user 0m4.864s
sys 0m0.454s
with the patch:
gandalf:silme zbraniecki$ time ./scripts/test.py
real 0m5.486s (to 0m5.520s)
user 0m4.970s
sys 0m0.449s
I identified several potential improvements already and I'm working on them
| Reporter | ||
Comment 11•18 years ago
|
||
So the two things I'm working on right now are:
1) Do not call toUnicode if the encoding is 'utf_8'. Saves 20% of what we lost with this patch
2) Do not use 'rU' in opening a file. Saves 80% of the hit.
Surprisingly I don't see any difference with putting the code inline vs. leaving it in a separate function. Maybe it doesn't do a difference because they're in one file and in result it's compiled (to pyc).
Adrian: can you tell me what you think about removing this 'rU'?
Comment 12•18 years ago
|
||
Universal line ending support (U) is pretty important. We have lots of files with mixed line endings, and you need to pay the price for that somewhere. Either, you do that on reading the file, or you do it while parsing. I found 'rU' to be the mentally cheap way, I didn't do perf tests on that though.
| Assignee | ||
Comment 13•18 years ago
|
||
'rU' is the easiest way to not run into trouble with universal line ending, which is used in many Mozilla l10n files.
If not using 'rU' then, like Axel said, we would need to solve this in the parsers.
| Reporter | ||
Updated•18 years ago
|
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•