285236 - [Devmo] build incorrectly when source file include BOM

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: General, defect)

Description

[victory <[e-mail]>]

now, Template-Toolkit brakes French source file that includes BOM. 
(created files seems Windows-1252 encoding.)
(now this happens only French but will also happen on other ASCII-alike languages.)

to avoid this, run the script that removes BOM before build, and Devmo tree will
be build correctly.
# this requires a directory that stores 'completely BOM-less' source tree.
# need to modify dest directive in ttree.cfg and need a replica of ttree.cfg for
this script.

Comment 1

[victory <[e-mail]>]

Attachment: diff from ttree

(line number may be different on another environment.)

Comment 2

[fantasai]

I can't tell what file this thing is patching.

Also, the idea of creating an intermediary tree just to remove the BOM seems
like an expensive hack to me.

Comment 3

[victory <[e-mail]>]

Attachment: replacement of 'ttree -f etc/ttree.cfg'

this script does below:
 generate listfle
 makes temporaly file, build, remove temporaly file
 remove temporaly directory
 remove listfile

Comment 4

[victory <[e-mail]>]

Attachment: fixed version of wrong handling on 176870

sorry, 176870 has wrong handling when processing top directory of source.

Comment 5

[victory <[e-mail]>]

Attachment: add check whether updated file includes BOM

don't create temp file if updated files has NO BOM.
1. generate listfle
2. ttree(same as default)
3. check update files whether file has BOM or not
4. if needed (makes temporaly file, build, remove temporaly file)
5. remove temporaly directory(if created above)
6. remove listfile

Comment 6

[Myk Melez [:myk] [@mykmelez]]

Reporter, could you attach a file that has a BOM so that we can test out the
patch and troubleshoot the Template Toolkit?

Comment 7

[fantasai]

Attachment: sample UTF8 file with BOM

The source file is UTF-8 with a BOM. After processing, Firefox auto-detects the
destination file as Windows 1252. Overriding it to UTF-8 (which is what happens
when we send it out with a UTF-8 charset header) causes the accented characters
to not show up correctly.

Comment 8

[fantasai]

Attachment: sample UTF8 file with BOM

upload the right file...

Comment 9

[fantasai]

Attachment: same UTF8 file without BOM

Comment 10

[Jeff Walden [:Waldo]]

Changing QA contact to the default QA so interested parties can watch bugs filed
before the default QA was changed.  Only bugs in an unfixed state have been
changed; fixed bugs are unlikely to be reopened, and more bugspam is always a
bad thing.  Sorry for the bugspam - filter it into your Junk folder by searching
for 'WhyDoesMediaWikiHaveToBeEvilAndNotAcceptArbitraryCharactersInTitlesAndLinks'.

Comment 11

[victory <[e-mail]>]

devmo has moved to wiki so this bug should be closed