Last Comment Bug 467920 - Email sending doesn't work if you save a template with DOS newlines
: Email sending doesn't work if you save a template with DOS newlines
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: Email Notifications (show other bugs)
: 3.2
: All All
: -- major with 2 votes (vote)
: Bugzilla 3.2
Assigned To: Max Kanat-Alexander
: default-qa
Mentors:
: 472788 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-12-04 07:32 PST by Dan
Modified: 2013-11-29 16:48 PST (History)
12 users (show)
mkanat: approval+
mkanat: approval3.2+
LpSolit: blocking3.2.3+
LpSolit: blocking3.2.1-
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Debug email log from 3.0.6 showing DOS and UNIX formatted templates work. (82.94 KB, text/plain)
2008-12-05 06:35 PST, Dan
no flags Details
Debug email log from 3.2 showing DOS formatted templates don't work anymore. (96.32 KB, text/plain)
2008-12-05 06:38 PST, Dan
no flags Details
DOS and UNIX UTF8 Formatted Templates to test with. (2.47 KB, application/x-zip-compressed)
2008-12-05 08:11 PST, Dan
no flags Details
Message sample (1.83 KB, text/plain)
2008-12-05 21:07 PST, Vitaly Fedrushkov
no flags Details
What Email::MIME gets and how it parses it (2.18 KB, application/octet-stream)
2009-02-06 10:35 PST, Max Kanat-Alexander
no flags Details
v1 (733 bytes, patch)
2009-02-16 19:10 PST, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v2 (733 bytes, patch)
2009-02-16 19:15 PST, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
Actual v2 (1022 bytes, patch)
2009-02-16 19:22 PST, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v3 (1.13 KB, patch)
2009-02-16 19:40 PST, Max Kanat-Alexander
glob: review+
Details | Diff | Splinter Review

Description Dan 2008-12-04 07:32:42 PST
User-Agent:       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.2)
Build Identifier: 

UTF-8 DOS formatted email templates nolonger work as of the 3.2 release.  

Reproducible: Always

Steps to Reproduce:
1) Use the CVS install method with the default CVS client settings for Windows or download the tar.gz package and use WinZip with the default Winzip settings to extract Bugzilla then all non-binary files will be have the CTRL-LF converted to UTF8 DOS format.  The default settings for both CVS and Winzip are to convert text files to DOS format for readability reasons, you must disable this functionality by default.

2) I've not tested a clean install at this time, only an upgrade from 2.22.4 and 3.0.6.  So upgrade to 3.2 from one of these releases.  I belive you will see the issue on a clean install as well.

3) Setup email to use SMTP.

4) Create or update an existing bug to cause Bugzilla to generate an outbound email message.



Actual Results:  
The email message that will be sent out will have corrupted email headers.

Expected Results:  
The email message shouldn't have corrupt email headers.

Here is what the email headers look like before the upgrade to 3.2...

Date: Sun, 23 Nov 2008 22:59:31 -0600
Message-ID: <200811240459.mAO4xVFU483646@fannin.mydomain.org>
Subject: [Bug 1345] test
X-Bugzilla-Product: TestProduct
In-Reply-To: <bug-1345-20-8080@http.cricket.mydomain.org/>
X-Mailer: Mail::Mailer[v2.04] Net::SMTP[v2.30]
X-Bugzilla-Severity: 4-normal
X-Bugzilla-Keywords:
X-Bugzilla-Reason: CC
X-Bugzilla-Component: TestComponent
To: <dmclaughlin@mydomain.org>
From: <TRF_CRISTechSupport@mydomain.org>
X-pstn-neptune: 0/0/0.00/0
X-pstn-levels: (S:10.86326/99.90000 CV:99.9999 R:95.9108 P:95.9108 M:97.0282 
C:98.6951 )
X-pstn-settings: 3 (1.0000:1.0000) s cv gt3 gt2 gt1 r p m c
X-pstn-addresses: from <TRF_CRISTechSupport@mydomain.org> forward (user 
good) [398/15]
MIME-Version: 1.0
Content-Type: text/plain
Return-Path: TRF_CRISTechSupport@mydomain.org

After the upgrade to the 3.2 release, several of the email headers are 
jammed in the From: mail header.plus a bunch of other garbage.

Date: Sun, 23 Nov 2008 23:11:13 -0600
From:
                <"TRF_CRISTechSupport@mydomain.org 
To:TRF_CRISTechSupport@mydomain.org.To:dmclaughlin"@mydomain.org.Subject:[Bug.3420]>,
                <"New:test.X-Bugzilla-Reason:Reporter.X-Bugzilla-Type:newchanged.X-Bugzilla-Watch-Reason:None.Auto-Submitted:auto-generated.Con"@mydomain.org>,
                <23@mydomain.org>, <Nov@mydomain.org>, <2008@mydomain.org>,
                <"23:23.Nov.2008.23:11:13.-0600"@mydomain.org>
X-Bugzilla-Product: CRIS
X-Bugzilla-Component: Common
X-Bugzilla-Keywords:
X-Bugzilla-Severity: 5-minor
X-Bugzilla-Who: dmclaughlin@mydomain.org
X-Bugzilla-Status: NEW
X-Bugzilla-Priority: P5
X-Bugzilla-Assigned-To: TRF_CRISTechSupport@mydomain.org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Changed-Fields:
Message-ID: <bug-3420-20-8180@http.cricket.mydomain.org/>
To: undisclosed-recipients:;
X-pstn-neptune: 0/0/0.00/0
X-pstn-levels: (S: 0.00889/97.87704 CV:99.9999 R:95.9108 P:95.9108 M:97.0282 
C:98.6951 )
X-pstn-settings: 3 (1.0000:1.0000) s cv gt3 gt2 gt1 r p m c
X-pstn-addresses: from <TRF_CRISTechSupport@mydomain.org 
To:TRF_CRISTechSupport@mydomain.org.To:dmclaughlin> [398/15]
X-pstn-disposition: quarantine
MIME-Version: 1.0
Content-Type: text/plain
Return-Path: TRF_CRISTechSupport@mydomain.org


The only bug I could find that looked remotely similar was this: 
https://bugzilla.mozilla.org/attachment.cgi?id=335914&action=diff&headers=1&context=file#mozilla/webtools/bugzilla/Bugzilla/Mailer.pm_sec1

Here are the versions of everything we are running.



httpd.exe -V
Server version: Apache/2.2.10 (Win32)
Server built:   Oct 10 2008 12:39:04
Server's Module Magic Number: 20051115:18
Server loaded:  APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture:   32-bit
Server MPM:     WinNT
  threaded:     yes (fixed thread count)
    forked:     no
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/winnt"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/apache"
 -D SUEXEC_BIN="/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error.log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
mysql -V
mysql  Ver 14.14 Distrib 5.1.30, for Win32 (ia32)

perl -V
Set up gcc environment - 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags 
='-DNDEBUG -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE 
 -DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO 
 -DPERL_MSVCRT_READFI
 -fno-strict-aliasing -mms-bitfields',
    optimize='-O2',
    cppflags='-DWIN32'
    ccversion='', gccversion='3.4.4 (cygming special, gdc 0.12, using dmd 
0.125)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', 
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags ='-L"D:\Perl\lib\CORE"'
    libpth=\lib
    libs=-lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 
 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 
 -lodbccp32 -lmsvcrt
    perllibs=-lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 
 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 
 -lodbccp32 -lmsvcrt
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-mdll -L"D:\Perl\lib\CORE"'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
                        PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 1004 [287188]
        33741 avoids segfaults invoking S_raise_signal() (on Linux)
        33763 Win32 process ids can have more than 16 bits
        32809 Load 'loadable object' with non-default file extension
        32728 64-bit fix for Time::Local
  Built under MSWin32
  Compiled at Sep  3 2008 13:16:37
  @INC:
    D:/Perl/site/lib
    D:/Perl/lib

checksetup.pl
* This is Bugzilla 3.2 on perl 5.10.0
* Running on WinXP/.Net Build 2600 (Service Pack 2)

Checking perl modules...
Checking for              CGI.pm (v3.33)   ok: found v3.41
Checking for            TimeDate (v2.21)   ok: found v2.22
Checking for           PathTools (v0.84)   ok: found v3.2501
Checking for                 DBI (v1.41)   ok: found v1.607
Checking for    Template-Toolkit (v2.15)   ok: found v2.20
Checking for          Email-Send (v2.16)   ok: found v2.192
Checking for          Email-MIME (v1.861)  ok: found v1.861
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.442

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for          DBD-Oracle (v1.19)   ok: found v1.21

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.41
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.32
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v5.814
Checking for         PatchReader (v0.9.4)  ok: found v0.9.5
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)     ok: found v0.39
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (any)     ok: found v0.69
Checking for         HTML-Parser (v3.40)   ok: found v3.56
Checking for       HTML-Scrubber (any)     ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any)     ok: found v1.315
Checking for         Email-Reply (any)     ok: found v1.202
Checking for            mod_perl (v1.999022)  not found
Comment 1 Dan 2008-12-04 07:36:59 PST
In all previous releases the templates could be either DOS or UNIX formatted 
and they would work.  But in the 3.2 release only a UNIX formatted 
newchangedmail.txt.tmpl will work.  There looks to be a lot of code in 3.2 
that has changed related to email encoding, so I'm assuming that has 
something to do with why DOS formatted templates won't work anymore.

For now the workarounds are:

1) Use the CVS method for installing, but only if you configure the CVS client to use UNIX formatting for txt files.
2) Use Cygwin tar to decompress the tar.gz.
3) Use Winzip to decompress the tar.gz, but only if you have the Smart CR/LF 
functionality turned off.
4) Use a thirdparty utility sed, vi/vim or Windows text editor such as 
UltraEdit to convert the templates so they unix formatted.
Comment 2 Max Kanat-Alexander 2008-12-05 00:08:20 PST
DOS newlines are not just \r-- I have no idea how you got such a file. Any line that ends with just \r is clearly not a valid email header, which makes it tempting to mark this INVALID.
Comment 3 Dan 2008-12-05 06:35:38 PST
Created attachment 351547 [details]
Debug email log from 3.0.6 showing DOS and UNIX formatted templates work.

This morning I downloaded a new 3.0.6 tar.gz and ran a test using a DOS then a UNIX formatted template.  There are no problems in the 3.0.6 release.
Comment 4 Dan 2008-12-05 06:38:49 PST
Created attachment 351548 [details]
Debug email log from 3.2 showing DOS formatted templates don't work anymore.

I downloaded a new 3.2 tar.gz this morning and in the 3.2 release only the UNIX formatted template works.
Comment 5 Dan 2008-12-05 06:43:01 PST
This morning I tested 3.2 on our origial Windows 2003 server, my Windows XP workstation and on Windows 2003 VM.  I was able to reproduce the issue in all three environments by simply converting the template back and forth from DOS to UNIX formatting.  Doing the same tests with the 3.0.6 release worked in every case.
Comment 6 Dan 2008-12-05 07:00:37 PST
If you look at the logs I attached you will see that on 3.0.6 the From header with both a DOS and UNIX formatted template always has the comma after the \r. 

Here is the DOS formatted on 3.0.6:

[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> From: TRF_TechSupport@mydomain.org\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345
[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> To: kmorgan@mydomain.org\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345
[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> Subject: [Bug 1345] test\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345
[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> X-Bugzilla-Reason: Reporter\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345
[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> X-Bugzilla-Type: newchanged\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345
[Fri Dec 05 08:20:29 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae378c)>>> X-Bugzilla-Watch-Reason: None\r, referer: http://cricket.mydomain.org:8180/show_bug.cgi?id=1345

Here is the UNIX formatted on 3.0.6:

[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> From: TRF_TechSupport@mydomain.org, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> To: kmorgan@mydomain.org, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> Subject: [Bug 1345] test, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> X-Bugzilla-Reason: Reporter, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> X-Bugzilla-Type: newchanged, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 08:22:00 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2ae55f4)>>> X-Bugzilla-Watch-Reason: None, referer: http://cricket.mydomain.org:8180/process_bug.cgi

When you look at the logs from the 3.2 release you will see that if the template is DOS formatted the comma after the \r is gone and all the headers are jammed together.

Here is the DOS formatted on 3.2:

[Fri Dec 05 07:33:41 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x2f3c4bc)>>> From: TRF_CRISTechSupport@mydomain.org\rTo: kmorgan@mydomain.org\rSubject: [Bug 1345] test\rX-Bugzilla-Reason: Reporter\rX-Bugzilla-Type: newchanged\rX-Bugzilla-Watch-Reason: None\rAuto-Submitted: auto-generated\rContent-Type: text/plain; charset="UTF-8"\rMIME-Version: 1.0\rDate: Fri,  5 Dec 2008 07:33:37 -0600\r\r, referer: http://cricket.mydomain.org:8180/process_bug.cgi

Here is the UNIX formatted on 3.2:

[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> From: TRF_CRISTechSupport@mydomain.org, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> To: kmorgan@mydomain.org, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> Subject: [Bug 1345] test, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> X-Bugzilla-Reason: Reporter, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> X-Bugzilla-Type: newchanged, referer: http://cricket.mydomain.org:8180/process_bug.cgi
[Fri Dec 05 07:35:59 2008] [error] [client 192.168.2.2] Net::SMTP=GLOB(0x20852f0)>>> X-Bugzilla-Watch-Reason: None, referer: http://cricket.mydomain.org:8180/process_bug.cgi

If I can be of any further assistance please don't hesitate to ask.  We have had a death in the family and I will be away for the next few days, so if I don't respond that is why.
Comment 7 Marc Schumann [:Wurblzap] 2008-12-05 07:04:26 PST
Hmmm... I see you're using Cygwin. Cygwin does some magic on its own when it
comes to line endings. I wouldn't be too surprised if this is what's behind it.
I'm using ActiveState Perl on Windows with a Bugzilla configured to use SMTP,
and I don't experience such a problem.

Please continue handling this at @support at your initial posting, as I believe
this is not an issue of Bugzilla itself (which would make this bug INVALID, as
Max suggested).
Comment 8 Marc Schumann [:Wurblzap] 2008-12-05 07:08:31 PST
Oh, and the comma is probably being added by the program doing the logging, I'd say.
Comment 9 Dan 2008-12-05 07:28:41 PST
Yes. I'm sure the \r is logging related because the actual mail headers as seen in my earlier postings to this ticket don't show the \r, only that the headers are all jammed together.  Sorry for the confusion, I wasn't trying to state that the \r has anything to do with it, but was only trying to describe where in the log file to see the issue in the logs.
Comment 10 Frédéric Buclin 2008-12-05 07:33:52 PST
Reopen if you can reproduce the problem using ActivePerl.
Comment 11 Dan 2008-12-05 07:56:58 PST
I am using Active Perl, seem my posting above.  I think there was an assumption that I was using Cygwin because I stated that you could use Cygwin tar to decompress the tar.gz to avoid having the templates converted to DOS format.  I've reproduced this issue on three completely seperate environments and only one of them has Cygwin installed.  Infact I just now uninstalled Cygwin from my XP workstation, deleted the Cywin directory and rebooted.  I'm still seeing the exact same issue.
Comment 12 Dan 2008-12-05 08:11:02 PST
Created attachment 351552 [details]
DOS and UNIX UTF8 Formatted Templates to test with.

Here are some test templates for you to use to reproducte the issue.  Switch them out with the default template in 3.2 (specifically the DOS formatted tmpl) and you should see the same issue.
Comment 13 Dan 2008-12-05 08:18:49 PST
Without access to vim, dos2unix, or sed on Windows....I've found that UltraEdit or jEdit are useful for converting files between different formats and encodings.
Comment 14 Marc Schumann [:Wurblzap] 2008-12-05 08:26:11 PST
The assumption that you're using Cygwin stems from here:

(In reply to comment #0)
> perl -V
> Set up gcc environment - 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)

So if you're running checksetup.pl, I assume it uses Cygwin, mangling templates
on precompilation. This *will* hurt even if your Apache and registry are set up
so that ActiveState Perl is used for CGI.

If in doubt, uninstall Perl from Cygwin.
http://www.bugzilla.org/docs/win32install.html as well as
http://www.bugzilla.org/docs/tip/en/html/os-specific.html ask for ActiveState
Perl.
Comment 15 Dan 2008-12-05 08:46:53 PST
The comment you are referring only means that Active State is using gcc from Cygwin.

Regardless, I only had Cygwin on one of the environments and I just uninstalled it and deleted the directory and rebooted and the problem doesn't go away.  To confirm, on the one system that had Cygwin that Cygwin has nothing to do with it I've deleted the entire bugzilla directory, reextracted the tar.gz, and restored the database to 2.22.4.  I will post my findings shortly.

D:\WINDOWS>perl -V
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DUSE_SITEC
USTOMIZE -DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX',
    optimize='-MD -Zi -DNDEBUG -O1',
    cppflags='-DWIN32'
    ccversion='12.00.8804', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf  -libpath:"D:\Perl\lib\CORE"  -machine:x86'
    libpth=\lib
    libs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    perllibs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.
lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf  -libpath:"D:\Perl\lib\CORE"  -machine:x86
'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
                        PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 1004 [287188]
        33741 avoids segfaults invoking S_raise_signal() (on Linux)
        33763 Win32 process ids can have more than 16 bits
        32809 Load 'loadable object' with non-default file extension
        32728 64-bit fix for Time::Local
  Built under MSWin32
  Compiled at Sep  3 2008 13:16:37
  @INC:
    D:/Perl/site/lib
    D:/Perl/lib
Comment 16 Dan 2008-12-05 09:10:27 PST
I restored my 2.22.4 database, extracted the 3.2 tar.gz using Winzip with default settings (this will convert text files to DOS format, but using the CVS install method will also convert text files to DOS format by default), confirmed the perl version, ran checksetup.pl.  The problem is still there.  All I do to fix it is open the the template in UltraEdit and from the File option choose Conversions->DOS to UNIX, save it, and problem is gone.  If I change the template back by choosing Conversions->UNIX to DOS then the problem is back.  BTW... I checked, and the newchangedmail.txt.tmpl is in the tar file and it is not modified by the checksetup.pl because the checksums before and after running checksetup.pl are identical.


D:\servers\bugzilla\bugzilla-3.2>perl -V
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DUSE_SITEC
USTOMIZE -DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX',
    optimize='-MD -Zi -DNDEBUG -O1',
    cppflags='-DWIN32'
    ccversion='12.00.8804', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf  -libpath:"D:\Perl\lib\CORE"  -machine:x86'
    libpth=\lib
    libs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    perllibs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.
lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf  -libpath:"D:\Perl\lib\CORE"  -machine:x86
'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
                        PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 1004 [287188]
        33741 avoids segfaults invoking S_raise_signal() (on Linux)
        33763 Win32 process ids can have more than 16 bits
        32809 Load 'loadable object' with non-default file extension
        32728 64-bit fix for Time::Local
  Built under MSWin32
  Compiled at Sep  3 2008 13:16:37
  @INC:
    D:/Perl/site/lib
    D:/Perl/lib
    .

D:\servers\bugzilla\bugzilla-3.2>perl checksetup.pl
* This is Bugzilla 3.2 on perl 5.10.0
* Running on WinXP/.Net Build 2600 (Service Pack 2)

Checking perl modules...
Checking for              CGI.pm (v3.33)   ok: found v3.41
Checking for            TimeDate (v2.21)   ok: found v2.22
Checking for           PathTools (v0.84)   ok: found v3.2501
Checking for                 DBI (v1.41)   ok: found v1.607
Checking for    Template-Toolkit (v2.15)   ok: found v2.20
Checking for          Email-Send (v2.16)   ok: found v2.192
Checking for          Email-MIME (v1.861)  ok: found v1.861
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.442

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for          DBD-Oracle (v1.19)   ok: found v1.21

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.41
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.32
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v5.814
Checking for         PatchReader (v0.9.4)  ok: found v0.9.5
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)     ok: found v0.39
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (any)     ok: found v0.69
Checking for         HTML-Parser (v3.40)   ok: found v3.56
Checking for       HTML-Scrubber (any)     ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any)     ok: found v1.315
Checking for         Email-Reply (any)     ok: found v1.202
Checking for            mod_perl (v1.999022)  not found

* NOTE: You must run any commands listed below as Administrator.

***********************************************************************
* Note For Windows Users                                              *
***********************************************************************
* In order to install the modules listed below, you first have to run *
* the following command as an Administrator:                          *
*                                                                     *
*   ppm repo add theory58S http://cpan.uwinnipeg.ca/PPMPackages/10xx/
*                                                                     *
***********************************************************************
**********************************************************************
* OPTIONAL MODULES                                                   *
**********************************************************************
* Certain Perl modules are not required by Bugzilla, but by          *
* installing the latest version you gain access to additional        *
* features.                                                          *
*                                                                    *
* The optional modules you do not have installed are listed below,   *
* with the name of the feature they enable. If you want to install   *
* one of these modules, just run the appropriate command in the      *
* "COMMANDS TO INSTALL" section.                                     *
**********************************************************************

***********************************************************************
* MODULE NAME * ENABLES FEATURE(S)                                    *
***********************************************************************
*  PerlMagick * Optionally Convert BMP Attachments to PNGs            *
* Authen-SASL * SMTP Authentication                                   *
*  RadiusPerl * RADIUS Authentication                                 *
*    mod_perl * mod_perl                                              *
***********************************************************************
COMMANDS TO INSTALL:

     PerlMagick: ppm install PerlMagick
    Authen-SASL: ppm install Authen-SASL
     RadiusPerl: ppm install RadiusPerl
       mod_perl: ppm install mod_perl
Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

    http://cyberelk.net/tim/patchutils/


This version of Bugzilla contains some variables that you may want to
change and adapt to your local settings. Please edit the file
./localconfig and rerun checksetup.pl.

The following variables are new to ./localconfig since you last ran
checksetup.pl:  create_htaccess, webservergroup, db_driver, db_host, db_name, db_user, db_pass, db_port, db_sock, db_che
ck, index_html, cvsbin, interdiffbin, diffpath


D:\servers\bugzilla\bugzilla-3.2>notepad localconfig

D:\servers\bugzilla\bugzilla-3.2>perl checksetup.pl
* This is Bugzilla 3.2 on perl 5.10.0
* Running on WinXP/.Net Build 2600 (Service Pack 2)

Checking perl modules...
Checking for              CGI.pm (v3.33)   ok: found v3.41
Checking for            TimeDate (v2.21)   ok: found v2.22
Checking for           PathTools (v0.84)   ok: found v3.2501
Checking for                 DBI (v1.41)   ok: found v1.607
Checking for    Template-Toolkit (v2.15)   ok: found v2.20
Checking for          Email-Send (v2.16)   ok: found v2.192
Checking for          Email-MIME (v1.861)  ok: found v1.861
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.442

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for          DBD-Oracle (v1.19)   ok: found v1.21

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.41
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.32
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v5.814
Checking for         PatchReader (v0.9.4)  ok: found v0.9.5
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)     ok: found v0.39
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (any)     ok: found v0.69
Checking for         HTML-Parser (v3.40)   ok: found v3.56
Checking for       HTML-Scrubber (any)     ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any)     ok: found v1.315
Checking for         Email-Reply (any)     ok: found v1.202
Checking for            mod_perl (v1.999022)  not found

* NOTE: You must run any commands listed below as Administrator.

***********************************************************************
* Note For Windows Users                                              *
***********************************************************************
* In order to install the modules listed below, you first have to run *
* the following command as an Administrator:                          *
*                                                                     *
*   ppm repo add theory58S http://cpan.uwinnipeg.ca/PPMPackages/10xx/
*                                                                     *
***********************************************************************
**********************************************************************
* OPTIONAL MODULES                                                   *
**********************************************************************
* Certain Perl modules are not required by Bugzilla, but by          *
* installing the latest version you gain access to additional        *
* features.                                                          *
*                                                                    *
* The optional modules you do not have installed are listed below,   *
* with the name of the feature they enable. If you want to install   *
* one of these modules, just run the appropriate command in the      *
* "COMMANDS TO INSTALL" section.                                     *
**********************************************************************

***********************************************************************
* MODULE NAME * ENABLES FEATURE(S)                                    *
***********************************************************************
*  PerlMagick * Optionally Convert BMP Attachments to PNGs            *
* Authen-SASL * SMTP Authentication                                   *
*  RadiusPerl * RADIUS Authentication                                 *
*    mod_perl * mod_perl                                              *
***********************************************************************
COMMANDS TO INSTALL:

     PerlMagick: ppm install PerlMagick
    Authen-SASL: ppm install Authen-SASL
     RadiusPerl: ppm install RadiusPerl
       mod_perl: ppm install mod_perl
Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

    http://cyberelk.net/tim/patchutils/

Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for           MySQL (v4.1.2)  ok: found v5.1.30-community

Creating database bugs...
The 'bugs' database could not be created. The error returned was:

    Access denied for user 'bugs'@'%' to database 'bugs'

This might have several reasons:

* MySQL is not running.
* MySQL is running, but there is a problem either in the
  server configuration or the database access rights. Read the Bugzilla
  Guide in the doc directory. The section about database configuration
  should help.
* Your password for the 'bugs' user, specified in $db_pass, is
  incorrect, in './localconfig'.
* There is a subtle problem with Perl, DBI, or MySQL. Make
  sure all settings in './localconfig' are correct. If all else fails, set
  '$db_check' to 0.


D:\servers\bugzilla\bugzilla-3.2>perl checksetup.pl
* This is Bugzilla 3.2 on perl 5.10.0
* Running on WinXP/.Net Build 2600 (Service Pack 2)

Checking perl modules...
Checking for              CGI.pm (v3.33)   ok: found v3.41
Checking for            TimeDate (v2.21)   ok: found v2.22
Checking for           PathTools (v0.84)   ok: found v3.2501
Checking for                 DBI (v1.41)   ok: found v1.607
Checking for    Template-Toolkit (v2.15)   ok: found v2.20
Checking for          Email-Send (v2.16)   ok: found v2.192
Checking for          Email-MIME (v1.861)  ok: found v1.861
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.442

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for          DBD-Oracle (v1.19)   ok: found v1.21

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.41
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.32
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v5.814
Checking for         PatchReader (v0.9.4)  ok: found v0.9.5
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)     ok: found v0.39
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (any)     ok: found v0.69
Checking for         HTML-Parser (v3.40)   ok: found v3.56
Checking for       HTML-Scrubber (any)     ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any)     ok: found v1.315
Checking for         Email-Reply (any)     ok: found v1.202
Checking for            mod_perl (v1.999022)  not found

* NOTE: You must run any commands listed below as Administrator.

***********************************************************************
* Note For Windows Users                                              *
***********************************************************************
* In order to install the modules listed below, you first have to run *
* the following command as an Administrator:                          *
*                                                                     *
*   ppm repo add theory58S http://cpan.uwinnipeg.ca/PPMPackages/10xx/
*                                                                     *
***********************************************************************
**********************************************************************
* OPTIONAL MODULES                                                   *
**********************************************************************
* Certain Perl modules are not required by Bugzilla, but by          *
* installing the latest version you gain access to additional        *
* features.                                                          *
*                                                                    *
* The optional modules you do not have installed are listed below,   *
* with the name of the feature they enable. If you want to install   *
* one of these modules, just run the appropriate command in the      *
* "COMMANDS TO INSTALL" section.                                     *
**********************************************************************

***********************************************************************
* MODULE NAME * ENABLES FEATURE(S)                                    *
***********************************************************************
*  PerlMagick * Optionally Convert BMP Attachments to PNGs            *
* Authen-SASL * SMTP Authentication                                   *
*  RadiusPerl * RADIUS Authentication                                 *
*    mod_perl * mod_perl                                              *
***********************************************************************
COMMANDS TO INSTALL:

     PerlMagick: ppm install PerlMagick
    Authen-SASL: ppm install Authen-SASL
     RadiusPerl: ppm install RadiusPerl
       mod_perl: ppm install mod_perl
Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

    http://cyberelk.net/tim/patchutils/

Checking for           DBD-mysql (v4.00)   ok: found v4.005
Checking for           MySQL (v4.1.2)  ok: found v5.1.30-community

Removing index 'bugs_short_desc_idx' from the bugs table...
Removing index 'longdescs_thetext_idx' from the longdescs table...
Bugzilla now uses the InnoDB storage engine in MySQL for most tables.
Converting tables to InnoDB:
Converting table attach_data... done.
Converting table attachments... done.
Converting table bug_group_map... done.
Converting table bug_severity... done.
Converting table bug_status... done.
Converting table bugs... done.
Converting table bugs_activity... done.
Converting table bz_schema... done.
Converting table category_group_map... done.
Converting table cc... done.
Converting table classifications... done.
Converting table components... done.
Converting table dependencies... done.
Converting table duplicates... done.
Converting table email_setting... done.
Converting table fielddefs... done.
Converting table flagexclusions... done.
Converting table flaginclusions... done.
Converting table flags... done.
Converting table flagtypes... done.
Converting table group_control_map... done.
Converting table group_group_map... done.
Converting table groups... done.
Converting table keyworddefs... done.
Converting table keywords... done.
Converting table logincookies... done.
Converting table longdescs... done.
Converting table milestones... done.
Converting table namedqueries... done.
Converting table op_sys... done.
Converting table priority... done.
Converting table products... done.
Converting table profile_setting... done.
Converting table profiles... done.
Converting table profiles_activity... done.
Converting table quips... done.
Converting table rep_platform... done.
Converting table resolution... done.
Converting table series... done.
Converting table series_categories... done.
Converting table series_data... done.
Converting table setting... done.
Converting table setting_value... done.
Converting table tokens... done.
Converting table user_group_map... done.
Converting table versions... done.
Converting table votes... done.
Converting table watch... done.
Converting table whine_events... done.
Converting table whine_queries... done.
Converting table whine_schedules... done.
Adding new table bugs_fulltext ...
Adding new table component_cc ...
Adding new table namedqueries_link_in_footer ...
Adding new table namedquery_group_map ...
Adding new table status_workflow ...
Creating ./data directory...
Creating ./data/attachments directory...
Creating ./data/duplicates directory...
Creating ./data/mining directory...
Creating ./data/webdot directory...
Creating ./skins/contrib/Dusk/yui directory...
Creating graphs directory...
Creating skins/custom directory...
Creating ./skins/contrib/Dusk/IE-fixes.css...
Creating ./skins/contrib/Dusk/admin.css...
Creating ./skins/contrib/Dusk/create_attachment.css...
Creating ./skins/contrib/Dusk/dependency-tree.css...
Creating ./skins/contrib/Dusk/duplicates.css...
Creating ./skins/contrib/Dusk/editusers.css...
Creating ./skins/contrib/Dusk/help.css...
Creating ./skins/contrib/Dusk/index.css...
Creating ./skins/contrib/Dusk/panel.css...
Creating ./skins/contrib/Dusk/params.css...
Creating ./skins/contrib/Dusk/release-notes.css...
Creating ./skins/contrib/Dusk/show_bug.css...
Creating ./skins/contrib/Dusk/show_multiple.css...
Creating ./skins/contrib/Dusk/summarize-time.css...
Creating ./skins/contrib/Dusk/voting.css...
Creating ./skins/contrib/Dusk/yui/calendar.css...
Creating ./Bugzilla/.htaccess...
Creating ./data/.htaccess...
Creating ./data/attachments/.htaccess...
Creating ./data/webdot/.htaccess...
Creating ./lib/.htaccess...
Creating ./template/.htaccess...
Creating .htaccess...

Bugzilla requires an SMTP server to function on Windows.
Please enter your SMTP server's hostname: smtp
Precompiling templates...done.
Adding new column 'type' to the 'fielddefs' table...
Adding new column 'custom' to the 'fielddefs' table...
Adding new column 'enter_bug' to the 'fielddefs' table...
Renaming column 'fielddefs.fieldid' to 'fielddefs.id'...
Updating column creator in table series ...
Old: mediumint NOT NULL
New: mediumint
Adding new column 'comment_id' to the 'longdescs' table...
Updating column id in table flags ...
Old: mediumint NOT NULL PRIMARY KEY
New: mediumint auto_increment NOT NULL PRIMARY KEY
Deleting the 'is_active' column from the 'flags' table...
Updating column short_desc in table bugs ...
Old: mediumtext NOT NULL
New: varchar(255) NOT NULL
Adding new column 'id' to the 'namedqueries' table...
Deleting the 'linkinfooter' column from the 'namedqueries' table...
Adding new column 'sortkey' to the 'classifications' table...
Adding new column 'disable_mail' to the 'profiles' table...
Updating column milestoneurl in table products ...
Old: tinytext NOT NULL
New: tinytext DEFAULT '' NOT NULL
Updating column disallownew in table products ...
Old: tinyint NOT NULL
New: tinyint DEFAULT 0 NOT NULL
Updating column votesperuser in table products ...
Old: smallint NOT NULL
New: smallint DEFAULT 0 NOT NULL
Updating column votestoconfirm in table products ...
Old: smallint NOT NULL
New: smallint DEFAULT 0 NOT NULL
Deleting the 'refreshed_when' column from the 'profiles' table...
Deleting the 'last_changed' column from the 'groups' table...
Updating column id in table flagtypes ...
Old: smallint NOT NULL PRIMARY KEY
New: smallint auto_increment NOT NULL PRIMARY KEY
Updating column id in table keyworddefs ...
Old: smallint NOT NULL PRIMARY KEY
New: smallint auto_increment NOT NULL PRIMARY KEY
Updating column userid in table tokens ...
Old: mediumint NOT NULL
New: mediumint
Updating column disabledtext in table profiles ...
Old: mediumtext NOT NULL
New: mediumtext DEFAULT '' NOT NULL
Updating column realname in table profiles ...
Old: varchar(255)
New: varchar(255) DEFAULT '' NOT NULL
Removing index 'longdescs_who_idx' from the longdescs table...
Adding new index 'longdescs_who_idx' to the longdescs table ...
Adding new column 'subclass' to the 'setting' table...
Updating column thetext in table longdescs ...
Old: mediumtext
New: mediumtext NOT NULL
Adding new column 'editcomponents' to the 'group_control_map' table...
Adding new column 'editbugs' to the 'group_control_map' table...
Adding new column 'canconfirm' to the 'group_control_map' table...
Adding new column 'type' to the 'longdescs' table...
Adding new column 'extra_data' to the 'longdescs' table...
Adding new column 'id' to the 'versions' table...
Adding new column 'id' to the 'milestones' table...
Adding new column 'is_open' to the 'bug_status' table...
Marking closed bug statuses as such...
Now filling the 'status_workflow' table with valid bug status transitions...
Adding new column 'icon_url' to the 'groups' table...
Updating column bug_file_loc in table bugs ...
Old: TEXT
New: mediumtext
Updating column description in table attachments ...
Old: mediumtext NOT NULL
New: tinytext NOT NULL
Updating column mimetype in table attachments ...
Old: mediumtext NOT NULL
New: tinytext NOT NULL
Updating column description in table flagtypes ...
Old: TEXT
New: mediumtext NOT NULL
Updating column description in table fielddefs ...
Old: mediumtext NOT NULL
New: tinytext NOT NULL
Updating column description in table groups ...
Old: TEXT NOT NULL
New: mediumtext NOT NULL
Updating column quip in table quips ...
Old: TEXT NOT NULL
New: mediumtext NOT NULL
Updating column query in table namedqueries ...
Old: mediumtext NOT NULL
New: mediumtext NOT NULL
Adding new column 'modification_time' to the 'attachments' table...
Setting the modification time for attachments...
Updating column modification_time in table attachments ...
Old: datetime
New: datetime NOT NULL
Adding new index 'attachments_modification_time_idx' to the attachments table ...
Updating column query_type in table namedqueries ...
Old: tinyint NOT NULL
New: tinyint DEFAULT 0 NOT NULL
Populating bugs_fulltext... (this can take a long time.)
Adding foreign key: attachments.submitter_id -> profiles.userid...
Adding foreign key: bugs_activity.who -> profiles.userid...
Adding foreign key: bugs_fulltext.bug_id -> bugs.bug_id...
Adding foreign key: cc.who -> profiles.userid...
Adding foreign key: component_cc.user_id -> profiles.userid...
Adding foreign key: components.initialowner -> profiles.userid...
Adding foreign key: components.initialqacontact -> profiles.userid...
Adding foreign key: email_setting.user_id -> profiles.userid...
Adding foreign key: logincookies.userid -> profiles.userid...
Adding foreign key: namedqueries.userid -> profiles.userid...
Adding foreign key: namedqueries_link_in_footer.namedquery_id -> namedqueries.id...
Adding foreign key: namedqueries_link_in_footer.user_id -> profiles.userid...
Adding foreign key: profile_setting.user_id -> profiles.userid...
Adding foreign key: profiles_activity.userid -> profiles.userid...
Adding foreign key: profiles_activity.who -> profiles.userid...
Adding foreign key: profiles_activity.fieldid -> fielddefs.id...
Adding foreign key: tokens.userid -> profiles.userid...
Adding foreign key: votes.who -> profiles.userid...
Adding foreign key: watch.watcher -> profiles.userid...
Adding foreign key: watch.watched -> profiles.userid...
Adding foreign key: whine_events.owner_userid -> profiles.userid...
Adding foreign key: whine_queries.eventid -> whine_events.id...
Adding foreign key: whine_schedules.eventid -> whine_events.id...
Adding a new user setting called 'skin'
Adding a new user setting called 'quote_replies'
Adding a new user setting called 'lang'
Adding a new user setting called 'zoom_textareas'
Adding a new user setting called 'state_addselfcc'

Now that you have installed Bugzilla, you should visit the
'Parameters' page (linked in the footer of the Administrator
account) to ensure it is set up as you wish - this includes
setting the 'urlbase' option to the correct URL.
Comment 17 Dan 2008-12-05 09:16:35 PST
In case it's not perfectly clear from my previous postings.  If I follow the exact same steps using the 3.0.6 release it doesn't matter what format the template is in it always works fine.  Something in 3.2 has changed that doesn't handle DOS formatted templates anymore.
Comment 18 Marc Schumann [:Wurblzap] 2008-12-05 09:36:20 PST
Looking through http://bonsai.mozilla.org/cvslog.cgi?file=mozilla/webtools/bugzilla/Bugzilla/Template.pm, it might be that we're talking about bug 413121. What happens if back out its patch? (Besides from re-introducing the bug it fixes.)
Comment 19 Vitaly Fedrushkov 2008-12-05 21:07:58 PST
Created attachment 351657 [details]
Message sample

Confirmed CVS tip, ActivePerl 5.10 and Test mailer.

Note how CR (0x0D) chars are escaped anywhere except MIME headers.  Note double CRs where

[% IF Param('useclassification') %]

was parsed.
Comment 20 Max Kanat-Alexander 2008-12-10 22:23:51 PST
What's the difference here between Windows and Linux? If we save the template with DOS newlines on Linux, does the same thing happen?
Comment 21 Dan 2008-12-11 01:41:57 PST
My Linux VM is messed up right now, but I just completed an install of on OS X 10.5 and the problem is seen there as well.  DOS formatted is broken, but UNIX format works fine.  I would assume the same is the case for Linux.
Comment 22 Dan 2008-12-11 01:42:53 PST
My Linux VM is messed up right now, but I just completed a clean install of Bugzilla 3.2 on OS X 10.5  (no upgrade) and the problem is seen there as well.  I would assume the same is the case for Linux.
Comment 23 Max Kanat-Alexander 2008-12-11 02:19:35 PST
Well, in that case this is probably something we should look at and get fixed pretty soon. It's not the *most* important bug we have (since most people won't have DOS newlines in that file), but it's clearly something we're doing (or more likely, a MIME module is doing) new in 3.2 that should be fixed.
Comment 24 Yann 2008-12-14 00:07:21 PST
Confirmed with w2k3, activestate perl 5.10 (latest version), bugzilla 3.2.

To fix, I took the 3.0 template, and modified manually to look like 3.2 one.
Comment 25 Thomas Stegh 2008-12-17 00:12:04 PST
Confirmed for Windows Server 2003, mysql-5.1.30-win32, ActivePerl 5.8.8.824 for Windows (x86), Bugzilla 3.2 (from CVS).

To fix, I took the newchangedmail.txt.u8-unix.tmpl from the attachment https://bugzilla.mozilla.org/attachment.cgi?id=351552 (see comment #c12)
Comment 26 Vitaly Fedrushkov 2009-01-08 00:26:31 PST
Bug 464992 comment 41: related bug can be remedied by upgrading Perl to 5.10.0.
Comment 27 Max Kanat-Alexander 2009-01-09 00:15:35 PST
*** Bug 472788 has been marked as a duplicate of this bug. ***
Comment 28 Yann 2009-01-09 05:57:40 PST
(In reply to comment #26)
> Bug 464992 comment 41: related bug can be remedied by upgrading Perl to 5.10.0.

I already run perl 5.10 (activeperl build 1004), and this bug occurs on that version.
Comment 29 Frédéric Buclin 2009-01-25 09:05:04 PST
mkanat, this is the last bug blocking 3.2.1. Is this a hard blocker or can we postpone it?
Comment 30 Max Kanat-Alexander 2009-01-25 12:48:45 PST
(In reply to comment #29)
> mkanat, this is the last bug blocking 3.2.1. Is this a hard blocker or can we
> postpone it?

  Due to the security fixes that we want to release in 3.2.1, and given that there is no patch here, yes, we can postpone it. (But I want a blocking3.2.2 flag for it.)
Comment 31 Frédéric Buclin 2009-02-04 07:00:02 PST
I just played with Bugzilla 3.3.3 on Windows 2000, and bugmail looks good, but emails you get for new accounts is clearly wrong. Almost everything is on a single line.
Comment 32 Max Kanat-Alexander 2009-02-06 10:29:24 PST
I'm working on this. Apparently what's happening is that one of the lines has two \r's in it, and Email::MIME->new thinks that thus the first header ends there. Not quite sure why.
Comment 33 Max Kanat-Alexander 2009-02-06 10:35:44 PST
Created attachment 360937 [details]
What Email::MIME gets and how it parses it

So, here's a file containing the same email twice.

The first part is the exact input that Email::MIME is getting on Windows. The second part is the output of $email->as_string after parsing the above input.

rjbs--Is this something that we have to fix in Bugzilla (somehow making sure that headers don't have two \r's in them in a row) or that needs to be fixed in the Email:: modules?
Comment 34 Max Kanat-Alexander 2009-02-06 10:37:06 PST
Comment on attachment 360937 [details]
What Email::MIME gets and how it parses it

I'm changing the MIME type to octet-stream so that the newlines come through intact. This file has to be downloaded, it can't just be viewed in the browser (since the browser will do newline conversion).
Comment 35 Max Kanat-Alexander 2009-02-16 18:50:46 PST
Okay, so, short story is that it seems to be a bug in Template-Toolkit. I just filed this:

  https://rt.cpan.org/Ticket/Display.html?id=43345

I don't know why we weren't hitting it before--maybe it's also a bug in Email::MIME that only happens with the newer versions that Bugzilla 3.2 requires.
Comment 36 Max Kanat-Alexander 2009-02-16 19:10:58 PST
Created attachment 362659 [details] [diff] [review]
v1

Okay, this fixes it. There are probably more efficient ways to fix it, but this won't really be slow and it's the simplest solution. Also, we're sending bad line endings (in violation of RFC 2822) sometimes, so I fixed that at the same time since it was now convenient to do so.

(Applicable section of RFC is: http://tools.ietf.org/html/rfc2822#section-2.1 )
Comment 37 Max Kanat-Alexander 2009-02-16 19:15:33 PST
Created attachment 362660 [details] [diff] [review]
v2

Oh, there was a slight bug in the previous patch (didn't check "ref $msg" in one place properly).
Comment 38 Max Kanat-Alexander 2009-02-16 19:22:35 PST
Created attachment 362662 [details] [diff] [review]
Actual v2

Accidentally uploaded v1 again and called it v2. Here's the right v2.
Comment 39 Max Kanat-Alexander 2009-02-16 19:40:13 PST
Created attachment 362664 [details] [diff] [review]
v3

glob pointed out that \n can mean something different on different platforms. Also, thanks to some inspiration from the perlport man page, I made it into one regex.
Comment 40 Byron Jones ‹:glob› 2009-02-16 19:47:00 PST
Comment on attachment 362664 [details] [diff] [review]
v3

r=glob
Comment 41 Frédéric Buclin 2009-02-21 05:20:13 PST
Max, please commit this patch so that we can do some testing on Windows too (so that we just have to cvs update rather than applying the patch manually).
Comment 42 Max Kanat-Alexander 2009-02-21 16:45:45 PST
tip:

Checking in Bugzilla/Mailer.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Mailer.pm,v  <--  Mailer.pm
new revision: 1.27; previous revision: 1.26
done

3.2:

Checking in Bugzilla/Mailer.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Mailer.pm,v  <--  Mailer.pm
new revision: 1.20.2.4; previous revision: 1.20.2.3
done
Comment 43 David Lawrence [:dkl] 2009-04-16 09:17:59 PDT
Max, ever since we applied this patch, we have been getting reports of emails being sent out with extra ^M's visible in them that were not there before. It shows up most in people who are using text mail clients such as Mutt. Here is an example of an email that a user reported:

----------

In-Reply-To: <bug-250412-92433@bugzilla.redhat.com>
References: <bug-250412-92433@bugzilla.redhat.com>
Auto-Submitted: auto-generated
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Scanned-By: MIMEDefang 2.58 on 172.16.52.254
Status: RO
Content-Length: 671
Lines: 21

Please do not reply directly to this email. All additional=0D
comments should be made in the comments box of this bug.=0D
=0D
=0D
https://bugzilla.redhat.com/show_bug.cgi?id=3D250412=0D
=0D
=0D
J=C3=A9r=C3=B4me Fenal <jfenal@redhat.com> changed:=0D
=0D
           What    |Removed                     |Added=0D
---------------------------------------------------------------------------=
-=0D
                 CC|                            |jfenal@redhat.com=0D
=0D
=0D
=0D
=0D
-- =0D
Configure bugmail: https://bugzilla.redhat.com/userprefs.cgi?tab=3Demail=0D
------- You are receiving this mail because: -------=0D
You are watching someone on the CC list of the bug.=

-----------

It seems to only effect emails with are sent through Bugzilla::Mailer::MessageToMTA (which is most) and is related to the recent change made there.

    my $email;
    if (ref $msg) {
        $email = $msg;
    }   
    else {
        # RFC 2822 requires us to have CRLF for our line endings and
        # Email::MIME doesn't do this for us. We use \015 (CR) and \012 (LF)
        # directly because Perl translates "\n" depending on what platform
        # you're running on. See http://perldoc.perl.org/perlport.html#Newlines
        # We check for multiple CRs because of this Template-Toolkit bug:
        # https://rt.cpan.org/Ticket/Display.html?id=43345
        $msg =~ s/(?:\015+)?\012/\015\012/msg;
        $email = new Email::MIME($msg);
    }   

If I make the following change while doing a reproducer, I do not get the extra ^M's:

-        $msg =~ s/(?:\015+)?\012/\015\012/msg;
+        $msg =~ s/(?:\015+)\012/\015\012/msg;

I understand that the RFC requires \015\012 at the end of each line in the message body but for some reason we are getting \015\015\012 in some cases.

Any ideas why this might be happening?

Dave
Comment 44 Max Kanat-Alexander 2009-04-16 15:24:02 PDT
(In reply to comment #43)
> Max, ever since we applied this patch, we have been getting reports of emails
> being sent out with extra ^M's visible in them that were not there before.

  bug 486206

Note You need to log in before you can comment on or make changes to this bug.