271529 - bonsai shows a /m/src/ns filename

Status: RESOLVED FIXED

(Webtools Graveyard :: Bonsai, defect, P2)

Description

[Christian :Biesinger (don't email me, ping me on IRC)]

note: I am not sure whether this is a bonsai bug or just an issue of
mozilla.org's installation. filing it here for now.

http://tinderbox.mozilla.org/bonsai/cvsguess.cgi?file=client.mk&rev=HEAD&mark=390#380
shows me /m/src/ns/client.mk as a possible filename. Clicking on it does not
show me the file (not even if I add a .. so that I'd get out of /l10n-cvsroot
:-) ), but it still sounds like I should not get that.

Comment 1

[Dave Miller [:justdave]]

that pathname doesn't exist on the server either, so it's not showing a live
pathname or anything...

That path sounds suspiciously like some of the pathnames we used to have when
stuff was set up at Netscape though.

Comment 2

[cls]

cvsguess.cgi just queries the bonsai db for files that match that name so it
sounds like mozilla & netscape used to share a bonsai installation and the
netscape paths were never cleared from the database.  When was the last time we
did a 'drop all tables & rebuilt from scratch'? 

Comment 3

[timeless]

afaik we haven't managed to successfully rebuild the database in years :(

Comment 4

[cls]

So it sounds like we're well overdue for one given the various checkin lossage
fixes that have gone in over the past few months.  Who wants to do the honors?

Comment 5

[Dave Miller [:justdave]]

(In reply to comment #4)
> So it sounds like we're well overdue for one given the various checkin lossage
> fixes that have gone in over the past few months.  Who wants to do the honors?

I tried to over this last weekend.  rebuildcvshistory.cgi is fatally broken.

It ran for 8 hours then hung.  I don't know why it hung.  Trying to get it to
pick up where it left off revealed that the "start with this directory" thing on
the admin page is also broken (it just doesn't do anything at all if you tell it
that), and that the "do only this directory" thing is also broken (it does
everything anyway).  I managed to fix the "do only this directory" which is
patched on bonsai.m.o right now, and attached to a bug somewhere...

Comment 6

[cls]

Does bonsai.m.o's rebuildcvshistory.cgi have all of the latest changes? In
particular, the fix from bug 258668?  I've run rebuildcvshistory.cgi (via
cmdline) on our local cvs repo which has 60gig of junk with a lot of non-unixy
filenames and it no longer times out, it just spits outs filenames for half a day.

Comment 7

[Dave Miller [:justdave]]

It was using the October 10th checkout.  That bugfix was checked in sometime in
September, so yes, we had that fix already.

Comment 8

[cls]

Ok, I rsync'd a copy of mozilla's cvs repo and setup bonsai locally to test
this.  When running rebuildcvshistory.cgi from the commandline, it completes but
does spit out warnings for about 40 files.  I'm not sure if these warnings will
cause apache to hang or not.  

[Fri Dec  3 16:57:03 2004] rebuildcvshistory.cgi: Malformed UTF-8 character
(unexpected non-continuation byte 0x6b, immediately after start byte 0xe1) in
substitution (s///) at globals.pl line 421, <RLOG_PROC> line 388.

The server is using perl 5.8.0 from RH9, if that makes a difference.  The full
rebuild took 30 hrs to complete on a AMD K6-2 333Mhz w/ 64mb ram.  I'm going to
guess that lounge has quite a bit more power than that so it shouldn't take
nearly as long.  I wonder if we're just hitting some apache limit with the
timeouts when calling the script as a cgi.

Comment 9

[Dave Miller [:justdave]]

(In reply to comment #8)
> Ok, I rsync'd a copy of mozilla's cvs repo and setup bonsai locally to test
> this.  When running rebuildcvshistory.cgi from the commandline, it completes
> but does spit out warnings for about 40 files.  I'm not sure if these
> warnings will cause apache to hang or not.  

If there's more than 4K of stderr before the script exits, it'll hang, so that
could very well be it, if there's lots of them.

> [Fri Dec  3 16:57:03 2004] rebuildcvshistory.cgi: Malformed UTF-8 character
> (unexpected non-continuation byte 0x6b, immediately after start byte 0xe1) in
> substitution (s///) at globals.pl line 421, <RLOG_PROC> line 388.
> 
> The server is using perl 5.8.0 from RH9, if that makes a difference.  The full
> rebuild took 30 hrs to complete on a AMD K6-2 333Mhz w/ 64mb ram.  I'm going
> to guess that lounge has quite a bit more power than that so it shouldn't take
> nearly as long.  I wonder if we're just hitting some apache limit with the
> timeouts when calling the script as a cgi.

Lounge is no longer running bonsai. Lounge was a slowpoke anyway ;)  Mecha is a
2 x 2.8 Ghz Hyperthreaded Xeon processor with 4 GB of RAM. :)  Should be much
faster.

Mecha also is running perl 5.6.1, which sucks at unicode stuff.  I wonder if
those are errantly-created filenames...  we've found a few of those in the past
that have been nuked.  (mostly people trying to upload from really old MacCVS
clients that sucked at charset conversion)

Comment 10

[cls]

It's about 7k of warnings.   We might be able to work around that utf warning by
using 'use bytes;' or somehow setting LANG=C before invoking the script.

Given the referenced line from globals.pl (421), I think the problem strings
come from checkin comments rather than the filenames.

Comment 11

[Dave Miller [:justdave]]

(In reply to comment #10)
> It's about 7k of warnings.   We might be able to work around that utf warning
> by using 'use bytes;' or somehow setting LANG=C before invoking the script.

which requires Perl 5.8+.  

> Given the referenced line from globals.pl (421), I think the problem strings
> come from checkin comments rather than the filenames.

At this point I would vote for recoding the checkin comments as UTF-8 in the RCS
files.  Got a way to get me a list of which files are affected?  On the other
hand, Perl 5.6.1 (which mecha is running) doesn't care about UTF-8, and
shouldn't be throwing those warnings to begin with.

Comment 12

[cls]

'use bytes' should work with perl 5.6.  See
http://bugzilla.gnome.org/show_bug.cgi?id=101221 .  

I can give you a list of the affected files but I didn't see anything odd in the
checkin comment of  ./mozilla/config/mkdetect/Attic/detect_motif.sh,v .

Comment 13

[cls]

Attachment: files with utf8 warnings

Comment 14

[Simon Montagu :smontagu]

(In reply to comment #12)

> I didn't see anything odd in the
> checkin comment of  ./mozilla/config/mkdetect/Attic/detect_motif.sh,v .

Rev. 1.5: Thanks to "Gábor Lipták" <[e-mail]> for poiting this out.

Comment 15

[Simon Montagu :smontagu]

I think "use bytes" is more practical than recoding. As nobody should be
surprised to learn, the checkin comments listed for the files in attachment
167989 [details] are in a mish-mosh of encodings: mostly ISO-8859-1, but some UTF-8
(mozilla/gfx/src/xlib/nsDeviceContextXlib.cpp) and some apparently IBM-850
(large numbers of checkins by [e-mail])

Comment 16

[cls]

Attachment: Use 'use bytes' & bump min perl version to 5.6.0 (checked in)

Comment 17

[cls]

Comment on attachment 168054 [details] [diff] [review]
Use 'use bytes' & bump min perl version to 5.6.0 (checked in)

The patch has been checked in.	A rebuild should work without a problem now.

Comment 18

[Dave Miller [:justdave]]

I'll do the rebuild during the week of Jun 20th.  We have other outages
scheduled that week anyway, it'll fit in well.

Comment 19

[Dave Miller [:justdave]]

This rebuild this is waiting on got done a long time ago.  We probably need a new one, but this isn't the bug for it. :)

Comment 20

[cls]

The URL from comment #0 still shows /m/src/ns/client.mk . Was the entire tree dropped & rebuilt?  I think that's going to be the only way to clear out those old entries (short of doing it by hand).