812719 - Python markup map needs improvement

Status: RESOLVED FIXED

(Webtools Graveyard :: MXR, defect)

Description

[Justin Wood (:Callek)]

So, today I finally got around to discussing with jakem about how we see "** Fatal" on some python files in the build/ (releng) section of mxr commonly.

Jake peeked at the logs and it seemed it was failing on &filelookup by actually stuffing a raw '[' into a regex and thus barfing since it didn't get a closing ']'

We hacked out by escaping [ and ] directly, because that "looked" like the sane answer, but it didn't seem to help much (and "seemed" to make things percieveably worse, other than finishing to render the whole file)

By mentally tracing lines it looks like the issue with that fatal was because of the combination of no single-quote chars matching for strings and the "include" matched with:
   \\b(?:from)\\s+
And terminates its fragment with: 
  (?:$|\\bimport\\b)

on a string like:
            diffs.append('buildHorizon changed from %s to %s' % (self.buildHorizon, setup['buildHorizon']))

so what we get is a sub-fragment of:
from %s to %s' % (self.buildHorizon, setup['buildHorizon']))

for the "from.*$" basically.

Then following that we try to map the "include" to a file that we can link with, which is then [1] modified in place with the regex:
    $frag =~ s#(\s*[\"\'])(.*?)([\"\'])#
              ($1)."\0$2\0".($3)#e

Yeilding \1 = from\ %s\ to\ %s'
         \2 = \ %\ (self.buildHorizon,\ setup[
         \3 = buildHorizon']))

(Not part of the problem but for completeness we THEN do [2] an conversion of \. to / for file paths and add .py ... if we know of the file in our index)

THEN we're calling into &filelookup with the value of \2

---

SOLUTION:

So it seems that if we properly guard "from" from inside a single quoted string with a proper comment section we won't have the above issue, I took the liberty of correcting a few less critical but equally annoying problems with the parser while I was here.

I unfortunately do not have an env locally that I can test this, nor do I know why timeless (tracing back to the original landing of the py parser) marked single quotes as broken so I do want to see this tested if someone has the abil to.


[1] http://hg.mozilla.org/webtools/mxr/file/c2cee2683916/lib/LXR/Common.pm#l1016
[2] http://hg.mozilla.org/webtools/mxr/file/c2cee2683916/lib/LXR/Common.pm#l1027

Comment 1

[Justin Wood (:Callek)]

Attachment: [mxr] v1

BMO barfed on me, and didn't attach, here it is

Comment 3

[Justin Wood (:Callek)]

jake: ping

Comment 4

[timeless]

Comment on attachment 682710 [details] [diff] [review]
[mxr] v1

Looks good, thanks.

Sorry, bugmail delivery to me doesn't get my attention - you can generally reach me on IRC (unless IRCCloud is broken, or I'm in some vacation/travel state).

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 682710 [details] [diff] [review]
[mxr] v1

landed in prep for jake pushing live http://hg.mozilla.org/webtools/mxr/rev/4cfb12c47866

Comment 6

[Jake Maul [:jakem]]

MXR is updated!

Hopefully we didn't horribly break anything... if we did, please re-open to back out this change (or deploy a subsequent fix). :)