684997 - Running the cross-reference generator generates LexErrors

Status: RESOLVED FIXED

(Webtools Graveyard :: DXR, defect, P1)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

Running the cross-reference generator to generate the indexes produces LexErrors for .idl files and hangs. The script cannot be interrupted by SIGINT, and needs to be killed by SIGTERM or SIGKILL.

I'm calling this from my dxr source folder like so:
sudo python dxr-index.py -f /var/www/html/dxr/dxr.config 

Here's an example error:
Error on file /home/mdas/Code/mozilla/src/other-licenses/ia2/AccessibleText.idl:
Traceback (most recent call last):
  File "dxr-index.py", line 59, in async_toHTML
    dxr.htmlbuilders.make_html(srcpath, dstfile, treeconfig, big_blob)
  File "/home/mdas/Code/dxr/dxr/dxr/htmlbuilders.py", line 198, in make_html
    builder.toHTML()
  File "/home/mdas/Code/dxr/dxr/dxr/htmlbuilders.py", line 45, in toHTML
    self.writeMainContent(out)
  File "/home/mdas/Code/dxr/dxr/dxr/htmlbuilders.py", line 88, in writeMainContent
    self.writeMainBody(out)
  File "/home/mdas/Code/dxr/dxr/dxr/htmlbuilders.py", line 122, in writeMainBody
    for syn in syntax_regions:
  File "/home/mdas/Code/dxr/dxr/xref-tools/moztools/indexer.py", line 213, in getSyntaxRegions
    tok = self.tokenizer.token()
  File "/home/mdas/Code/dxr/dxr/xref-tools/moztools/indexer.py", line 182, in token
    return self.lexer.token()
  File "/home/mdas/Code/dxr/dxr/ply/lex.py", line 387, in token
    raise LexError("Scanning error. Illegal character '%s'" % (lexdata[lexpos]), lexdata[lexpos:])
LexError: Scanning error. Illegal character '.'

Comment 1

[Carlos Garnacho]

I've just made a pull request containing several patches to the idl parser to fix this bug:
https://github.com/mozilla/dxr/pull/28

With those patches no idl file issues any error, and all end up available in the indexed content with proper syntax highlighting

Comment 2

[(no longer active)]

Thanks, merged.