Bug 125313 - GroovyLexer.nextToken() swallows last doublequote in input
GroovyLexer.nextToken() swallows last doublequote in input
Status: VERIFIED FIXED
Product: groovy
Classification: Unclassified
Component: Editor
6.x
All All
: P1 (vote)
: 6.x
Assigned To: schmidtm
issues@groovy
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-01-15 17:43 UTC by schmidtm
Modified: 2008-10-30 14:25 UTC (History)
0 users

See Also:
Issue Type: DEFECT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description schmidtm 2008-01-15 17:43:55 UTC
Hi,

The GroovyLexer swallows the very last doublequote (") from the input stream. This means the last Token is one char
shorter than it should be.

These are the tests which expose this error (in GroovyLexerBatchTest.java):

testGstringLexing4
testMultipleStringConstants
testMultipleStringConstantsClosed
testMultipleStringConstantsDosTerminated
testLengthTest2
testLengthTest5
testQuotes

One sidenode: If one changes in GroovyLexer.java in (as of this writing) Line 115 : lexerInput.readLength()
into lexerInput.readLengthEOF()


   109      public Token<GroovyTokenId> nextToken() {
   110          try {
   111              groovyjarjarantlr.Token antlrToken = scanner.nextToken();
   112              if (antlrToken != null) {
   113                  int intId = antlrToken.getType();
   114
   115                  int len = lexerInput.readLengthEOF() - myCharBuffer.getExtraCharCount();
   116                  if ( antlrToken.getText() != null ) {
   117                      len = Math.max( len, antlrToken.getText().length() );

all the tests succeed but then i get a reproducible exception in the editor:

WARNING [org.netbeans.modules.editor.lib2.highlighting.CompoundHighlightsContainer]: The layer failed to supply highligh
ts: org.netbeans.modules.editor.lib2.highlighting.SyntaxHighlighting@111cdd6
java.lang.IllegalStateException: Lexer org.netbeans.modules.groovy.editor.lexer.GroovyLexer@116a2ac returned null token
but lexerInput.readLength()=1 - these characters need to be tokenized. Fix the lexer.
        at org.netbeans.lib.lexer.LexerUtilsConstants.checkLexerInputFinished(LexerUtilsConstants.java:116)
        at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:276)
        at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingContainerImpl(IncTokenList.java:197)
        at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingContainer(IncTokenList.java:192)
        at org.netbeans.api.lexer.TokenSequence.moveNext(TokenSequence.java:405)
        at org.netbeans.modules.editor.lib2.highlighting.SyntaxHighlighting$HSImpl.moveTheSequence(SyntaxHighlighting.ja
va:541)
        at org.netbeans.modules.editor.lib2.highlighting.SyntaxHighlighting$HSImpl.moveNext(SyntaxHighlighting.java:285)

        at org.netbeans.spi.editor.highlighting.support.OffsetsBag.addAllHighlightsImpl(OffsetsBag.java:574)
Comment 1 martin_adamek 2008-01-16 07:10:57 UTC
See issue 124912. Problem was fixed there I guess.
Comment 2 Lukas Jungmann 2008-09-29 12:01:24 UTC
v.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo