This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
With a fresh trunk build as of tonight (August 19th 2007) I suddenly get exceptions opening certain Ruby files. The stacktrace starts as follows: java.lang.ArrayIndexOutOfBoundsException: 64 at org.netbeans.lib.lexer.LanguageOperation.embeddingPresence(LanguageOperation.java:200) at org.netbeans.lib.lexer.EmbeddingContainer.embeddedTokenList(EmbeddingContainer.java:85) at org.netbeans.lib.lexer.LexerUtilsConstants.embeddedTokenList(LexerUtilsConstants.java:246) at org.netbeans.api.lexer.TokenSequence.embeddedImpl(TokenSequence.java:290) at org.netbeans.api.lexer.TokenSequence.embedded(TokenSequence.java:284) I will attach the full stacktrace. To reproduce: Save the second attachment, foo_controller.rb, as that exact name, somewhere. Then start up NetBeans and use File | Open File to open foo_controller.rb. For me, on OSX, this starts spewing exceptions (and repainting in the project manager etc. starts failing since exceptions are thrown from the event dispatch loop). Marking as a P1 since this seems to block all Rails files from being opened. I thought something odd had happened to my build, so I checked out a complete new build of NetBeans in a separate directory and reproduced the same problem.
Created attachment 46860 [details] Full stacktrace
Created attachment 46861 [details] Sample source file: open file from blank userdir with modules installed to reproduce
The RubyLexer returns a token with id that is not contained in the ruby language in RubyTokenId. I have added a check after Lexer.nextToken() gets created for its TokenId: java.lang.IllegalStateException: Invalid TokenId=org.netbeans.api.gsf.GsfTokenId:NONUNARY_OP:64 returned from lexer=org.netbeans.modules.ruby.lexer.RubyLexer@1ab05a3 for language=text/x-ruby, LH: org.netbeans.modules.ruby.lexer.RubyTokenId$1: Language.maxOrdinal()=58 > 64 at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:267) at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingContainerImpl(IncTokenList.java:142) at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingContainer(IncTokenList.java:137) at org.netbeans.api.lexer.TokenSequence.moveNext(TokenSequence.java:377) Checking in spi/lexer/TokenFactory.java; /cvs/lexer/src/org/netbeans/spi/lexer/TokenFactory.java,v <-- TokenFactory.java new revision: 1.8; previous revision: 1.7 done Checking in lib/lexer/LexerInputOperation.java; /cvs/lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java,v <-- LexerInputOperation.java new revision: 1.9; previous revision: 1.8 Reassigning back to ruby.
I've done a little typo "58 > 64" :) it's now fixed: Checking in LexerInputOperation.java; /cvs/lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java,v <-- LexerInputOperation.java new revision: 1.10; previous revision: 1.9 I've looked to RubyTokenId whether I could fix it quickly but even if I would add the NONUNARY_OP to getUsedTokens() there are still e.g. COLON3 that can IMHO be created (returned by RubyLexer.getTokenId() which is called by RubyLexer.nextToken()). Please note that from now on all the token creations where the token ids would not be contained in the language will be reported. So I was a bit scared and checked opening of java files :) and I'm also going to tell Hanz to update and check Schliemann.
*** Issue 113180 has been marked as a duplicate of this issue. ***
Ugh... can you disable the check until we fix this, or perform the check only when assertions are enabled? This is causing a massive regression at the moment, even if the bug is really in Ruby since this check wasn't performed before. I wish I could fix this immediately but I have to head in to an all-day meeting right now.
*** Issue 113171 has been marked as a duplicate of this issue. ***
I took a quick look at the revision history for RubyTokenId and attempted a hotfix - but I don't have time to check it yet. On the other hand it can't make matters worse so checking in and hitting the road. IDE:------------------------------------------------- IDE: [8/20/07 7:41 AM] Committing started Checking in RubyTokenId.java; /cvs/ruby/editing/src/org/netbeans/modules/ruby/lexer/RubyTokenId.java,v <-- RubyTokenId.java new revision: 1.4; previous revision: 1.3 done IDE: [8/20/07 7:41 AM] Committing finished
Was still failing, now hopefully ok, ruby/editing tests pass, Tor please double check and close. Thanks. Checking in RubyTokenId.java; /cvs/ruby/editing/src/org/netbeans/modules/ruby/lexer/RubyTokenId.java,v <-- RubyTokenId.java new revision: 1.5; previous revision: 1.4
Fantastic, thank you very much. I'll test it myself tonight. By the way, I had a lot of unit tests for the lexer but they weren't failing before this fix; is there a way to trigger the lexer (such as the batch token dump) to check the token lists during lexing (the way that it's done during painting)? I'd like to have the unit tests catch any future accidental omissions from the token list.
This should be fixed now. Mila, I've updated the lexer TokenDumpCheck class to check that all the tokens it encounters during the lex check are also specified in the language token set. This should make it easier to avoid accidentally missing (rare) tokens in the future. Before this, all my lexing tests were passing. I'll attach the patch (it's trivial).
Created attachment 46918 [details] Patch to lexer unit test infrastructure to catch similar problems in the future
*** Issue 113271 has been marked as a duplicate of this issue. ***
Tor, apologies for this I did not realize that this was the first thing actively using the tokenids' ordinals. The ordinals are beneficial since they allow to speed up searches for token-id related associations (arrays can be used instead of regular maps) i.e. to have an array SomeClass[lang.maxOrdinal() + 1] and do SomeClass[id.ordinal()]. Basically the same thing like EnumSet or EnumMap. I think that the patch for TokenDumpCheck is now superfluous since it's checking the tokens from a token sequence but those tokens were already created through the LexerInputOperation (just a single place where Lexer.nextToken() gets called) so it would fail even earlier with the ISE from the check that I've added yesterday. Generally I don't see any way how to prevent the situation with missing tokenids in the language completely since I would have to force the lexer to create all the possible tokens that its code can produce possibly by providing some sort of random text inputs but it's generally hard. Or the other possibility could be to introspect the lexer class and check "xxxLexer implements Lexer<xxxTokenId>" and search for all the occurrences of "xxxTokenId" or better "instanceof xxxTokenId".