This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 243651

Summary: IndexOutOfBoundsException: tokenLength=0 <= 0. Fix the lexer implementation to use proper token length value.
Product: editor Reporter: matiskov
Component: LexerAssignee: Miloslav Metelka <mmetelka>
Status: VERIFIED FIXED    
Severity: normal CC: Andrey1990, benp, brettryan, d.lisovskiy, lawrence615, mrugendrabhure, pmaselkowski
Priority: P2    
Version: 8.0   
Hardware: All   
OS: All   
Issue Type: DEFECT Exception Reporter: 206306
Attachments: stacktrace
stacktrace
stacktrace
File which can be used to 100% reproduce the problem.

Description matiskov 2014-04-08 10:52:56 UTC
Build: NetBeans IDE 8.0 (Build 201403101706)
VM: Java HotSpot(TM) 64-Bit Server VM, 24.45-b08, Java(TM) SE Runtime Environment, 1.7.0_45-b18
OS: Windows 8

User Comments:
GUEST: Editing attribute data-bind="click: createReview, disable: !createEnabled(), css: {'disabled': createEnabled() }
Wanted to add "!" before createEnabled()

GUEST: text editing

GUEST: Sometimes when I edit a smarty template with data-bind attributes, from knockoutjs framework, it throws IndexOutOfBoundsException and i can't edit the line with the data-bind

mrugendrabhure: Using Knockout.js / PHP, while trying to add a PHP tag in between, the UI locks up and error is thrown

<input data-bind="attr: {'onkeypress': 'return isNumber(event);', 'onkeyup': 'enableTooltip(this);',name: 'itr_submission[properties]['+$index()+'][rent_info][<?php echo $i+6 ?>' }" type="text" size="8">

I am trying to add a "]" after the <?php echo $i + 6;?>, the moment I try to do that,  error occurs.

matiskov: knockout data-bind dont work

GUEST: This is probably a Smarty plugin problem.
I opened a js object bracket and closed it, it was seen as smarty markup. After I tried to add a space inside the brackets some part of the text became uneditable.
Here is the problematic part:
" data-bind="template: {"

I can edit around it but not inside it

matiskov: knockout bind does not work




Stacktrace: 
java.lang.IndexOutOfBoundsException: tokenLength=0 <= 0. Fix the lexer implementation to use proper token length value.
   at org.netbeans.lib.lexer.LexerInputOperation.assignTokenLength(LexerInputOperation.java:405)
   at org.netbeans.lib.lexer.JoinLexerInputOperation.assignTokenLength(JoinLexerInputOperation.java:188)
   at org.netbeans.lib.lexer.LexerInputOperation.createToken(LexerInputOperation.java:315)
   at org.netbeans.spi.lexer.TokenFactory.createToken(TokenFactory.java:105)
   at org.netbeans.spi.lexer.TokenFactory.createToken(TokenFactory.java:89)
   at org.netbeans.modules.html.knockout.KODataBindLexer.nextToken(KODataBindLexer.java:196)
Comment 1 matiskov 2014-04-08 10:52:59 UTC
Created attachment 146623 [details]
stacktrace
Comment 2 brettryan 2016-05-30 23:42:55 UTC
Created attachment 159920 [details]
stacktrace

This is related to https://netbeans.org/bugzilla/show_bug.cgi?id=262276 but provides a different exception.

There's somethign not right with the KnockoutJS module.
Comment 3 Miloslav Metelka 2016-09-15 12:02:57 UTC
I'm not able to reproduce the problem. If someone is able to 100% reproduce in particular situation could you please attach a full source and describe at which line/column to type what character in order to get the exception? Thanks.

The exception basically means that KODataBindLexer attempts to produce a token of a zero length (token with zero characters) which the lexer infrastructure currently prohibits. KODataBindLexer implementation should be fixed to not produce zero-length tokens.
Comment 4 brettryan 2016-09-21 07:30:56 UTC
Created attachment 162146 [details]
stacktrace

Can not edit knockoutjs data-bind attributes in some circumstances.
Comment 5 brettryan 2016-09-21 07:37:13 UTC
Created attachment 162147 [details]
File which can be used to 100% reproduce the problem.

Here, have worked out how to 100% reproduce. I've trimmed a file down as much as I could figure out that causes the issue.

Steps to Reproduce
------------------

These have to be followed exactly, as the cause when activated will not go away but the cause is somewhat finicky to reproduce.

Open the file and look at line 21. Click the first bracket after "text: ". The caret may be either after or before the (.

Now type any letter to get the exception.

Note that now that you've activated the bug you can not type in any data-bind attribute anywhere within the file.
Comment 6 Miloslav Metelka 2016-09-27 15:12:51 UTC
(In reply to brettryan from comment #5)
> Created attachment 162147 [details]
> File which can be used to 100% reproduce the problem.
> 
Thanks a lot for these steps to reproduce!
After hours of logger reports searching and debugging :-) I've finally realized what was wrong.
It could possibly be a cause of issue #267354 too but it needs to be checked after this fix gets into nightly builds.
http://hg.netbeans.org/jet-main/rev/3214e3cfbf9f
Comment 7 Quality Engineering 2016-09-28 01:57:40 UTC
Integrated into 'main-silver', will be available in build *201609280002* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/3214e3cfbf9f
User: Miloslav Metelka <mmetelka@netbeans.org>
Log: #243651 - IndexOutOfBoundsException: tokenLength=0 <= 0. Fix the lexer implementation to use proper token length value.
Comment 8 Jiri Kovalsky 2016-09-29 10:53:58 UTC
This can be a candidate for NetBeans 8.2 Patch 1. Can some of the reporters please verify the bug fix in #201609280002 or later? Thanks!
Comment 9 brettryan 2016-10-03 01:17:23 UTC
(In reply to Jiri Kovalsky from comment #8)
> This can be a candidate for NetBeans 8.2 Patch 1. Can some of the reporters
> please verify the bug fix in #201609280002 or later? Thanks!

Confirmed exception has gone, excellent, thanks.

There is however one smaller regression where only the first data-bind attribute in the file has correct syntax colouring.

Take the following

  <select class="form-control" data-bind="options: opts1, optionsText: 'name', value: v1"></select>
  <select class="form-control" data-bind="options: opts2, optionsText: 'name', value: v2"></select>
  <select class="form-control" data-bind="options: opts3, optionsText: 'name', value: v3"></select>
  <select class="form-control" data-bind="options: opts4, optionsText: 'name', value: v4"></select>
  <select class="form-control" data-bind="options: opts5, optionsText: 'name', value: v5"></select>
  <select class="form-control" data-bind="options: opts6, optionsText: 'name', value: v6"></select>

The first select has `options' coloured where all other lines are black, removing the line (opts1) will now colour `opts2' correctly.
Comment 10 Miloslav Metelka 2016-10-03 14:44:26 UTC
(In reply to brettryan from comment #9)
> (In reply to Jiri Kovalsky from comment #8)
> > This can be a candidate for NetBeans 8.2 Patch 1. Can some of the reporters
> > please verify the bug fix in #201609280002 or later? Thanks!
> 
> Confirmed exception has gone, excellent, thanks.
> 
> There is however one smaller regression where only the first data-bind
> attribute in the file has correct syntax colouring.
> 
> Take the following
> 
>   <select class="form-control" data-bind="options: opts1, optionsText:
> 'name', value: v1"></select>
>   <select class="form-control" data-bind="options: opts2, optionsText:
> 'name', value: v2"></select>
>   <select class="form-control" data-bind="options: opts3, optionsText:
> 'name', value: v3"></select>
>   <select class="form-control" data-bind="options: opts4, optionsText:
> 'name', value: v4"></select>
>   <select class="form-control" data-bind="options: opts5, optionsText:
> 'name', value: v5"></select>
>   <select class="form-control" data-bind="options: opts6, optionsText:
> 'name', value: v6"></select>
> 
> The first select has `options' coloured where all other lines are black,
> removing the line (opts1) will now colour `opts2' correctly.

That's because the maintainer of "text/x-jsp/text/html/text/ko-data-bind" embedding requests the text sections to be joined and lexed together. The lexer framework just does what was requested by its client. With joined lexing the lexer sees input like this:
"options: opts1, optionsText: 'name', value: v1options: opts2, optionsText: 'name', value: v2options: opts3, optionsText: 'name', value: v3..."
and produces tokens according to the input.
IMHO it's inappropriate to use joined sections in this case so I've filed issue #268323.
Comment 11 Miloslav Metelka 2016-12-09 23:05:04 UTC
Transplanted into release82 as 18792d8b0768.
Comment 12 Jiri Prox 2016-12-13 11:01:53 UTC
verified in path