This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 101620 - Tokens embedding gets lost after modification of the top level token under some circumstances
Summary: Tokens embedding gets lost after modification of the top level token under so...
Status: VERIFIED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Lexer (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: Miloslav Metelka
URL:
Keywords:
: 102746 103088 (view as bug list)
Depends on:
Blocks: 100225
  Show dependency tree
 
Reported: 2007-04-18 16:42 UTC by Marek Fukala
Modified: 2007-05-02 09:20 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:

Attachments
The JSP file to reproduce (967 bytes, text/plain)
2007-04-18 16:43 UTC, Marek Fukala 		Details
NPE and AIOOBE stack trace (3.98 KB, text/plain)
2007-04-27 10:12 UTC, Jiri Skrivanek 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Marek Fukala 2007-04-18 16:42:18 UTC 
To reproduce open the attached JSP file, wait for a few seconds until the
javascript and css sections gets colored, and try to put some spaces inside the
javascript code on line 10. The javascript coloring gets lost just after the
modification, it is created again after a while when the parser finishes, but
the  coloring is broken meanwhile and code completion has also problem with this
behaviour.

From what I remembed and understood when we debugged the problem on my box, the
problem is in the bigger lookahead of JSPLexer in the jsp close tag than is the
token length which causes the jsp token being relexed and thus the
embedding-preserving code is doesn't work.

I also am attaching the debug messages:
<<<<<<<<<<<<<<<<<< LEXER CHANGE START ------------------
ROOT CHANGE: "text/x-jsp", index=11, offset=52
R[0]: E["text/x-jsp/text/html"] IHC(28887505): DefT(TEXT, <null-text>, 52, 147)
IHC(15149474)
R[1]: DefT(ENDTAG, <null-text>, 199, 13) IHC(6189862)
A[0]: DefT(TEXT, "\n    <p>\n        text\n    </p>\n    \n    <script
type="text/javascript">\n     function hello() {\n         abst     x \n     } 
   \n    </script>\n    \n", 52, 148) IHC(13229344)
A[1]: DefT(ENDTAG, "</jsp:useBean", 200, 13) IHC(13925107)
EVENT: modOffset=163, insertedLength=1, Affected(52,213)
>>>>>>>>>>>>>>>>>> LEXER CHANGE END (TokenHierarchy Check OK) ------------------
Comment 1 Marek Fukala 2007-04-18 16:43:05 UTC 
Created attachment 41184 [details]
The JSP file to reproduce
Comment 2 Miloslav Metelka 2007-04-24 11:18:47 UTC 
The lookahead of the big tokens spans the whole next token plus one character.
The incremental algorithm correctly relexes the subsequent token as well.
Consider a similar situation with a language having the following tokens:
"+" PLUS token
"-" MINUS token
"+-+" PMP token

When given an input "+--" it would be lexed as 
PLUS(lookahead=2)
MINUS(lookahead=1)
MINUS(lookahead=0)

Now if initial "+" would get removed i.e. the input would be "--" then if only
the PLUS token would get removed then it would become
MINUS(lookahead=1)
MINUS(lookahead=0)

However by batch scanning the list would be
MINUS(lookahead=0)
MINUS(lookahead=0)

Allowing the lookaheads of the incremental setup become different from the batch
lexing setup depending on the order of modifications and their place is
dangerous since the correctness of incremental modifications could not be easily
verified and therefore the present algorithm prohibits that by relexing an extra
token till the end of the modification token's lookahead.
Therefore the desired fix is to check that the second relexed token is the same
like the original one and only report the change of the first token.
I will fix it ASAP.
Comment 3 Martin Schovanek 2007-04-24 15:04:29 UTC 
It blocks high visible P1 issue #100225, increasing priority to P1.
Comment 4 Petr Jiricka 2007-04-25 10:25:29 UTC 
Really -> P1
Comment 5 Martin Schovanek 2007-04-26 15:13:41 UTC 
After discussion with Pavel B. we agreed that is not necessary to fix the issue
#100225 for M9, so decreasing this back to P2.
Comment 6 Miloslav Metelka 2007-04-26 17:00:07 UTC 
As you like, but the fix should be safe.
The reproducing case should be fixed and when running with
-J-Dorg.netbeans.lib.lexer.TokenHierarchyOperation.level=FINE
also shows just one removed and added token. I have added a test as well.

Checking in src/org/netbeans/lib/lexer/inc/TokenListUpdater.java;
/cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListUpdater.java,v  <-- 
TokenListUpdater.java
new revision: 1.10; previous revision: 1.9
done
Checking in src/org/netbeans/lib/lexer/inc/TokenListChange.java;
/cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListChange.java,v  <-- 
TokenListChange.java
new revision: 1.9; previous revision: 1.8
done
Checking in src/org/netbeans/lib/lexer/LAState.java;
/cvs/lexer/src/org/netbeans/lib/lexer/LAState.java,v  <--  LAState.java
new revision: 1.4; previous revision: 1.3
done
RCS file:
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java,v
done
Checking in
test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java,v
 <--  TokenListUpdaterExtraTest.java
initial revision: 1.1
done
RCS file:
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java,v
done
Checking in
test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java,v
 <--  TestSaveTokensInLATokenId.java
initial revision: 1.1
Comment 7 Miloslav Metelka 2007-04-26 19:53:31 UTC 
Sorry, my browser did not refresh the priority field correctly so changing back
to P2.
Comment 8 Jiri Skrivanek 2007-04-27 10:11:15 UTC 
Reopening because lexer throws NPEs and AIOOBEs while testing JSF or Struts
support in web applications (see attached stack traces).
Comment 9 Jiri Skrivanek 2007-04-27 10:12:33 UTC 
Created attachment 41811 [details]
NPE and AIOOBE stack trace
Comment 10 Miloslav Metelka 2007-04-27 12:44:49 UTC 
Sorry, I did not realize that there can be removed token(s) without any added
ones e.g. if a sinle-char removed token had no lookahead. Fixed.

Checking in TokenListUpdater.java;
/cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListUpdater.java,v  <-- 
TokenListUpdater.java
new revision: 1.11; previous revision: 1.10
Comment 11 Miloslav Metelka 2007-04-27 16:13:30 UTC 
*** Issue 102746 has been marked as a duplicate of this issue. ***
Comment 12 Jiri Skrivanek 2007-04-27 16:40:11 UTC 
OK, tests are passing without exceptions since build 20070427-1216.
Comment 13 Marek Fukala 2007-05-02 09:20:17 UTC 
*** Issue 103088 has been marked as a duplicate of this issue. ***