Bug 91184 - Remove incomplete token ids by providing a PartType enum
Remove incomplete token ids by providing a PartType enum
Status: RESOLVED FIXED
Product: editor
Classification: Unclassified
Component: Lexer
6.x
PC Linux
: P2 (vote)
: 6.x
Assigned To: Miloslav Metelka
issues@editor
: API_REVIEW_FAST
Depends on:
Blocks: 87014
  Show dependency treegraph
 
Reported: 2006-12-18 10:35 UTC by Miloslav Metelka
Modified: 2007-03-23 13:08 UTC (History)
4 users (show)

See Also:
Issue Type: TASK
:


Attachments
Diff of the proposed changes (66.49 KB, patch)
2007-03-03 20:01 UTC, Miloslav Metelka
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Miloslav Metelka 2006-12-18 10:35:45 UTC
This is a request to remove incomplete token ids (e.g.
JavaTokenId.BLOCK_COMMENT_INCOMPLETE) that clutter the language definition.
Instead there would be an enum TokenPartType { COMPLETE, START, INNER, END }.
Token.partType() would return a non-null enum value. For the most typical
situation when the token is complete the COMPLETE would be returned. For the
present incomplete token ids the START would be used. The INNER and END would
cover the requirements of issue 87014.
The TokenFactory would be enhanced accordingly to allow the creation of the
token parts.
Comment 1 Miloslav Metelka 2007-03-03 19:59:58 UTC
I have added PartType class and modified JavaTokenId to no longer contain
*_INCOMPLETE tokens and updated the depending java code accordingly.
I was selecting between
 PartType { COMPLETE, START, INNER, END }
and 
 PartType { START, INNER, END } and <null> meaning complete token
I have chosen the first solution.
Marking for fasttrack review, diff attached.
Comment 2 Miloslav Metelka 2007-03-03 20:01:05 UTC
Created attachment 39153 [details]
Diff of the proposed changes
Comment 3 Vitezslav Stejskal 2007-03-04 22:39:08 UTC
The apichanges.xml is talking about PartInfo and Token.partInfo while the added
enum is PartType and method Token.partType.
Comment 4 Miloslav Metelka 2007-03-05 08:45:31 UTC
Oops, fixed. Thanks, Vita for finding this.
Comment 5 Marek Fukala 2007-03-05 12:36:12 UTC
IMO the Token.partType() mechanism is generally OK and seems to cover my
usecases so I agree with commiting to trunk. 

I have just two comments:

1) I do not see a reasonable way how to implement the Lexer to recognize that
the created token should be PartType.END or INNER since there is no info that it
previously returned START token. More context information is needed here. OTOH
even if the context was wider and containing the info about previously returned
token, how I would recognize that I should return INNER instead of END? I would
have to somehow look forward and guess that there will be another token breaking
the current one.

2) How the PartType connected with the embedding? More concrete, will the
embedding on one token part be applied to the rest of the token parts? Or do I
have to handle this manually in the implementation?

3) Just an idea - it would be probably handy to have a TokenSequence method like 
TokenSequence ts = ...;
ts.nextWholeToken();

so the clients of the Lexer API doesn't have to (if they explicitly do not want
to) even know about the token parts.


Comment 6 Miloslav Metelka 2007-03-05 13:40:54 UTC
Yes, the 1) will be solved as part of issue 87014.
Regarding INNER versus END: you would return END only if the token would be
complete e.g. "/** ..." would have to be closed by "*/" otherwise you would
return INNER.
ad 2) and 3) again this should be finalized as part of issue 87014. My current
assumption is that there would be a method to return the complete token but the
embedding would only be done on the individual sections so that the whole token
hierarchy remains to be a tree of tokens in terms of coverage of the lexed text.
Comment 7 Jan Jancura 2007-03-05 14:16:17 UTC
Looks good.
Comment 8 Miloslav Metelka 2007-03-05 15:20:17 UTC
Fixed:
Checking in java/editor/nbproject/project.xml;
/cvs/java/editor/nbproject/project.xml,v  <--  project.xml
new revision: 1.25; previous revision: 1.24
done
Checking in java/editor/src/org/netbeans/modules/editor/java/Utilities.java;
/cvs/java/editor/src/org/netbeans/modules/editor/java/Utilities.java,v  <-- 
Utilities.java
new revision: 1.18; previous revision: 1.17
done
Checking in java/lexer/manifest.mf;
/cvs/java/lexer/manifest.mf,v  <--  manifest.mf
new revision: 1.5; previous revision: 1.4
done
Checking in java/lexer/nbproject/project.xml;
/cvs/java/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.5; previous revision: 1.4
done
Checking in java/lexer/src/org/netbeans/api/java/lexer/JavaTokenId.java;
/cvs/java/lexer/src/org/netbeans/api/java/lexer/JavaTokenId.java,v  <-- 
JavaTokenId.java
new revision: 1.8; previous revision: 1.7
done
Checking in java/lexer/src/org/netbeans/lib/java/lexer/JavaLexer.java;
/cvs/java/lexer/src/org/netbeans/lib/java/lexer/JavaLexer.java,v  <-- 
JavaLexer.java
new revision: 1.5; previous revision: 1.4
done
Checking in java/lexer/test/unit/data/testfiles/testInput.java.txt.tokens.txt;
/cvs/java/lexer/test/unit/data/testfiles/testInput.java.txt.tokens.txt,v  <-- 
testInput.java.txt.tokens.txt
new revision: 1.2; previous revision: 1.1
done
Checking in
java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLanguageTest.java;
/cvs/java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLanguageTest.java,v
 <--  JavaLanguageTest.java
new revision: 1.3; previous revision: 1.2
done
Checking in
java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLexerBatchTest.java;
/cvs/java/lexer/test/unit/src/org/netbeans/lib/java/lexer/JavaLexerBatchTest.java,v
 <--  JavaLexerBatchTest.java
new revision: 1.3; previous revision: 1.2
done
Checking in
java/source/test/unit/src/org/netbeans/api/java/source/gen/WhitespaceIgnoringDiff.java;
/cvs/java/source/test/unit/src/org/netbeans/api/java/source/gen/WhitespaceIgnoringDiff.java,v
 <--  WhitespaceIgnoringDiff.java
new revision: 1.4; previous revision: 1.3
done
Checking in javadoc/nbproject/project.xml;
/cvs/javadoc/nbproject/project.xml,v  <--  project.xml
new revision: 1.19; previous revision: 1.18
done
Checking in javadoc/src/org/netbeans/modules/javadoc/hints/JavadocUtilities.java;
/cvs/javadoc/src/org/netbeans/modules/javadoc/hints/JavadocUtilities.java,v  <--
 JavadocUtilities.java
new revision: 1.3; previous revision: 1.2
done
Checking in languages/engine/src/org/netbeans/modules/languages/lexer/SLexer.java;
/cvs/languages/engine/src/org/netbeans/modules/languages/lexer/SLexer.java,v 
<--  SLexer.java
new revision: 1.16; previous revision: 1.15
done
Checking in lexer/manifest.mf;
/cvs/lexer/manifest.mf,v  <--  manifest.mf
new revision: 1.15; previous revision: 1.14
done
Checking in lexer/api/apichanges.xml;
/cvs/lexer/api/apichanges.xml,v  <--  apichanges.xml
new revision: 1.13; previous revision: 1.12
done
RCS file: /cvs/lexer/src/org/netbeans/api/lexer/PartType.java,v
done
Checking in lexer/src/org/netbeans/api/lexer/PartType.java;
/cvs/lexer/src/org/netbeans/api/lexer/PartType.java,v  <--  PartType.java
initial revision: 1.1
done
Checking in lexer/src/org/netbeans/api/lexer/Token.java;
/cvs/lexer/src/org/netbeans/api/lexer/Token.java,v  <--  Token.java
new revision: 1.8; previous revision: 1.7
done
Checking in lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java;
/cvs/lexer/src/org/netbeans/lib/lexer/LexerInputOperation.java,v  <-- 
LexerInputOperation.java
new revision: 1.7; previous revision: 1.6
done
Checking in lexer/src/org/netbeans/lib/lexer/LexerUtilsConstants.java;
/cvs/lexer/src/org/netbeans/lib/lexer/LexerUtilsConstants.java,v  <-- 
LexerUtilsConstants.java
new revision: 1.8; previous revision: 1.7
done
Checking in lexer/src/org/netbeans/lib/lexer/token/AbstractToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/AbstractToken.java,v  <-- 
AbstractToken.java
new revision: 1.5; previous revision: 1.4
done
RCS file: /cvs/lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java,v
done
Checking in lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/ComplexToken.java,v  <-- 
ComplexToken.java
initial revision: 1.1
done
Checking in lexer/src/org/netbeans/lib/lexer/token/CustomTextToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/CustomTextToken.java,v  <-- 
CustomTextToken.java
new revision: 1.5; previous revision: 1.4
done
Checking in lexer/src/org/netbeans/lib/lexer/token/DefaultToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/DefaultToken.java,v  <-- 
DefaultToken.java
new revision: 1.6; previous revision: 1.5
done
Checking in lexer/src/org/netbeans/lib/lexer/token/PreprocessedTextToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/PreprocessedTextToken.java,v  <-- 
PreprocessedTextToken.java
new revision: 1.4; previous revision: 1.3
done
Removing lexer/src/org/netbeans/lib/lexer/token/PropertyCustomTextToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyCustomTextToken.java,v  <--
 PropertyCustomTextToken.java
new revision: delete; previous revision: 1.3
done
Removing lexer/src/org/netbeans/lib/lexer/token/PropertyPreprocessedTextToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyPreprocessedTextToken.java,v
 <--  PropertyPreprocessedTextToken.java
new revision: delete; previous revision: 1.3
done
Checking in lexer/src/org/netbeans/lib/lexer/token/PropertyToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/PropertyToken.java,v  <-- 
PropertyToken.java
new revision: 1.4; previous revision: 1.3
done
Checking in lexer/src/org/netbeans/lib/lexer/token/TextToken.java;
/cvs/lexer/src/org/netbeans/lib/lexer/token/TextToken.java,v  <--  TextToken.java
new revision: 1.5; previous revision: 1.4
done
Checking in lexer/src/org/netbeans/spi/lexer/TokenFactory.java;
/cvs/lexer/src/org/netbeans/spi/lexer/TokenFactory.java,v  <--  TokenFactory.java
new revision: 1.7; previous revision: 1.6
done
Checking in lexer/src/org/netbeans/spi/lexer/TokenPropertyProvider.java;
/cvs/lexer/src/org/netbeans/spi/lexer/TokenPropertyProvider.java,v  <-- 
TokenPropertyProvider.java
new revision: 1.6; previous revision: 1.5
done
Checking in lexer/test/unit/src/org/netbeans/api/lexer/CustomTokenClassTest.java;
/cvs/lexer/test/unit/src/org/netbeans/api/lexer/CustomTokenClassTest.java,v  <--
 CustomTokenClassTest.java
new revision: 1.4; previous revision: 1.3
done
Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/LexerTestUtilities.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/LexerTestUtilities.java,v 
<--  LexerTestUtilities.java
new revision: 1.9; previous revision: 1.8
done
Checking in
lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpCheck.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpCheck.java,v
 <--  TokenDumpCheck.java
new revision: 1.4; previous revision: 1.3
done
Checking in
lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpLexer.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/dump/TokenDumpLexer.java,v
 <--  TokenDumpLexer.java
new revision: 1.4; previous revision: 1.3
done
Checking in web/jspsyntax/lexer/nbproject/project.xml;
/cvs/web/jspsyntax/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.8; previous revision: 1.7
done
Checking in web/jspsyntax/lexer/src/org/netbeans/lib/jsp/lexer/JspLexer.java;
/cvs/web/jspsyntax/lexer/src/org/netbeans/lib/jsp/lexer/JspLexer.java,v  <-- 
JspLexer.java
new revision: 1.21; previous revision: 1.20
Comment 9 Vitezslav Stejskal 2007-03-05 15:50:49 UTC
Just a note: current syntax highlighting ignores PartTypes and there is no way
to define a different coloring for particular parts of a token, which in some
sense used to be possible with *_INCOMPLETE tokens, but had not been used AFAIK.
Do we need this (1) or shall we leave it for later (2) when and if we really
need it? As an implication of #2 any colorings registered in the layer files for
incomplete tokens (that have now been removed) should also be deleted. Don't
forget about the CityLights profile's coloring files.
Comment 10 Miloslav Metelka 2007-03-05 16:12:32 UTC
Yes, I was thinking about this before implementation and I assumed that the
coloring for the particular incomplete token-ids will usually be the same like
the coloring for their corresponding complete counterpart.
I have also thought that we could eventually make a generic algorithm that would
inspect the incomplete tokens and make e.g. a composition of their color with
red color to point out that their are incomplete (or it could e.g. make a curly
underline). But this needs to be done after completion of issue 87014 because
some of these incomplete tokens will be result of lexing multiple embedded
sections at once.
I'll fix the colorings.
Comment 11 Miloslav Metelka 2007-03-23 13:08:12 UTC
I have checked the colorings and they are only per-category e.g. for "string" so
they are unafected by this change.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo