Bug 120052 - Lexer API Cleanup
Lexer API Cleanup
Status: RESOLVED FIXED
Product: editor
Classification: Unclassified
Component: Lexer
6.x
PC All
: P2 (vote)
: 6.x
Assigned To: Miloslav Metelka
issues@editor
: API, API_REVIEW_FAST
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-10-25 15:26 UTC by Miloslav Metelka
Modified: 2007-11-06 08:27 UTC (History)
4 users (show)

See Also:
Issue Type: TASK
:


Attachments
Committed files log (31.19 KB, text/plain)
2007-11-05 17:10 UTC, Miloslav Metelka
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Miloslav Metelka 2007-10-25 15:26:41 UTC
NB6.0 will be the first release where the new Lexer API will be published as an official API. There are several things
that I would like to fix before the release:

1) Remove Token Hierarchy Snapshots support (TokenHierarchy.createSnapshot() etc.) since currently there is no client
actively using them. Moreover after fixing of issue 87014 the present snapshot support is broken. Snapshot creation and
maintaining design cannot work in its current form after that fix (i.e. cheap snapshot creation that shares token
instances with the original hierarchy) - it would be overly complex. Also the current design still delegates to the
"live chars storage" (e.g. Document) for unchanged tokens which is unpractical in many cases since it still requires
document read-locking.
 I need to revisit the whole idea and find a better solution in a next release. It should be possible to add the
snapshot support later compatibly.

2) Remove Token.preprocessedText() and basically all four Token.*preprocess*() methods and CharPreprocessor. Present
JavaLexer impl does not support proper recognizing of Unicode escape sequences yet anyway and IMO the four methods
pollute the Token class too much. Instead of a generic support I would like to create direct support classes for Unicode
escapes in a next release and use Token.getProperty() for getting information whether the token has unicode escapes in
it and get the preprocessed text. This should be a compatible change.

3) Generics use: TokenHierarchy.mutableInputSource() => TokenHierarchy.inputSource() and TH.create() methods for
non-mutable sources would not return TokenHierarchy<Void> but TokenHierarchy<I> normally. I have no strong reason for
not returning immutable input sources and this way the code and signatures should be more readable.

  Also there are several places with <? extends TokenId> e.g.
      protected abstract Language<? extends TokenId> language();
  in fact it can be simplified to
      protected abstract Language<?> language();
  because
      public final class Language<T extends TokenId> { ... }

4) Add MutableInputSource.checkRead/WriteLockAcquired() to improve checking whether some critical methods are called
under the appropriate locks.
  The checking would be done when an extra logging is turned on.

5) TokenHierarchy.get(Document) may return null now if the token hierarchy is unavailable for the given document at the
time (e.g. there is no appropriate language). I would rather always return non-null TH instance and return null from
tokenSequence() instead in the described case. The difference would be that if the non-null TH would be returned then
clients could start listening for token hierarchy rebuilds and changes immediately which should simplify their logic.
Comment 1 Miloslav Metelka 2007-10-25 15:29:16 UTC
Asking for fastrack review.
Comment 2 Jan Jancura 2007-10-25 16:11:51 UTC
I agree with all changes.
Comment 3 Marek Fukala 2007-10-25 18:06:24 UTC
I am OK with the changes.
Comment 4 Vitezslav Stejskal 2007-10-26 13:56:36 UTC
I'm fine with the changes. For #5 you will have to go through all the code using TokenHierarchy.get(Document) and fix
the cases that check for null. For example this is used in highlighting layer factories to check if a language has
already been lexerified and new SyntaxHighlighting layer can be used or if we need NonLexerBasedSyntaxHighlighting layer
that works with old o.n.editor.Syntax, etc. There might be other places using it in a similar way. Maybe it needs to be
replaced by Language.find(document.getProperty("mimeType")).

6. Speaking of Language.find(String mimePath), as Hanz pointed out this method currently only works for top level mime
types. In fact it makes no sense to use it for mimePath and we should change the parameter's name and update its
javadoc. The same for LanguageProvider.findLanguage() that it delegates to. I can do that if you want.
Comment 5 Torbjorn Norbye 2007-10-26 20:07:36 UTC
Looks fine to me. Please update this issue when you integrate so I can test it a lot once the change (only #5 affects
me) goes in.
Comment 6 Vitezslav Stejskal 2007-11-02 19:50:38 UTC
#6: It's done. Language.find(String) and LanguageProvider.findLanguage(String) can only be used for top level mime types.

Checking in lexer/test/unit/src/org/netbeans/lib/lexer/LanguageManagerTest.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/LanguageManagerTest.java,v  <--  LanguageManagerTest.java
new revision: 1.11; previous revision: 1.10
done
Checking in lexer/nbbridge/test/build.xml;
/cvs/lexer/nbbridge/test/build.xml,v  <--  build.xml
new revision: 1.5; previous revision: 1.4
done
Checking in lexer/src/org/netbeans/lib/lexer/LanguageManager.java;
/cvs/lexer/src/org/netbeans/lib/lexer/LanguageManager.java,v  <--  LanguageManager.java
new revision: 1.13; previous revision: 1.12
done
Checking in languages/engine/src/org/netbeans/modules/languages/lexer/SLanguageProvider.java;
/cvs/languages/engine/src/org/netbeans/modules/languages/lexer/SLanguageProvider.java,v  <--  SLanguageProvider.java
new revision: 1.19; previous revision: 1.18
done
Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLanguageProvider.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLanguageProvider.java,v  <--  SimpleLanguageProvider.java
new revision: 1.13; previous revision: 1.12
done
Checking in lexer/src/org/netbeans/spi/lexer/LanguageProvider.java;
/cvs/lexer/src/org/netbeans/spi/lexer/LanguageProvider.java,v  <--  LanguageProvider.java
new revision: 1.10; previous revision: 1.9
done
Checking in lexer/nbbridge/src/org/netbeans/modules/lexer/nbbridge/MimeLookupLanguageProvider.java;
/cvs/lexer/nbbridge/src/org/netbeans/modules/lexer/nbbridge/MimeLookupLanguageProvider.java,v  <-- 
MimeLookupLanguageProvider.java
new revision: 1.9; previous revision: 1.8
done
Checking in lexer/src/org/netbeans/api/lexer/Language.java;
/cvs/lexer/src/org/netbeans/api/lexer/Language.java,v  <--  Language.java
new revision: 1.13; previous revision: 1.12
done
Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/TestLanguageProvider.java;
/cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/TestLanguageProvider.java,v  <--  TestLanguageProvider.java
new revision: 1.3; previous revision: 1.2
done
Comment 7 Miloslav Metelka 2007-11-05 17:10:38 UTC
Created attachment 52541 [details]
Committed files log
Comment 8 Miloslav Metelka 2007-11-05 17:11:42 UTC
I will also increase the spec. version and update apichanges.xml.
Comment 9 Miloslav Metelka 2007-11-05 18:18:45 UTC
Fixed apichanges, lexer javadocs, fixed dependency versions.
Checking in lexer/src/org/netbeans/spi/lexer/TokenFactory.java;
/cvs/lexer/src/org/netbeans/spi/lexer/TokenFactory.java,v  <--  TokenFactory.java
new revision: 1.11; previous revision: 1.10
done
Checking in lexer/src/org/netbeans/spi/lexer/MutableTextInput.java;
/cvs/lexer/src/org/netbeans/spi/lexer/MutableTextInput.java,v  <--  MutableTextInput.java
new revision: 1.8; previous revision: 1.7
done
Checking in lexer/nbproject/project.properties;
/cvs/lexer/nbproject/project.properties,v  <--  project.properties
new revision: 1.19; previous revision: 1.18
done
Checking in web/el/lexer/nbproject/project.xml;
/cvs/web/el/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.4; previous revision: 1.3
done
Checking in editor/lib/nbproject/project.xml;
/cvs/editor/lib/nbproject/project.xml,v  <--  project.xml
new revision: 1.15; previous revision: 1.14
done
Checking in editor/bracesmatching/nbproject/project.xml;
/cvs/editor/bracesmatching/nbproject/project.xml,v  <--  project.xml
new revision: 1.2; previous revision: 1.1
done
Checking in lexer/arch.xml;
/cvs/lexer/arch.xml,v  <--  arch.xml
new revision: 1.13; previous revision: 1.12
done
Checking in contrib/EditorBracesMatching/nbproject/project.xml;
/cvs/contrib/EditorBracesMatching/nbproject/project.xml,v  <--  project.xml
new revision: 1.12; previous revision: 1.11
done
Checking in lexer/src/org/netbeans/api/lexer/Token.java;
/cvs/lexer/src/org/netbeans/api/lexer/Token.java,v  <--  Token.java
new revision: 1.11; previous revision: 1.10
done
Checking in xml/tageditorsupport/nbproject/project.xml;
/cvs/xml/tageditorsupport/nbproject/project.xml,v  <--  project.xml
new revision: 1.15; previous revision: 1.14
done
Checking in ruby/gsf/nbproject/project.xml;
/cvs/ruby/gsf/nbproject/project.xml,v  <--  project.xml
new revision: 1.13; previous revision: 1.12
done
Checking in editor/lib2/nbproject/project.xml;
/cvs/editor/lib2/nbproject/project.xml,v  <--  project.xml
new revision: 1.6; previous revision: 1.5
done
Checking in java/editor/nbproject/project.xml;
/cvs/java/editor/nbproject/project.xml,v  <--  project.xml
new revision: 1.45; previous revision: 1.44
done
Checking in editor/options/nbproject/project.xml;
/cvs/editor/options/nbproject/project.xml,v  <--  project.xml
new revision: 1.28; previous revision: 1.27
done
Checking in html/editor/nbproject/project.xml;
/cvs/html/editor/nbproject/project.xml,v  <--  project.xml
new revision: 1.29; previous revision: 1.28
done
Checking in scripting/php/lexer/nbproject/project.xml;
/cvs/scripting/php/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.4; previous revision: 1.3
done
Checking in xml/lexer/nbproject/project.xml;
/cvs/xml/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.3; previous revision: 1.2
done
Checking in ruby/editing/nbproject/project.xml;
/cvs/ruby/editing/nbproject/project.xml,v  <--  project.xml
new revision: 1.20; previous revision: 1.19
done
Checking in javadoc/nbproject/project.xml;
/cvs/javadoc/nbproject/project.xml,v  <--  project.xml
new revision: 1.32; previous revision: 1.31
done
Checking in java/hints/nbproject/project.xml;
/cvs/java/hints/nbproject/project.xml,v  <--  project.xml
new revision: 1.38; previous revision: 1.37
done
Checking in java/source/nbproject/project.xml;
/cvs/java/source/nbproject/project.xml,v  <--  project.xml
new revision: 1.38; previous revision: 1.37
done
Checking in web/jspsyntax/nbproject/project.xml;
/cvs/web/jspsyntax/nbproject/project.xml,v  <--  project.xml
new revision: 1.41; previous revision: 1.40
done
Checking in lexer/api/apichanges.xml;
/cvs/lexer/api/apichanges.xml,v  <--  apichanges.xml
new revision: 1.25; previous revision: 1.24
done
Checking in ruby/refactoring/nbproject/project.xml;
/cvs/ruby/refactoring/nbproject/project.xml,v  <--  project.xml
new revision: 1.4; previous revision: 1.3
done
Checking in java/lexer/nbproject/project.xml;
/cvs/java/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.7; previous revision: 1.6
done
Checking in languages/engine/nbproject/project.xml;
/cvs/languages/engine/nbproject/project.xml,v  <--  project.xml
new revision: 1.22; previous revision: 1.21
done
Checking in html/lexer/nbproject/project.xml;
/cvs/html/lexer/nbproject/project.xml,v  <--  project.xml
new revision: 1.7; previous revision: 1.6
done
Checking in ruby/rhtml/nbproject/project.xml;
/cvs/ruby/rhtml/nbproject/project.xml,v  <--  project.xml
new revision: 1.16; previous revision: 1.15
done
Checking in contrib/JavadocFirstSentence/nbproject/project.xml;
/cvs/contrib/JavadocFirstSentence/nbproject/project.xml,v  <--  project.xml
new revision: 1.4; previous revision: 1.3
Comment 10 Jesse Glick 2007-11-05 23:09:51 UTC
You seem to have broken compilation of scripting/ejs, at least. Please check for other users of this API and make sure
they are still compilable.
Comment 11 Miloslav Metelka 2007-11-06 08:27:51 UTC
Apologies, I did a clean build of the full cluster config and I have also attempted to fix tests of the affected
modules. But I did not know about the javac bugs that will be revealed by incremental building.
I have fixed scripting/ejs.
Checking in EJSTokenId.java;
/cvs/scripting/ejs/src/org/netbeans/modules/languages/ejs/lexer/api/EJSTokenId.java,v  <--  EJSTokenId.java
new revision: 1.4; previous revision: 1.3


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo