This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
NB6.0 will be the first release where the new Lexer API will be published as an official API. There are several things that I would like to fix before the release: 1) Remove Token Hierarchy Snapshots support (TokenHierarchy.createSnapshot() etc.) since currently there is no client actively using them. Moreover after fixing of issue 87014 the present snapshot support is broken. Snapshot creation and maintaining design cannot work in its current form after that fix (i.e. cheap snapshot creation that shares token instances with the original hierarchy) - it would be overly complex. Also the current design still delegates to the "live chars storage" (e.g. Document) for unchanged tokens which is unpractical in many cases since it still requires document read-locking. I need to revisit the whole idea and find a better solution in a next release. It should be possible to add the snapshot support later compatibly. 2) Remove Token.preprocessedText() and basically all four Token.*preprocess*() methods and CharPreprocessor. Present JavaLexer impl does not support proper recognizing of Unicode escape sequences yet anyway and IMO the four methods pollute the Token class too much. Instead of a generic support I would like to create direct support classes for Unicode escapes in a next release and use Token.getProperty() for getting information whether the token has unicode escapes in it and get the preprocessed text. This should be a compatible change. 3) Generics use: TokenHierarchy.mutableInputSource() => TokenHierarchy.inputSource() and TH.create() methods for non-mutable sources would not return TokenHierarchy<Void> but TokenHierarchy<I> normally. I have no strong reason for not returning immutable input sources and this way the code and signatures should be more readable. Also there are several places with <? extends TokenId> e.g. protected abstract Language<? extends TokenId> language(); in fact it can be simplified to protected abstract Language<?> language(); because public final class Language<T extends TokenId> { ... } 4) Add MutableInputSource.checkRead/WriteLockAcquired() to improve checking whether some critical methods are called under the appropriate locks. The checking would be done when an extra logging is turned on. 5) TokenHierarchy.get(Document) may return null now if the token hierarchy is unavailable for the given document at the time (e.g. there is no appropriate language). I would rather always return non-null TH instance and return null from tokenSequence() instead in the described case. The difference would be that if the non-null TH would be returned then clients could start listening for token hierarchy rebuilds and changes immediately which should simplify their logic.
Asking for fastrack review.
I agree with all changes.
I am OK with the changes.
I'm fine with the changes. For #5 you will have to go through all the code using TokenHierarchy.get(Document) and fix the cases that check for null. For example this is used in highlighting layer factories to check if a language has already been lexerified and new SyntaxHighlighting layer can be used or if we need NonLexerBasedSyntaxHighlighting layer that works with old o.n.editor.Syntax, etc. There might be other places using it in a similar way. Maybe it needs to be replaced by Language.find(document.getProperty("mimeType")). 6. Speaking of Language.find(String mimePath), as Hanz pointed out this method currently only works for top level mime types. In fact it makes no sense to use it for mimePath and we should change the parameter's name and update its javadoc. The same for LanguageProvider.findLanguage() that it delegates to. I can do that if you want.
Looks fine to me. Please update this issue when you integrate so I can test it a lot once the change (only #5 affects me) goes in.
#6: It's done. Language.find(String) and LanguageProvider.findLanguage(String) can only be used for top level mime types. Checking in lexer/test/unit/src/org/netbeans/lib/lexer/LanguageManagerTest.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/LanguageManagerTest.java,v <-- LanguageManagerTest.java new revision: 1.11; previous revision: 1.10 done Checking in lexer/nbbridge/test/build.xml; /cvs/lexer/nbbridge/test/build.xml,v <-- build.xml new revision: 1.5; previous revision: 1.4 done Checking in lexer/src/org/netbeans/lib/lexer/LanguageManager.java; /cvs/lexer/src/org/netbeans/lib/lexer/LanguageManager.java,v <-- LanguageManager.java new revision: 1.13; previous revision: 1.12 done Checking in languages/engine/src/org/netbeans/modules/languages/lexer/SLanguageProvider.java; /cvs/languages/engine/src/org/netbeans/modules/languages/lexer/SLanguageProvider.java,v <-- SLanguageProvider.java new revision: 1.19; previous revision: 1.18 done Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLanguageProvider.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLanguageProvider.java,v <-- SimpleLanguageProvider.java new revision: 1.13; previous revision: 1.12 done Checking in lexer/src/org/netbeans/spi/lexer/LanguageProvider.java; /cvs/lexer/src/org/netbeans/spi/lexer/LanguageProvider.java,v <-- LanguageProvider.java new revision: 1.10; previous revision: 1.9 done Checking in lexer/nbbridge/src/org/netbeans/modules/lexer/nbbridge/MimeLookupLanguageProvider.java; /cvs/lexer/nbbridge/src/org/netbeans/modules/lexer/nbbridge/MimeLookupLanguageProvider.java,v <-- MimeLookupLanguageProvider.java new revision: 1.9; previous revision: 1.8 done Checking in lexer/src/org/netbeans/api/lexer/Language.java; /cvs/lexer/src/org/netbeans/api/lexer/Language.java,v <-- Language.java new revision: 1.13; previous revision: 1.12 done Checking in lexer/test/unit/src/org/netbeans/lib/lexer/test/TestLanguageProvider.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/TestLanguageProvider.java,v <-- TestLanguageProvider.java new revision: 1.3; previous revision: 1.2 done
Created attachment 52541 [details] Committed files log
I will also increase the spec. version and update apichanges.xml.
Fixed apichanges, lexer javadocs, fixed dependency versions. Checking in lexer/src/org/netbeans/spi/lexer/TokenFactory.java; /cvs/lexer/src/org/netbeans/spi/lexer/TokenFactory.java,v <-- TokenFactory.java new revision: 1.11; previous revision: 1.10 done Checking in lexer/src/org/netbeans/spi/lexer/MutableTextInput.java; /cvs/lexer/src/org/netbeans/spi/lexer/MutableTextInput.java,v <-- MutableTextInput.java new revision: 1.8; previous revision: 1.7 done Checking in lexer/nbproject/project.properties; /cvs/lexer/nbproject/project.properties,v <-- project.properties new revision: 1.19; previous revision: 1.18 done Checking in web/el/lexer/nbproject/project.xml; /cvs/web/el/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.4; previous revision: 1.3 done Checking in editor/lib/nbproject/project.xml; /cvs/editor/lib/nbproject/project.xml,v <-- project.xml new revision: 1.15; previous revision: 1.14 done Checking in editor/bracesmatching/nbproject/project.xml; /cvs/editor/bracesmatching/nbproject/project.xml,v <-- project.xml new revision: 1.2; previous revision: 1.1 done Checking in lexer/arch.xml; /cvs/lexer/arch.xml,v <-- arch.xml new revision: 1.13; previous revision: 1.12 done Checking in contrib/EditorBracesMatching/nbproject/project.xml; /cvs/contrib/EditorBracesMatching/nbproject/project.xml,v <-- project.xml new revision: 1.12; previous revision: 1.11 done Checking in lexer/src/org/netbeans/api/lexer/Token.java; /cvs/lexer/src/org/netbeans/api/lexer/Token.java,v <-- Token.java new revision: 1.11; previous revision: 1.10 done Checking in xml/tageditorsupport/nbproject/project.xml; /cvs/xml/tageditorsupport/nbproject/project.xml,v <-- project.xml new revision: 1.15; previous revision: 1.14 done Checking in ruby/gsf/nbproject/project.xml; /cvs/ruby/gsf/nbproject/project.xml,v <-- project.xml new revision: 1.13; previous revision: 1.12 done Checking in editor/lib2/nbproject/project.xml; /cvs/editor/lib2/nbproject/project.xml,v <-- project.xml new revision: 1.6; previous revision: 1.5 done Checking in java/editor/nbproject/project.xml; /cvs/java/editor/nbproject/project.xml,v <-- project.xml new revision: 1.45; previous revision: 1.44 done Checking in editor/options/nbproject/project.xml; /cvs/editor/options/nbproject/project.xml,v <-- project.xml new revision: 1.28; previous revision: 1.27 done Checking in html/editor/nbproject/project.xml; /cvs/html/editor/nbproject/project.xml,v <-- project.xml new revision: 1.29; previous revision: 1.28 done Checking in scripting/php/lexer/nbproject/project.xml; /cvs/scripting/php/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.4; previous revision: 1.3 done Checking in xml/lexer/nbproject/project.xml; /cvs/xml/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.3; previous revision: 1.2 done Checking in ruby/editing/nbproject/project.xml; /cvs/ruby/editing/nbproject/project.xml,v <-- project.xml new revision: 1.20; previous revision: 1.19 done Checking in javadoc/nbproject/project.xml; /cvs/javadoc/nbproject/project.xml,v <-- project.xml new revision: 1.32; previous revision: 1.31 done Checking in java/hints/nbproject/project.xml; /cvs/java/hints/nbproject/project.xml,v <-- project.xml new revision: 1.38; previous revision: 1.37 done Checking in java/source/nbproject/project.xml; /cvs/java/source/nbproject/project.xml,v <-- project.xml new revision: 1.38; previous revision: 1.37 done Checking in web/jspsyntax/nbproject/project.xml; /cvs/web/jspsyntax/nbproject/project.xml,v <-- project.xml new revision: 1.41; previous revision: 1.40 done Checking in lexer/api/apichanges.xml; /cvs/lexer/api/apichanges.xml,v <-- apichanges.xml new revision: 1.25; previous revision: 1.24 done Checking in ruby/refactoring/nbproject/project.xml; /cvs/ruby/refactoring/nbproject/project.xml,v <-- project.xml new revision: 1.4; previous revision: 1.3 done Checking in java/lexer/nbproject/project.xml; /cvs/java/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.7; previous revision: 1.6 done Checking in languages/engine/nbproject/project.xml; /cvs/languages/engine/nbproject/project.xml,v <-- project.xml new revision: 1.22; previous revision: 1.21 done Checking in html/lexer/nbproject/project.xml; /cvs/html/lexer/nbproject/project.xml,v <-- project.xml new revision: 1.7; previous revision: 1.6 done Checking in ruby/rhtml/nbproject/project.xml; /cvs/ruby/rhtml/nbproject/project.xml,v <-- project.xml new revision: 1.16; previous revision: 1.15 done Checking in contrib/JavadocFirstSentence/nbproject/project.xml; /cvs/contrib/JavadocFirstSentence/nbproject/project.xml,v <-- project.xml new revision: 1.4; previous revision: 1.3
You seem to have broken compilation of scripting/ejs, at least. Please check for other users of this API and make sure they are still compilable.
Apologies, I did a clean build of the full cluster config and I have also attempted to fix tests of the affected modules. But I did not know about the javac bugs that will be revealed by incremental building. I have fixed scripting/ejs. Checking in EJSTokenId.java; /cvs/scripting/ejs/src/org/netbeans/modules/languages/ejs/lexer/api/EJSTokenId.java,v <-- EJSTokenId.java new revision: 1.4; previous revision: 1.3