You need to log in before you can comment on or make changes to this bug.
Editor component in NB6.0m6 replaces non-ascii chars to \uxxxx ater an action like fix-imports, refactoring, ... This is very critical issue, at least asian countries.
Created an attachment (id=37472) [details] Before fix-import..
Created an attachment (id=37473) [details] After fix-imports..
This issue also reproducible copy-paste action from system clipboard, like copy from notepad paste to nb6m6 editor.
This might be related to 92642, in which if new project main class has name with multibyte, that the escaped ascii is inserted, so this might be a more general kind of situation in java projects or in editor operations perhaps for other kinds of projects or file types. ken.frank@sun.com
BTW could you please check whether would it also happen if you run the following and do the copy-paste? $jdk_location/bin/java -jar $jdk_location/demo/jfc/Notepad/Notepad.jar Thanks.
Did not happen to me - running in ja locale on solaris, used the notepad to enter multibyte then saved file and reopened it - multibyte looks ok, no escapped ascii. of course in notepad cant do the other things done in ide like fix imports etc but if this is related to 92642, then you dont need to do anything specific for it to happen since happens on creation of initial java file. ken.frank@sun.com
This problem was reproduced, through the action of fix-imports, in my environments below: - Windows XP SP2(Japanese), jdk1.6.0(b103), NetBeans 6.0m6 - Windows XP SP2(Japanese), jdk1.5.0_10, NetBeans 6.0m6 - Solaris 10 3/05 on SPARC, jdk1.5.0(fcs), NetBeans 6.0m6, Locale ja - Mac OS X 10.4.8(Japanese), jdk1.5.0_07, NetBeans 6.0m6 This problem is always reproducible by actions like fix-imports or refactoring. However, copy-paste action didn't cause this problem all the time. I'll identifiy which condition causes this problem in copy-paste action.
So it seems solely related to the code generation in java. Reassigning to java module for evaluation.
SHould be fixed by the #42638
> SHould be fixed by the #42638 And exist any chance that this will be fixed before 6.0M8 release? This was working before M6. It's too big problem?
I think, that probem is here: // XXX - this might be with wrong encoding Iterable<? extends CompilationUnitTree> trees = currentInfo.getJavacTask().parse(new JavaFileObject[] {currentInfo.jfo}); in JavaSource.java (Java Source module)
The JavaSource is not the problem, the comment is misleading. The problem is that the JavaEditorSupport misses the loadFromStreamToKit and storeFromKitToStream methods. Here are these methods implemented using new FEQ API. I will fix it when the FEQ will be in the build. @Override protected void loadFromStreamToKit(StyledDocument doc, InputStream stream, EditorKit kit) throws IOException, BadLocationException { final Charset c = FileEncodingQuery.getEncoding(this.getDataObject().getPrimaryFile()); final Reader r = new InputStreamReader (stream, c); kit.read(r, doc, 0); } @Override protected void saveFromKitToStream(StyledDocument doc, EditorKit kit, OutputStream stream) throws IOException, BadLocationException { final Charset c = FileEncodingQuery.getEncoding(this.getDataObject().getPrimaryFile()); final Writer w = new OutputStreamWriter (stream, c); kit.write(w, doc, 0, doc.getLength()); }
Checking in openide/loaders/manifest.mf; /cvs/openide/loaders/manifest.mf,v <-- manifest.mf new revision: 1.31; previous revision: 1.30 done Checking in openide/loaders/api/apichanges.xml; /cvs/openide/loaders/api/apichanges.xml,v <-- apichanges.xml new revision: 1.26; previous revision: 1.25 done Checking in openide/loaders/nbproject/project.xml; /cvs/openide/loaders/nbproject/project.xml,v <-- project.xml new revision: 1.27; previous revision: 1.26 done RCS file: /cvs/openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v done Checking in openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation; /cvs/openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v <-- org.netbeans.spi.queries.FileEncodingQueryImplementation initial revision: 1.1 done RCS file: /cvs/openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java,v done Checking in openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java; /cvs/openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java,v <-- DataObjectEncodingQueryImplementation.java initial revision: 1.1 done Checking in openide/loaders/src/org/openide/text/DataEditorSupport.java; /cvs/openide/loaders/src/org/openide/text/DataEditorSupport.java,v <-- DataEditorSupport.java new revision: 1.40; previous revision: 1.39 done Checking in openide/loaders/test/unit/src/org/openide/text/DataEditorSupportTest.java; /cvs/openide/loaders/test/unit/src/org/openide/text/DataEditorSupportTest.java,v <-- DataEditorSupportTest.java new revision: 1.7; previous revision: 1.6 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProject.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProject.java,v <-- J2SEProject.java new revision: 1.78; previous revision: 1.77 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProjectGenerator.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProjectGenerator.java,v <-- J2SEProjectGenerator.java new revision: 1.54; previous revision: 1.53 done RCS file: /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java,v done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java,v <-- J2SEProjectEncodingQueryImpl.java initial revision: 1.1 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl,v <-- build-impl.xsl new revision: 1.83; previous revision: 1.82 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties,v <-- Bundle.properties new revision: 1.86; previous revision: 1.85 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.form; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.form,v <-- CustomizerSources.form new revision: 1.9; previous revision: 1.8 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.java,v <-- CustomizerSources.java new revision: 1.14; previous revision: 1.13 done Checking in java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java,v <-- J2SEProjectProperties.java new revision: 1.63; previous revision: 1.62 done RCS file: /cvs/java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java,v done Checking in java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java; /cvs/java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java,v <-- FileEncodingQueryTest.java initial revision: 1.1 done Checking in java/source/src/org/netbeans/modules/java/source/parsing/SourceFileObject.java; /cvs/java/source/src/org/netbeans/modules/java/source/parsing/SourceFileObject.java,v <-- SourceFileObject.java new revision: 1.6; previous revision: 1.5 done Checking in projects/projectapi/overview.html; /cvs/projects/projectapi/overview.html,v <-- overview.html new revision: 1.5; previous revision: 1.4 done RCS file: /cvs/projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v done Checking in projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation; /cvs/projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v <-- org.netbeans.spi.queries.FileEncodingQueryImplementation initial revision: 1.1 done Checking in projects/projectapi/src/org/netbeans/api/project/Project.java; /cvs/projects/projectapi/src/org/netbeans/api/project/Project.java,v <-- Project.java new revision: 1.18; previous revision: 1.17 done RCS file: /cvs/projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java,v done Checking in projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java; /cvs/projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java,v <-- ProjectFileEncodingQueryImplementation.java initial revision: 1.1 done Checking in projects/queries/apichanges.xml; /cvs/projects/queries/apichanges.xml,v <-- apichanges.xml new revision: 1.7; previous revision: 1.6 done Checking in projects/queries/arch.xml; /cvs/projects/queries/arch.xml,v <-- arch.xml new revision: 1.11; previous revision: 1.10 done Checking in projects/queries/manifest.mf; /cvs/projects/queries/manifest.mf,v <-- manifest.mf new revision: 1.12; previous revision: 1.11 done Checking in projects/queries/nbproject/project.xml; /cvs/projects/queries/nbproject/project.xml,v <-- project.xml new revision: 1.11; previous revision: 1.10 done RCS file: /cvs/projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java,v done Checking in projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java; /cvs/projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java,v <-- FileEncodingQuery.java initial revision: 1.1 done RCS file: /cvs/projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java,v done Checking in projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java; /cvs/projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java,v <-- UnknownEncoding.java initial revision: 1.1 done RCS file: /cvs/projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java,v done Checking in projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java; /cvs/projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java,v <-- FileEncodingQueryImplementation.java initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java,v <-- FileEncodingQueryTest.java initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties,v <-- data.properties initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block,v <-- encoding_after_block initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block,v <-- encoding_on_block initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long,v <-- encoding_on_block_long initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start,v <-- encoding_on_start initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long,v <-- encoding_on_start_long initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding,v <-- no_encoding initial revision: 1.1 done RCS file: /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long,v done Checking in projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long; /cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long,v <-- no_encoding_long initial revision: 1.1 done Checking in ide/golden/deps.txt; /cvs/ide/golden/deps.txt,v <-- deps.txt new revision: 1.473; previous revision: 1.472 done
I tested it in Build 200703151900, but it still doesn't work.
Do you have j2seproject (general/java application) and correctly set encoding in the project's customizer? It's not yet implemented in the other project types. Can you attach the project or at least the test file? Maybe the refactoring code access directly the file object content and should be updated to use FileEncodingQuery.
I've probably found a code that read, writes directly to FileObjects in the Commit. I will try to fix it when I find out what it does.
Yes, I used general/java application. Source files are correctly in UTF8, but when I write this: /// ěščřžýáíé And try fix import, get: // \u011B\u0161\u010D\u0159\u017Eýáíé Also when I have __DATE__ in teplates it's create: 16. b\uFFFDezen 2007 I can't try define my own temlate varibles, because I don't found configuration for it in NB6 :-(
Templates are different thing and have nothing in common with this, here is an issue for template manager: #94676. In general everything what handles text using Document should work fine. The problem is with code accessing directly files. This issue is about the change of "ěščřžýáíé" to unicode escapes. The problem is probably caused by by wrong handling in jackpot's Commit which I am now fixing.
I've fixed Commit to use FEQ when accessing files on disk, but it didn't help much. Then I've found classes: Prety and VeryPrety which do following with all identifiers: if (s == null) return; int limit = s.length(); for (int i = 0; i < limit; i++) { char c = s.charAt(i); if (c <= 255) out.append(c); else { out.append("\\u"); out.append(hex[(c >> (3 * 4)) & 0xF]); out.append(hex[(c >> (2 * 4)) & 0xF]); out.append(hex[(c >> (1 * 4)) & 0xF]); out.append(hex[(c >> (0 * 4)) & 0xF]); } } This destroys all the correctly encoded chars. I am missing the reason of this code since out is a Writer which works on character level, does not need to encode anything. I will need to ask about the intention behind this.
Checking in src/org/netbeans/api/java/source/ModificationResult.java; /cvs/java/source/src/org/netbeans/api/java/source/ModificationResult.java,v <-- ModificationResult.java new revision: 1.8; previous revision: 1.7 done Checking in src/org/netbeans/modules/java/source/pretty/VeryPretty.java; /cvs/java/source/src/org/netbeans/modules/java/source/pretty/VeryPretty.java,v <-- VeryPretty.java new revision: 1.12; previous revision: 1.11 done
in case this related to 92642, some comments added to that issue: using 0316 (which does not have these fixes), now the multibyte in the packagename in java file is not correct, not showing escaped ascii as in original, but random or garbage characters. see most recent gif in that issue. Don't know if this is related the new project properties for java projects, which defaults to utf8 and assume would handle encoding ok so that the mbyte would show ok, no matter what locale user is in when running ide. ken.frank@sun.com
*** Issue 95574 has been marked as a duplicate of this issue. ***
*** Issue 97919 has been marked as a duplicate of this issue. ***
I tested templates in NB build 200703221900. A) I created project with UTF-8 encoding. Open Java Class template and put ěščřžýáí chars to it. Create new Class file. Result: ěščřžýáíé is OK but __DATE__ is 23. b�ezen 2007 ( I'm using cz_cs locale) B) I switch project to Windows-1250 I create new Class Result: ěščřžýáíé is created as: ěščřžýáĂĂ© but __DATE is corectly displayed as 23. březen 2007 Are these bugs related to this issue? I don't understand it. Why NB doesn't use for all internall operation unicode? Only when user load or save .java file, NB take conversion from/to unicode.
I can file seprate issues as needed; please let me know details and which component/subcomponents for each. Also, is 92642 is related to this one ? ken.frank@sun.com
Sorry, I send my comment to another issue :-(
Created an attachment (id=39917) [details] A button with no-ascii text in GUI editor
Created an attachment (id=39918) [details] Generated code
The GUI editor has a similar behavior, the text of JButton, JLabel etc were converted to \uxxxx sequence.
am reopening since seems not to be fixed in recent builds like of 0321, - is this one dependent on 42638 or other issues being fixed ? (which has many dependent issues itself) if so, can this one stay open until that one or others are fixed ? ken.frank@sun.com
to: gtlang I think, that this is not so big problem. This behavior is very very old and technically is all OK. But I agree, that now, when Netbeans support settings for project encoding will be bettet generate string for form in the same encoding. But I think that this is not related to this issue.
The problem in form editor is not related to this issue. The unicode escapes (\u) are generated by form editor and placed into java infrastructure as Strings ("\uxxx\uxxx"). The change to unicode escapes are not created as part of character encoding during save. I've talk about this with form engineers and they don't see it as a problem since the code is guarded and generated. I've added a new bug to form editor #98930. I want to keep this issue related only to java document handling. From the user point of view the problems may be very similar but are caused by different parts of the IDE.
To Ken: If the only problem with "\u" was in the forms I will close it since there is a new issue filled to form #98930. The java module cannot affect the form inserted code it comes as an already escaped string. Can you verify it?
to tprochazka: about your comments below: I tested templates in NB build 200703221900. A) I created project with UTF-8 encoding. Open Java Class template and put ěščřžýáí chars to it. Create new Class file. Result: ěščřžýáíé is OK but __DATE__ is 23. b�ezen 2007 ( I'm using cz_cs locale) B) I switch project to Windows-1250 I create new Class Result: ěščřžýáíé is created as: ěščřžýáĂĂ© but __DATE is corectly displayed as 23. březen 2007 Are these bugs related to this issue? I don't understand it. Why NB doesn't use for all internall operation unicode? Only when user load or save .java file, NB take conversion from/to unicode. ---> does that still happen ? if so, is it expected since 42638 might still be in process of being fixed ? if so, I think another issue could be filed since this one was about escaped asci in editor rather than about other incorrect display of characters such as you are seeing (ie random ascii or graphics characters) I suggest also trying it on solaris utf8 ja or czech locales since sometimes, if there are encoding problems, they might show ok on windows but not in solaris utf8 locale. ken.frank@sun.com
as to closing this, can all check to see if the original issue about escaped ascii in java editor is solved, except for in source code related to form editor, since separate but was filed on that ? if so, then this can be closed, with understanding that what tprochazka observed below will be filed first as a separate issue pending his comments (see previous update to this issue on that) suggest lets wait for both confirmations or clarifications before closing this. ken.frank@sun.com
to kfrank: I entered new issue for templates encoding problem here: http://www.netbeans.org/issues/show_bug.cgi?id=98874
The problem with templates reported by tprochazka and solved in issue #98874 is fixed. The only remaining problem is the escaped characters enterd by formeditor into guarded block issue #98930 which cannot be solved in java module. Closing as fixed.
Okazaki and Gtlang - is this fixed for you now ? If so, I can mark it as verified. ken.frank@sun.com
I confirmed this problem is no longer reproducible in NB6.0m9. Thanks a million!
Based on comments from submitter (okazaki) and others in this issue, am marking as verified. ken.frank@sun.com
