This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 55810

Summary:	I18N - Wrong character encoding during compilation
Product:	java	Reporter:	aralocram <aralocram>
Component:	Project	Assignee:	Tomas Zezula <tzezula>
Status:	RESOLVED FIXED
Severity:	blocker	CC:	abs, anebuzelsky, jf4jbug, kfrank, mgrummich, pjiricka, prehak
Priority:	P2	Keywords:	API, I18N
Version:	4.x
Hardware:	PC
OS:	Windows XP
Issue Type:	DEFECT	Exception Reporter:
Bug Depends on:	42638
Bug Blocks:

Description aralocram 2005-03-03 03:32:44 UTC

Java Version: JDK1.5.0_01, JRE1.5.0_01 

I created a project in Netbeans (an idex.html 
screen reporting to a servlet). Both index.html 
and in the servlet generated html page contains 
the line:
<meta http-equiv="Content-Type" 
content="text/html; charset=utf-8">
Additional, I setup the character code settings 
in Netbeans:
(tools-options-Java sources-Expert-default 
encoding=UTF-8

When I run the project, index.html displays 
itself perfectly, with the Chinese characters 
displayed properly. The problem comes when the 
html created servlet is displayed, which instead 
of the Chinese characters some strange characters 
are displayed (ä½ instead of Chinese).

I have tried different encodings from 
http://java.sun.com/j2se/1.4.2/docs/guide/intl/enc
oding.doc.html without any luck. I also setup the 
encoding of the file itself (using right click-
properties in the project menu of Netbeans).

Also, when I am editing the servlet, the 
characters are displayed properly. I type them 
directly without any issue, but then the display 
is wrong at runtime.

This is the doPost method of the servlet:

response.setCharacterEncoding("UTF-8"); //Not 
necessary because the next line should take care 
of it, but anyways...
response.setContentType("text/html; charset=UTF-
8; pageEncoding=UTF-8");
PrintWriter out = response.getWriter();
out.println("<html>");
out.println("<head>");
out.println("<meta http-equiv='Content-Language' 
content='en-us'>");
out.println("<meta http-equiv='Content-Type' 
content='text/html; charset=utf-8; 
pageEncoding=utf-8'>"); //Again not necessary 
line, but anyways
out.println("<title>Servlet</title>");
out.println("</head>");
out.println("<body>");
out.println("this is a test &#20320;&#22909; this 
is a test");
out.println("</body>");
out.println("</html>");
out.close();

This servlet is called from a html file. Also, 
when I load this in the browser, I right click on 
the screen and I can see that UTF-8 is setup 
allright. Doing some detective job I found that:
- If the editor displays the characters perfectly 
and...
- Other html pages (no servlets) in the same 
application can display chinese characters well 
and...
- I have configured the encoding as UTF-8 in the 
servlet properties and in general properties 
and...
- The file "web.xml" contains the encoding UTF-8 
in its first line then...

...my only guess is that something goes wrong 
during the building of the project itself (ant?). 
Unfortunately I have no idea about configuring 
ant to that level, but I began to think that the 
problem may be there, during the compilation...

Comment 1 Petr Jiricka 2005-03-03 17:23:13 UTC

Do you really need to set the following?

Additional, I setup the character code settings 
in Netbeans:
(tools-options-Java sources-Expert-default 
encoding=UTF-8

Looks like your servlet does not contain any UTF-8 characters
directly, only through &#20320;&#22909;

Try resetting the default encoding to the default value.

Also, can you somehow find out whether this happens for regular Java
classes? E.g. generate the HTML to a file in a class with a main()
method and then display it in the browser. Thanks.

Comment 2 aralocram 2005-03-04 03:37:29 UTC

Hello. Thanks for the help.

Answering your questions:

1) Do you really need to set the following?
Additional, I setup the character code settings 
in Netbeans:
(tools-options-Java sources-Expert-default 
encoding=UTF-8

Answer: I don't really know. I just thought the problem may have 
something to do with it, so I already set this in case that the 
problem was there.

2)Looks like your servlet does not contain any UTF-8 characters
directly, only through &#20320;&#22909;

Answer: Actually this is an issue with issuezilla. In the original 
message I wrote the Chinese characters, but I guess they can not be 
displayed here, so think that I wrote Chinese instead 
of "&#20320;&#22909"

3) Try resetting the default encoding to the default value.

Answer. At the beginning all the encoding (project, file and such) 
was default, but since I could not see the characters I opted for 
changing them

4) Also, can you somehow find out whether this happens for regular 
Javaclasses? E.g. generate the HTML to a file in a class with a main()
method and then display it in the browser. Thanks.

Answer: I am not sure exactly what you wanted me to do. Do you want 
me to create a normal java file which creates a html file, and then I 
just open that html file with a browser?

Additionally, I performed new tests.

A) I setup Ant to encode UTF-8 (Ant properties file.encoding="UTF-8") 
without luck

B) I opened the example web projects coming with netbeans (the one 
of 'Hello world!") but I changed the message by some Chinese, without 
luck. Still the message is displayed incorrectly.

C) In all my tests, while watching the generated html file, I right 
clicked on the file to check the encoding, and I even changed it to 
other Chinese encoding system without any luck. I can see the 
characters are not properly store during compilation.

Comment 3 Petr Jiricka 2005-03-10 08:10:40 UTC

I tried to reproduce this issue and got the same behavior.

Can you please try setting the following property in the project.properties file
of your project?

javac.compilerargs=-encoding UTF-8

The line containing  'javac.compilerargs='  is already there, so you only need
to add '-encoding UTF-8'. Then clean and build the project again. This worked
for me. Does it work for you as well? 

Thanks.

Comment 4 Petr Jiricka 2005-03-14 13:07:39 UTC

Reassigning to Java module. Currently, the Java source encoding in IDE options
is not in any way ties with the javac.compilerargs property of Java-based
project types. It would be useful to integrate these better.

This issue is very related to enhancement 19928.

Would it be possible to somehow make it clear to the user that
javac.compilerargs has to match the source encoding setting? A message box that
explains this appearing at an appropriate place would help.

Comment 5 Martin Matula 2005-03-14 15:20:57 UTC

Seems to be related to projects.

Comment 6 Tomas Zezula 2005-03-22 08:13:29 UTC

The encoding should not be set in the Options. It should be project specific.
There should be something like EncodingQuery which will be used by editor, build
system, export to HTML, ...
The j2seproject should provide EncodingQueryImplementation. The JSP and other
documents which have explicit encoding can provide EncodingQueryImplementation to.

But there is already an request for such an API on the editor for several years:
See: http://www.netbeans.org/issues/show_bug.cgi?id=37182

Comment 7 Ken Frank 2005-06-03 21:10:53 UTC

Is what the filer saw a common situation related to encodings ?
its not clear from discussion if this is a problem or RFE.

It seems like, if user cannot see the multibyte properly when using
product encoding or other encoding features or specification,
that it would be a p2 kind of issue, as has been other encoding related
issues.

ken.frank@sun.com
06/03/2005

Comment 8 Tomas Zezula 2005-11-01 10:55:44 UTC

Requires a new API.

Comment 9 Petr Nejedly 2006-01-31 09:39:08 UTC

*** Issue 37182 has been marked as a duplicate of this issue. ***

Comment 10 Ken Frank 2006-07-26 04:54:55 UTC

based on reading this issue and customer impact, as well as about
other bugs on encoding handling or setting, marking this as p2
as it seems to really be needed.

ken.frank@sun.com

Comment 11 Tomas Zezula 2006-07-31 10:09:31 UTC

The issue has an easy workaround for user, adding "-encoding UTF-8" into the
javac  additional args in the project's customizer as Peter J. described above.
Anyway the issue should be fixed in the dev. It shouldn't be part of the 5.5
since it needs new API EncodingQuery which is big amount of changes across the
whole IDE.

Comment 12 Ken Frank 2006-07-31 18:09:56 UTC

Tomas,

What's the best way to make this visible for fixing in nb6.0 ? Should there be a
keyword for it, or a separate kind of issue ? (since it mentions below that
another api is needed first)

ken.frank@sun.com

Comment 13 Tomas Zezula 2006-08-01 14:45:09 UTC

I've added an API keyword. I think that the issue need to be resolved into the
6.0 because the new java language model requires also encoding information.

Comment 14 Petr Jiricka 2006-08-02 16:44:02 UTC

It sounds like implementing the API suggested in issue 42638 is required to
solve this problem, correct?

Comment 15 Tomas Zezula 2006-08-02 17:13:33 UTC

Yes.

Comment 16 Tomas Zezula 2007-03-14 15:44:44 UTC

Fixed: New encoding API and implementation of this API for j2seproject.

Checking in openide/loaders/manifest.mf;
/cvs/openide/loaders/manifest.mf,v  <--  manifest.mf
new revision: 1.31; previous revision: 1.30
done
Checking in openide/loaders/api/apichanges.xml;
/cvs/openide/loaders/api/apichanges.xml,v  <--  apichanges.xml
new revision: 1.26; previous revision: 1.25
done
Checking in openide/loaders/nbproject/project.xml;
/cvs/openide/loaders/nbproject/project.xml,v  <--  project.xml
new revision: 1.27; previous revision: 1.26
done
RCS file:
/cvs/openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v
done
Checking in
openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation;
/cvs/openide/loaders/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v
 <--  org.netbeans.spi.queries.FileEncodingQueryImplementation
initial revision: 1.1
done
RCS file:
/cvs/openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java,v
done
Checking in
openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java;
/cvs/openide/loaders/src/org/netbeans/modules/openide/loaders/DataObjectEncodingQueryImplementation.java,v
 <--  DataObjectEncodingQueryImplementation.java
initial revision: 1.1
done
Checking in openide/loaders/src/org/openide/text/DataEditorSupport.java;
/cvs/openide/loaders/src/org/openide/text/DataEditorSupport.java,v  <-- 
DataEditorSupport.java
new revision: 1.40; previous revision: 1.39
done
Checking in
openide/loaders/test/unit/src/org/openide/text/DataEditorSupportTest.java;
/cvs/openide/loaders/test/unit/src/org/openide/text/DataEditorSupportTest.java,v
 <--  DataEditorSupportTest.java
new revision: 1.7; previous revision: 1.6
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProject.java;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProject.java,v
 <--  J2SEProject.java
new revision: 1.78; previous revision: 1.77
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProjectGenerator.java;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEProjectGenerator.java,v
 <--  J2SEProjectGenerator.java
new revision: 1.54; previous revision: 1.53
done
RCS file:
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java,v
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/queries/J2SEProjectEncodingQueryImpl.java,v
 <--  J2SEProjectEncodingQueryImpl.java
initial revision: 1.1
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl,v
 <--  build-impl.xsl
new revision: 1.83; previous revision: 1.82
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties,v
 <--  Bundle.properties
new revision: 1.86; previous revision: 1.85
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.form;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.form,v
 <--  CustomizerSources.form
new revision: 1.9; previous revision: 1.8
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.java;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerSources.java,v
 <--  CustomizerSources.java
new revision: 1.14; previous revision: 1.13
done
Checking in
java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java;
/cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java,v
 <--  J2SEProjectProperties.java
new revision: 1.63; previous revision: 1.62
done
RCS file:
/cvs/java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java,v
done
Checking in
java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java;
/cvs/java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/queries/FileEncodingQueryTest.java,v
 <--  FileEncodingQueryTest.java
initial revision: 1.1
done
Checking in
java/source/src/org/netbeans/modules/java/source/parsing/SourceFileObject.java;
/cvs/java/source/src/org/netbeans/modules/java/source/parsing/SourceFileObject.java,v
 <--  SourceFileObject.java
new revision: 1.6; previous revision: 1.5
done
Checking in projects/projectapi/overview.html;
/cvs/projects/projectapi/overview.html,v  <--  overview.html
new revision: 1.5; previous revision: 1.4
done
RCS file:
/cvs/projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v
done
Checking in
projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation;
/cvs/projects/projectapi/src/META-INF/services/org.netbeans.spi.queries.FileEncodingQueryImplementation,v
 <--  org.netbeans.spi.queries.FileEncodingQueryImplementation
initial revision: 1.1
done
Checking in projects/projectapi/src/org/netbeans/api/project/Project.java;
/cvs/projects/projectapi/src/org/netbeans/api/project/Project.java,v  <-- 
Project.java
new revision: 1.18; previous revision: 1.17
done
RCS file:
/cvs/projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java,v
done
Checking in
projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java;
/cvs/projects/projectapi/src/org/netbeans/modules/projectapi/ProjectFileEncodingQueryImplementation.java,v
 <--  ProjectFileEncodingQueryImplementation.java
initial revision: 1.1
done
Checking in projects/queries/apichanges.xml;
/cvs/projects/queries/apichanges.xml,v  <--  apichanges.xml
new revision: 1.7; previous revision: 1.6
done
Checking in projects/queries/arch.xml;
/cvs/projects/queries/arch.xml,v  <--  arch.xml
new revision: 1.11; previous revision: 1.10
done
Checking in projects/queries/manifest.mf;
/cvs/projects/queries/manifest.mf,v  <--  manifest.mf
new revision: 1.12; previous revision: 1.11
done
Checking in projects/queries/nbproject/project.xml;
/cvs/projects/queries/nbproject/project.xml,v  <--  project.xml
new revision: 1.11; previous revision: 1.10
done
RCS file:
/cvs/projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java,v
done
Checking in projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java;
/cvs/projects/queries/src/org/netbeans/api/queries/FileEncodingQuery.java,v  <--
 FileEncodingQuery.java
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java,v
done
Checking in projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java;
/cvs/projects/queries/src/org/netbeans/modules/queries/UnknownEncoding.java,v 
<--  UnknownEncoding.java
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java,v
done
Checking in
projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java;
/cvs/projects/queries/src/org/netbeans/spi/queries/FileEncodingQueryImplementation.java,v
 <--  FileEncodingQueryImplementation.java
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/FileEncodingQueryTest.java,v
 <--  FileEncodingQueryTest.java
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/data.properties,v
 <--  data.properties
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_after_block,v
 <--  encoding_after_block
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block,v
 <--  encoding_on_block
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_block_long,v
 <--  encoding_on_block_long
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start,v
 <--  encoding_on_start
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/encoding_on_start_long,v
 <--  encoding_on_start_long
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding,v
 <--  no_encoding
initial revision: 1.1
done
RCS file:
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long,v
done
Checking in
projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long;
/cvs/projects/queries/test/unit/src/org/netbeans/api/queries/data/no_encoding_long,v
 <--  no_encoding_long
initial revision: 1.1
done
Checking in ide/golden/deps.txt;
/cvs/ide/golden/deps.txt,v  <--  deps.txt
new revision: 1.473; previous revision: 1.472
done

Comment 17 Petr Jiricka 2007-03-14 16:22:11 UTC

Wait, this was reported against web project. So how can it be fixed if you only
fixed it in j2se project?

Comment 18 Tomas Zezula 2007-03-14 16:32:52 UTC

Since you have changed this issue to an API request and moved it to
java/project. You have a new P2 issue to implement FEQ API in the web project: 
http://www.netbeans.org/issues/show_bug.cgi?id=97850

Comment 19 Tomas Zezula 2007-03-14 16:35:02 UTC

This umbrella issue is probably more helpful to track the requests to update the
modules. http://www.netbeans.org/issues/show_bug.cgi?id=97848

Comment 20 Petr Jiricka 2007-03-14 16:38:33 UTC

Ok, then it's fine. Thanks.

Comment 21 Ken Frank 2007-05-29 17:26:51 UTC

to clarify, for nb6, will this issue be verifiable once the feq for web project
is done ?
or is this separate situation that can be verified now ?

ken.frank@sun.com

Comment 22 Tomas Zezula 2007-05-29 17:37:33 UTC

We need to wait until the FEQ in web project is done.