Please use the Apache issue tracking system for new NetBeans issues ( !!
Bug 120508 - I18N - Plain Text File Editing Broken (Encoding issue)
I18N - Plain Text File Editing Broken (Encoding issue)
Product: java
Classification: Unclassified
Component: Freeform
All All
: P2 (vote)
: 6.x
Assigned To: Milan Kubec
: I18N
Depends on: 120848
Blocks: 97854
  Show dependency treegraph
Reported: 2007-10-30 17:08 UTC by cberger
Modified: 2008-03-06 20:10 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT

Editor with broken UTF-8 (39.62 KB, image/png)
2007-10-31 10:23 UTC, cberger
Yeah, here (under src/java, the java root) it works (38.63 KB, image/png)
2007-10-31 10:24 UTC, cberger
Yeah, project is set to UTF-8 (76.98 KB, application/octet-stream)
2007-10-31 10:26 UTC, cberger
ant/freeform diff (3.66 KB, patch)
2007-10-31 17:09 UTC, Milan Kubec
Details | Diff
java/freeform diff (6.99 KB, patch)
2007-10-31 17:10 UTC, Milan Kubec
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description cberger 2007-10-30 17:08:04 UTC
When I open a text file (foo.txt) on my Mac OS X, everything
is grabled if it contains UTF-8 text.
This, despite the fact that:
   - UTF-8 is the default in Mac OS X (vi encode this way)
   - The project property sets the (Java) Encoding to UTF8.

I think that 2 actions are needed here:

   1) UTF-8 should be the default on Mac OS X.
   2) The project property "Encoding" should apply to all
      source files, not only Java source files.

Putting P1 because editing plain text is a basic functionality
of any IDE IMO.

Product Version: NetBeans IDE Dev (Build 200710220000)
Java: 1.5.0_07; Java HotSpot(TM) Client VM 1.5.0_07-87
System: Mac OS X version 10.4.10 running on i386; MacRoman; en_US (nb)
Userdir: /Users/cedric/.netbeans/dev
Comment 1 Vitezslav Stejskal 2007-10-31 09:39:19 UTC
Tomas is looking into it. Just a few things - the encoding setting in project properties *is* applied for all files and
not only java. Are you sure that the text file is really encoded by UTF8? And are you sure that the file really belongs
to a project, which encoding is set to UTF8? This seems to be working fine on my Ubuntu, but I don't have much
experience with Mac OS X so can't tell if things work the same.
Comment 2 Tomas Zezula 2007-10-31 10:03:00 UTC
Works fine on Solaris, Linux and Apple.
Point 1) The default encoding is the encoding the OS uses (found by Charset.defaultCahrset().name()). On Linux and
Solaris it's UTF-8, on the Apple it's MacRoman (at least in setup on Apple available in QA), but you should complain
about this elsewhere. It's a good habit that application store the file in system provided encoding by default, so we
cannot violate this rule on Apple. On the Linux and Solaris the system provided encoding is based on exported
environment variable and can be changed during login or in shell session.
Point 2) The project encoding applies to all files which are owned by the project, (are under the project folder or
under one of the source roots),

Anyway you can use Open In Encoding/ Save In Encodin actions, see attachment of issue: #114123. I will make them
available  using autoupdate.
Comment 3 cberger 2007-10-31 10:19:06 UTC
imac:~/Work/svn-www/head/jtkrepo/src cedric$ uname -a
Darwin imac.local 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386
imac:~/Work/svn-www/head/jtkrepo/src cedric$ cat utf8.txt 
déjà vu
imac:~/Work/svn-www/head/jtkrepo/src cedric$ hexdump -C utf8.txt 
00000000  64 c3 a9 6a c3 a0 20 76  75 0a                    |d..j.. vu.|

Comment 4 cberger 2007-10-31 10:23:09 UTC
Created attachment 52097 [details]
Editor with broken UTF-8
Comment 5 cberger 2007-10-31 10:24:20 UTC
Created attachment 52098 [details]
Yeah, here (under src/java, the java root) it works
Comment 6 cberger 2007-10-31 10:26:29 UTC
Created attachment 52099 [details]
Yeah, project is set to UTF-8
Comment 7 cberger 2007-10-31 10:33:23 UTC
point 1) Yeah, Maybe, that's not clear to me what is the default encoding
in Mac OS X. Some say MacRoman, but the terminal/vi seem to understand
unicode, and filenames in MAC OS X are stored in UTF-8.

point 2) I think that the current behaviour is completely unintuitive
and totally confusing. If you specify an encoding for the project, it should
be applied to EVERYTHING in that project IMO.  Then if "Java Source Encoding"
is applied to non-java sources, then the UI should be much clearer.

Finally, I looked into all the settings panel everywhere (in advanced or
simplified mode) to find how to change the encoding for text files,
and could not find everything at all. This is really very confusing.

What I would really love is a per-project encoding, which apply to
everything under the project root. I'm reopening that until that mess
is sorted out somehow.

Comment 8 cberger 2007-10-31 10:42:48 UTC
Just to be clearer on one point. You said:

Point 2) The project encoding applies to all files which are owned by the project, (are under the project folder or
under one of the source roots),

-> First, this is incorrect, see my first attachment (utf8.txt IS under the project folder).
-> Second, I think the "project properties" panel UI is not well designed, because it
       is not clear if the "Encoding" apply to all project files or only "Java Sources".
      (I think this is GOOD that the encoding apply to all project files, it's just not clear
       when you open the Project Properties file) 
Comment 9 Tomas Zezula 2007-10-31 10:53:07 UTC
Point 1) The default encoding on Apple is MacRoman,
simple way how to find it Charset.detDefaultCharset().name()

Pint 2) Works fine even on Apple, but it's project specific. The source encoding affects also non java files including
text files. I've tried it on normal j2se project. Which project type do you use (standard java, web project, project
from existing sources)? Thanks
Comment 10 cberger 2007-10-31 10:58:20 UTC
Java Project with Existing Ant Script
Comment 11 cberger 2007-10-31 11:13:08 UTC
Can you guys reproduce the problem now?
Comment 12 Tomas Zezula 2007-10-31 11:38:35 UTC
Thanks, for feedback now it's clear. I've tried the java project which works fine, but you have a freeform project
(project from existing ant script) where it behaves as you described, which is a bug. The correct behavior is that all
files owned by project are in project encoding (as the java project and others do).
Comment 13 Milan Kubec 2007-10-31 17:09:57 UTC
Created attachment 52152 [details]
ant/freeform diff
Comment 14 Milan Kubec 2007-10-31 17:10:48 UTC
Created attachment 52153 [details]
java/freeform diff
Comment 15 Milan Kubec 2007-10-31 17:14:56 UTC
I've attached proposed fix for the issue. New element was added to freeform schema called <encoding> and it will provide
encoding for project folder itself. Adding API_REVIEW_FAST since there is change in the schema. Please review. I will
provide test of course. Thanks.
Comment 16 Tomas Zezula 2007-10-31 17:35:12 UTC
Adding encoding element to projects seems good to me.
Only one thing is:
+            // #120508: return projectEncoding for any file under project folder
+            if (FileUtil.isParentOf(helper.getProjectDirectory(), file) && projectEncoding != null) {
+                return projectEncoding;
+            }
This is probably not needed. When the query was called it is for file which belongs to the project since FEQ returned
the project for this file.
Maybe something like:
if is_in_source_group
 return source_group_encoding;
 return project_encoding.

The patch seems fine.
Comment 17 cberger 2007-10-31 17:50:19 UTC
Or better:
if is_in_source_group and source_group_encoding != nil
 return source_group_encoding;
 return project_encoding.

so if only project_encoding is set, it will do the right thing.

Comment 18 Jesse Glick 2007-10-31 18:08:52 UTC
Not a P1.

No schema or other API change is necessary. Freeform projects already have an <encoding> element, in <source-folder>.
(You wrote the code to interpret it yourself!) If you want it to apply to all files in the project, simply place it on
the generic <source-folder> rather than a Java <source-folder> and you are done. Probably the only change that is needed
is that the java/freeform GUI customizer neglected to set <encoding> in the right place.
Comment 19 Ken Frank 2007-11-01 06:14:05 UTC

is there anything about this fix that needs to be added to help kind of info on feq ? docs writers
are working on 116041 about feq; I've shared
with them the doc I collated based on your
and others explanations and feedback.
Comment 20 cberger 2007-11-01 08:11:33 UTC
So, following is the beginning of my freeform definition.
Where is the "generic <source-folder>"?

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="">
        <general-data xmlns="">
        <general-data xmlns="">
                    <label>Source Files</label>
                    <label>Web Files</label>

Comment 21 Milan Kubec 2007-11-01 08:27:50 UTC
Generic was meant as untyped, so I assume something like this:

    <label>Project name</label>
Comment 22 cberger 2007-11-01 08:45:36 UTC
Ok, so I changed my project like the following, but it makes NO difference and does not work:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="">
        <general-data xmlns="">
        <general-data xmlns="">
    <label>Project name</label>
                    <label>Source Files</label>
                    <label>Web Files</label>
Comment 23 Tomas Zezula 2007-11-01 09:12:42 UTC
To Ken: There is nothing to document about this bug for NB 6.0. It has to be fixed in 6.0 release to behave as other
Comment 24 Milan Kubec 2007-11-01 16:33:33 UTC
Fixed. Source folder for storing encoding of project dir itself was added.

(to cberger: the problem was that the '.' wasn't interpreted correctly)

Checking in ant/freeform/src/org/netbeans/modules/ant/freeform/;
/cvs/ant/freeform/src/org/netbeans/modules/ant/freeform/,v  <--
new revision: 1.4; previous revision: 1.3
Checking in java/freeform/src/org/netbeans/modules/java/freeform/ui/;
/cvs/java/freeform/src/org/netbeans/modules/java/freeform/ui/,v  <--
new revision: 1.17; previous revision: 1.16
Comment 25 Jesse Glick 2007-11-01 17:08:13 UTC
In ProjectModel, "projectdir" is a poor choice for the label. Use the project name. This label may appear e.g. in the
New File wizard. And for robustness, if (sf.label.equals("projectdir") && sf.location.equals(".")) should simply be if
(sf.location.equals(".")), so that the label can be freely changed later.

The code also looks confused as to null vs. "" for String enc.
Comment 26 Milan Kubec 2007-11-02 19:18:16 UTC
Fixed too, see issue #120848.
Comment 27 cberger 2007-11-02 19:20:48 UTC
So, does something need to be changed in the project.xml?
Should we just open the project properties dialog and press ok?
Comment 28 Milan Kubec 2007-11-02 22:27:22 UTC
You don't need to change anything in project.xml.
Comment 29 Ken Frank 2007-11-04 16:21:22 UTC
as result of the fix, is there a new label in project = from below
"Source folder for storing encoding of project dir itself was added"

if so, where is this label seen, what kind of projects, can it be different
value than project encoding, is it a changeable property in props or elsewhere ?
Comment 30 Milan Kubec 2007-11-05 08:31:58 UTC
I might not understand correctly the question, but encoding in the projectdir source-folder is set in project properties
along with other source folders. The label cannot be set except editing project.xml file. And I think that the label is
not visible anywhere.
Comment 31 Ken Frank 2008-03-06 20:10:50 UTC
cberger - does it work ok for you as to the fix ?

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo