Bug 128729 - 18N - Ide doesn't show new MySQL table with multibyte chars in name
18N - Ide doesn't show new MySQL table with multibyte chars in name
Status: RESOLVED WORKSFORME
Product: db
Classification: Unclassified
Component: Code
6.x
All All
: P2 (vote)
: 6.x
Assigned To: David Vancouvering
issues@db
: I18N
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-28 17:26 UTC by kaa
Modified: 2008-07-31 15:49 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
image (170.74 KB, image/jpeg)
2008-03-06 17:05 UTC, kaa
Details
ide log (59.69 KB, text/plain)
2008-03-06 17:25 UTC, kaa
Details
sql_err.jpg (35.28 KB, image/jpeg)
2008-04-30 17:53 UTC, kaa
Details
query.JPG (65.13 KB, image/jpeg)
2008-04-30 17:53 UTC, kaa
Details
Screenshot showing multibyte tables on Windows XP (52.45 KB, image/jpeg)
2008-06-19 22:03 UTC, David Vancouvering
Details

Note You need to log in before you can comment on or make changes to this bug.
Description kaa 2008-02-28 17:26:22 UTC
Product Version: NetBeans IDE Dev (Build 200802191203)
Java: 1.6.0_03; Java HotSpot(TM) Client VM 1.6.0_03-b05
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)

Below is the extract from the list of variables of the server environment:

| character_set_client            | utf8                                                           |
| character_set_connection        | utf8                                                           |
| character_set_database          | utf8                                                           |
| character_set_filesystem        | binary                                                         |
| character_set_results           | utf8                                                           |
| character_set_server            | utf8                                                           |
| character_set_system            | utf8                                                           |
| character_sets_dir              | C:\Program Files\MySQL\MySQL Server 5.1\share\charsets\        |
| collation_connection            | utf8_general_ci                                                |
| collation_database              | utf8_general_ci                                                |
| collation_server                | utf8_general_ci

Steps:
1. Connect to MySQL server from the IDE
2. Try to create table using mbyte in its name (wizard didn't report any error)

The table wasn't shown in the list of tables.
With eng name it was created and shown ok. I was able to add mbyte in it and view table data.
Comment 1 kaa 2008-02-28 17:27:17 UTC
Probably its about communication of ide/db module with mysql and use of wrong encoding assumption - like using win31j
when should be utf8 or visa versa.
Comment 2 David Vancouvering 2008-02-28 19:01:23 UTC
Hi, Ken.  I'm going to need some help reproducing this.  How can I, without making my environment completely unusable
since I don't read Japanese, enter a multi-byte string as a table name?  

I also am confused about how the db module could be possible communicating with MySQL in any way except utf-8 given that
it's using JDBC, which is fully unicode.  Can you explain more what you meant when you said "use of wrong encoding
assumption - like using win31j when should be utf8 or visa versa."

Thanks.
Comment 3 Ken Frank 2008-02-28 19:06:28 UTC
David,

kaa, the bug filer (Andrey) is on our i18n testing team and he is our dbase
expert and focus on that i18n testing along with many other things.
Thus he can communicate with you about all of this; feel free to cc me in any communications.

as to setup on windows or solaris, there is no need at all to read some other language
to run in other locale.

I'll send you separately mail about 10 minute setup for windows for other locales
and how to get some multibyte chars easily and same info about solaris.

ken.frank@sun.com
Comment 4 David Vancouvering 2008-03-01 00:12:05 UTC
I just did this on Mac OSX with the exact settings you have below and it worked for me.

I will try to reproduce on the Windows.

Can you confirm this is consistently reproducible?
Comment 5 David Vancouvering 2008-03-04 06:16:05 UTC
I have tested this on Mac, Ubuntu and Windows using the same server settings and Simplified Chines multibyte characters,
and in all cases I do not see this behavior.

Can you please verify that this is still a problem for you.
Comment 6 kaa 2008-03-06 17:04:23 UTC
Product Version: NetBeans IDE 6.1 Beta (Build 200803041230)
Java: 1.6.0_03; Java HotSpot(TM) Client VM 1.6.0_03-b05
System: Windows XP version 5.1 running on x86; MS932; ja_JP

I tried again using Japanese chars in the table name. No errors were shown. The table wasn't shown under connection node
-> Tables. Attempt to create another table with the same name gives err message with bad multibyte characters.

Comment 7 kaa 2008-03-06 17:05:17 UTC
Created attachment 57903 [details]
image
Comment 8 kaa 2008-03-06 17:25:23 UTC
Created attachment 57905 [details]
ide log
Comment 9 David Vancouvering 2008-03-06 21:55:34 UTC
Very frustrating, it's very hard to fix something you can't reproduce. I'll try again with Japanese chars on Win XP
Comment 10 David Vancouvering 2008-03-06 22:04:46 UTC
I noticed you're running in a fully localized system, where I'm running in an English system where I am entering in
characters using a specialized keyboard input tool.  I wonder if that's making a difference.

I would not be able to use an operating system fully localized to Japanese, I think.

The error log does not show anything useful except that the AWT thread was blocked.

If you create the table using the SQL editor and refresh the view, does that work?

Also, if you refresh the view after creating the table the first time, does the table then show up?

Thanks.
Comment 11 David Vancouvering 2008-03-07 19:17:58 UTC
I can't seem to figure out how to set up my environment to use the Japanese character set, let alone set it up so that
the entire OS is localized for Japanese.  I tried it again with Chinese and no problems.

I need your help reproducing this; perhaps what we should do is set up a VNC session.  You could pull the latest code
from the trunk, build from that, and then we can walk through this, and maybe I can do some debugging.

I don't know your location - what times work well for you to do this?

Thanks.
Comment 12 Ken Frank 2008-03-07 20:23:34 UTC
Andrey, please clarify and resummarize the problem; this does not seem as clear;  in initial report it says
table not shown in listed tables, but gif shows some error message, so is it that
the table was not created at all ?

I'm marking issue as incomplete so developers will know more info is needed; please remove
that keywd when the info is added.

also, later in the issue it states
I tried again using Japanese chars in the table name. No errors were shown. The table wasn't shown under connection node
-> Tables. Attempt to create another table with the same name gives err message with bad multibyte characters.

does it mean  that this time the error popup did not happen as per gif ?  or does error popup
happen only sometimes ?

please clarify and consolidate to one unified description with steps and numbered comments.

thanks.

ken.frank@sun.com
Comment 13 David Vancouvering 2008-03-08 01:23:18 UTC
More information:

I can reproduce this with Japanese characters on Windows
I have been unsuccessful reproducing on Ubuntu, as I need to correctly configure MySQL
This does not fail on Macintosh, the table is correctly redisplayed.

There is a separate issue, which is how the table name is displayed in the error message.  I tested this on Macintosh
using both the Create DB Dialog and the SQL Editor, and in both cases the table name is displayed incorrectly.

I then tried this with Java DB, and although I had to quote the table name (you *are* quoting MB characters in the Creat
Table dialog, right?  This is *required*), when I get the error back, the table name is correctly displayed.

So I believe this is an issue with how MySQL is returning the error message -- the table name is not correctly encoded,
or the way we display the error message string has some issue.

I am going to talk to Andrei about this to better understand localization issues in NB
Comment 14 David Vancouvering 2008-03-10 17:23:34 UTC
Removing incomplete keyword, as I can reproduce the issue.
Comment 15 David Vancouvering 2008-03-10 17:36:33 UTC
I'm starting to be convinced this is a MySQL issue.  I am going to work on a standalone JDBC test case that reproduces
the bug outside of NetBeans.  If I can do this I'll log an issue against MySQL, link it to this bug, and close this issue.
Comment 16 kaa 2008-03-11 15:05:24 UTC
Clarification about the attachment (id=57903):
The err message showed on the image appeared after I had tried to create another table with the same name.
The goal was to illustrate that the table was created at the first time and wasn't shown in the list.
On the first attempt to create the table there were no err messages.
I got from dev team that the problem came from MySQL server:
http://bugs.mysql.com/bug.php?id=1406 
Comment 17 kaa 2008-03-11 15:11:20 UTC
I got from dev team that the problem stated in the issue 128732 came from MySQL server:
http://bugs.mysql.com/bug.php?id=1406 
Comment 18 David Vancouvering 2008-03-21 17:01:32 UTC
http://hg.netbeans.org/main?cmd=changeset;node=509dabfe5a56

This change configures the connection to use Unicode/utf8, and my tests show the table appearing now.

The error message for a duplicate table is still incorrect, and I have a JDBC program that reproduces that outside of
NetBeans, and will report this to NetBeans.  This issue is being tracked by issue 128372
Comment 19 kaa 2008-04-30 17:52:10 UTC
The fix looks unstable. I used the following MySQL bundle:

Product Version: NetBeans IDE 6.1 (Build 200804211638)
Java: 1.6.0_05; Java HotSpot(TM) Client VM 10.0-b19
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)

ja_JP locale. I didn't create any project.
The experiment was for MySQL usage itself.

I. Mbyte in table name, Column name was in eng.

If the name of the table consists of 1 to 6 mbyte chars then the table created ok and listed under Tables node. If the
name has more then 6 mbyte chars there are several situations:

1. The name will be truncated by 'E' at the end (see sql_err.JPG) and shown under the 'Tables'
   node. For this kind of table delete action doesn't work and err message with square boxes
   appears. I noticed about it in the issue 128732. This err message looks close to the one from
   MySQL issue that will not be fixed. Also I can't expand table node to see table columns.

2. The name doesn't appear under the 'Tables' node. The table actually was created.
   I executed sql command and it was finished ok:

   a) created  table さささささささ
   b) There are no errors. The table doesn't exist under the Tables node
   c) sql query executes ok. see query.JPG

I didn't find any rule for: 1 or 2 will happen if the name has more then 6 mbyte chars.
It seems to me #1 will happen at the first time when the name has more then 6 mbyte chars.
Then will be case #2.

II. Eng chars only in table name, Mbyte only in Column name - looks ok.
III. Mbyte chars only in table name, Mbyte only in Column name : - the same as #1
Comment 20 kaa 2008-04-30 17:53:07 UTC
Created attachment 60886 [details]
sql_err.jpg
Comment 21 kaa 2008-04-30 17:53:31 UTC
Created attachment 60887 [details]
query.JPG
Comment 22 David Vancouvering 2008-06-19 21:59:29 UTC
I am unable to reproduce. Can you still reproduce with the latest build?
Comment 23 David Vancouvering 2008-06-19 22:03:30 UTC
Created attachment 63135 [details]
Screenshot showing multibyte tables on Windows XP
Comment 24 David Vancouvering 2008-06-19 22:53:13 UTC
Please reopen if you can get me a scenario which enables me to reproduce.

I can't reproduce with this environment:

Product Version: NetBeans IDE Dev (Build 080619)
Java: 1.6.0_05; Java HotSpot(TM) Client VM 10.0-b19
System: Windows XP version 5.1 running on x86; Cp1252; en_US (nb)

I didn't set the actual locale to ja_JP, I am not sure that's needed.

The MySQL server is also set up as you describe below
Comment 25 David Vancouvering 2008-06-19 22:58:21 UTC
FYI, SELECT VERSION() against MySQL connection returns "5.0.51a-community-nt"
Comment 26 Ken Frank 2008-07-10 00:30:24 UTC
Andrey,

is it still a problem; seems like its not seen by developer though he was not
in ja locale according to info below and I do thats important factor,
thus please reopen this if you still see it, since IMO its needed to be
in ja locale to look at issues seen in that locale like these.

also lets make sure mysql functionality in the dbase svcs tab is ok about other
things like create table, add to table, view data, etc

both for table and col names that DON'T have mbyte but do have mbyte data

and for table or col names that do.

And also, if user in win31j locale when run nb, but dbase in utf-8, is it ok ?
or should it be, since if dbase setup for utf-8 would it mean that we can't assume
win31j is ok ?

(that is, since dbase functionality of this area is not related to a given
project, I don;t know if it uses global project encoding at that moment
or the encoding of the locale user is in when running nb.

thus scenarios would be

win31j project encoding and utf-8 project encoding with dbase  encoding of
a. win31j/sjis

b. utf-8/

ken.frank@sun.com
Comment 27 kaa 2008-07-10 18:49:44 UTC
Product Version: NetBeans IDE Dev (Build 200807060002)
Java: 1.6.0_06; Java HotSpot(TM) Client VM 10.0-b22
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)
I'm in Japanese locale running pseudo localized NetBeans.

I started bundled MySQL server and established connection with test database.
Results of my experiment:

1. I created a table using mbyte char in its name. It wasn't listed under the Tables node in the Ide.
   But I was able execute select * query from it without error. I believe it exists but the Ide doesn't show it.

2. Table creation using English names everywhere works properly. Tables were created and shown in the Ide properly.
3. I was able to add mbyte chars and get them back using tables from the step 2.
4. MySQL issues looks reproducible. See 134178 and 128732

I'll re-check red-write operation from a project with UTF-8/win-31j encoding

Comment 28 Ken Frank 2008-07-22 20:12:18 UTC
andrey,

is this still an issue ? in last comment you were going to see about it;
please reopen if its an issue still.

am assuming this is not a restriction or limitation for mysql that has
utf8 encoding for dbase 

ken.frank@sun.com
Comment 29 kaa 2008-07-23 13:08:56 UTC
Reproducible with the following environment:

Product Version: NetBeans IDE Dev (Build 20080721233246)
Java: 1.6.0_06; Java HotSpot(TM) Client VM 10.0-b22
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)

I am in Japanese locale. The table wasn't shown in the list of tables.
Comment 30 Ken Frank 2008-07-23 17:26:43 UTC
David and Andrey, questions for you here:

David,
as per comment below
http://hg.netbeans.org/main?cmd=changeset;node=509dabfe5a56

This change configures the connection to use Unicode/utf8, and my tests show the table appearing now.

is this a setting of encoding of how driver communicates to mysql server or nb ?
am wondering how this impacts if user is running in non utf8 locale and/or if project
is not a utf-8 project ?

(I don't know what value was used/assumed before that change)

Andrey,  just want to clarify that this issue is about the table with mbyte in its name
not showing ok in nb -- but that it is really created and can be seen using mysql query tool
or ui ?

ken.frank@sun.com
Comment 31 kaa 2008-07-25 17:57:13 UTC
Ken, the issue is about a table with mbyte in its name. I didn't see it among listed tables after it had been created.
I decided to re-check again with the latest mysql bundle I have:

Product Version: NetBeans IDE Dev (Build 200807221016)
Java: 1.6.0_06; Java HotSpot(TM) Client VM 10.0-b22
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)
Userdir: U:\tmp\0722_003

I took the following bundle: netbeans-trunk-nightly-200807221016-mysql-51b-windows.exe
The issue is not reproducible there. The table was shown ok under the Tables node.
I think it is fixed now.
Comment 32 Ken Frank 2008-07-25 18:44:53 UTC
this should not be about using mysql/nb  bundle per se,
but about how nb works with any valid mysql.
users might get it standalone, they might get it from amp pkgs for diferent platforms

if this is only about using the netbeans-mysql bundle then that could be separate
issue.

I'm assuming for normal testing that the regular nb trunk installers are being used
and the mysql you are using is either a standalone one or from a amp pkgs
(apache, mysql, php) ?

ken.frank@sun.com 
Comment 33 David Vancouvering 2008-07-29 00:28:54 UTC
Sorry for not responding sooner, I was on vacation.

Given that: 

- I can't reproduce it
- Ken can't reproduce it
- Andrey can only reproduce it with his standalone mysql, and not with the bundle

I suspect that Andrey's MySQL is different somehow.  What version of MySQL are you using standalone, Andrey?

Ken, your questions:

KEN: is this a setting of encoding of how driver communicates to mysql server or nb ?

Yes, it's a setting of how the driver communicates to MySQL

KEN: am wondering how this impacts if user is running in non utf8 locale and/or if project
is not a utf-8 project?

That's a good question, as it hardcoded the connection to use UTF-8.  Is there a way you can test this with a non-utf8
locale/project?  I think you know how to do this better than I.
Comment 34 Ken Frank 2008-07-29 01:24:05 UTC
my comment 0f 6/25 neglected to mention who it was for, sorry.
its for kaa (Andrey):
this should not be about using mysql/nb  bundle per se,
but about how nb works with any valid mysql.
users might get it standalone, they might get it from amp pkgs for diferent platforms

if this is only about using the netbeans-mysql bundle then that could be separate
issue.

I'm assuming for normal testing that the regular nb trunk installers are being used
and the mysql you are using is either a standalone one or from a amp pkgs
(apache, mysql, php) ?


-->? lets use a mysql from both a standalone and an amp pkg, and then explore the different
requirements for encoding setup of the dbase server, connection, etc ie as per the
variables you mention in first entry of this issue, character_set_client, etc.

and find out what is needed to be set to utf8 or some japanese encoding so that
table or col names with mbyte can be used; and so we can communicate about the
expecatations about this to users.

then, you can see if this problem happens using those installations and using
nb database functionality.

(of course assumption here is that its legal in a general sense for such mbyte
to be used in table and col names; David do you know thats true ?

ken.frank@sun.com
Comment 35 David Vancouvering 2008-07-30 19:58:57 UTC
It's OK or multibytes to be used in table and column names as long as the client and server are configured for
multibyte, as they are in this case.

I'm not clear: are you going to do more research, or are you asking me to do more research?
Comment 36 Ken Frank 2008-07-30 20:46:46 UTC
thanks David for the info on table and col names.
I think its the part of config client and server that we need
to make sure for verifying are clear about, since so many encoding
related values for mysql and that the encoding setting can be different
from some other dbases we have looked at in past.  And also want to share
this info with users since am communicating with some on this topic in general -
and the mysql manual docs on encoding, while good and interesting, don't give in one
place a way how to set the various encodings, so we'll continue to research
this part.

ken.frank@sun.com
Comment 37 kaa 2008-07-31 15:49:53 UTC
Product Version: NetBeans IDE Dev (Build 200807301401)
Java: 1.6.0_06; Java HotSpot(TM) Client VM 10.0-b22
System: Windows XP version 5.1 running on x86; MS932; ja_JP (nb)

Re-checked with standalone server on WinXP, Japanese locale using downloaded mysql-5.0.51b-win32 from mysql.com
The issue is not reproducible in this case also. Resolving as worksforme since the same behavior I saw with bundled server. 


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo