This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 187345 - wrong characters in Terminals
Summary: wrong characters in Terminals
Status: STARTED
Alias: None
Product: cnd
Classification: Unclassified
Component: Terminalemulator (show other bugs)
Version: 7.0
Hardware: PC Linux
: P3 normal with 1 vote (vote)
Assignee: ilia
URL: http://aperiodic.net/phil/prompt/
Keywords:
Depends on: 188024
Blocks:
  Show dependency tree
 
Reported: 2010-06-08 17:39 UTC by seebi
Modified: 2014-11-19 13:49 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
screenshot part with wrong characters (22.67 KB, image/png)
2010-06-08 17:39 UTC, seebi
Details
log of a sample netbeans session with zsh in terminalemulator (1.17 KB, text/plain)
2010-06-10 12:12 UTC, seebi
Details

Note You need to log in before you can comment on or make changes to this bug.
Description seebi 2010-06-08 17:39:34 UTC
Created attachment 99906 [details]
screenshot part with wrong characters

I have strange characters in the terminal emulator in my prompt

I use zsh and have a attached an screenshot. The URL given is a link to the prompt script ...

seems the terminal does not allow some characters which are used by modern shells ...

but: I like this feature very much :)
Comment 1 ivan 2010-06-08 19:42:00 UTC
This looks like some ansi escape characters emitted around the 
prompt which aren't interpreted correctly. 

do you customize your prompt in your .zshrc? How?
What is $TERM set to?

Run the IDE with this flag:
    -J-DTerm.debug=output,ops
that will emit hex values of the data being handled by the terminal and
clarify what sequences are causing trouble.
Comment 2 seebi 2010-06-10 12:12:46 UTC
Created attachment 99976 [details]
log of a sample netbeans session with zsh in terminalemulator

> do you customize your prompt in your .zshrc? How?

I use Phils ZSH Prompt from here: http://aperiodic.net/phil/prompt/

> What is $TERM set to?

TERM is set to "ansi" but unsetting it or setting it to "vt100" or "vt220" doesnt solve it ...

> Run the IDE with this flag:
>    -J-DTerm.debug=output,ops
> that will emit hex values of the data being handled by the terminal and
> clarify what sequences are causing trouble.

a part of the log is attached ...
Comment 3 ivan 2010-06-10 21:18:06 UTC
The log attachment seems like some kind of RDF entry ... 

However, from the "prompt" howto I see what the main problem is.
The NB terminal doesn't not yet render "graphics" characters,
the horizontal and vertical lines and corners. That's what the
light and dark blue '?'s are about. 

You could tweak your script and replace them with '-'s and '|'s'.
But I think there are some other issues as well ... I probably 
should create my own prompt from the HOWTO and give it a good spin.
Comment 4 ivan 2010-06-17 05:31:08 UTC
Can you get me the output of a 'locale' command on your system?

That's not part of the problem, but might affect how quickly you
can get a solution.

There are several issues here.

Nb term doesn't yet process and output "graphics" characters, like the
ones at the bottom of http://www.asciitable.com/ or the ones in  
http://vt100.net/docs/vt220-rm/table2-4.html. Getting the escape 
interpreter to handle the escape sequences is trivial.

But I had postponed doing this because these graphic fonts are not
readily available in the common fonts shipped with Java or other systems.
One has to resort to mapping to best approximations in unicode 
or actually drawing the graphics in each displayed character cell.
But this is all doable with medium effort expenditure.

But there is a more fundamental problem. 
- ansi terminal graphic characters use extended (8-bit) ascii characters (as in
  http://www.asciitable.com/ )
- xterm terminal graphic characters use 7-bit ascii characters (as in
  http://vt100.net/docs/vt220-rm/table2-4.html).
- Most modern systems use UTF-8 encoding where extended
  ascii characters are utf "escapes". It is these that Java translates to
  the unrecognised <?>'s
In other words ANSI terminal emulation and utf-8 are fundamentally
incompatible. No wonder that gnome-terminal, konsole and, let's
say terminator, implement xterm terminal emulation where graphic
characters are 7-bit and compatible with UTF-8.

In other words, NB Term has to emulate xterm and not ansi if it is
to serve systems with UTF-8 as the default LANG. Alternately, _once
I get the graphic character code implemented_, setting LANG to 
something like ISO-8859-1 will work with the ansi emulation.

So, to get to a nice solution the following has to happen:
1 interpret graphics escape characters, use unicode approximations to render
   them.
2 Implement an InterpXTerm and make "xterm" as the default emulation mode.
  Fortunately there's enough commonality between xterm and ansi that a fair
  amount of this can be factored.
3 depending on how acceptable the unicode approximations are render them
  by drawing them.

A somewhat orthogonal task would be to create a terminfo entry for "nbterm"
but it will take a long time for it to spread to various systems' terminfo databases.
Comment 5 seebi 2010-06-17 07:00:56 UTC
here is the output of an locale command ... thx

LANG=de_DE.utf8
LC_CTYPE="de_DE.utf8"
LC_NUMERIC="de_DE.utf8"
LC_TIME="de_DE.utf8"
LC_COLLATE="de_DE.utf8"
LC_MONETARY="de_DE.utf8"
LC_MESSAGES="de_DE.utf8"
LC_PAPER="de_DE.utf8"
LC_NAME="de_DE.utf8"
LC_ADDRESS="de_DE.utf8"
LC_TELEPHONE="de_DE.utf8"
LC_MEASUREMENT="de_DE.utf8"
LC_IDENTIFICATION="de_DE.utf8"
LC_ALL=
Comment 6 seebi 2010-06-17 08:59:59 UTC
btw. after closing a netbeans with an active shell, such a process is not killed and runs and runs with 100% of one core!

seebi    26829 26.2  0.0   1956   644 ?        R    10:58   0:02 /opt/netbeans/ide/bin/nativeexecution/Linux-x86/pty /bin/sh



And another enhancements issue: you should start the shell given in /etc/password, not /bin/sh ...
Comment 7 ivan 2010-12-01 05:51:30 UTC
(In reply to comment #6)
> btw. after closing a netbeans with an active shell, such a process is not
> killed and runs and runs with 100% of one core!
> 

This is bug #191834  which has been fixed.

> 
> 
> And another enhancements issue: you should start the shell given in
> /etc/password, not /bin/sh ...

This is bug #187382 which has been fixed.
Comment 8 ivan 2010-12-16 08:38:19 UTC
Changeset: a9740597c452
Author:    ivan@netbeans.org
Date:      2010-11-30 22:12
Message:   Partial fix for #187345 - wrong characters in Terminals
Issue #187345 - wrong characters in Terminals
Comment 9 ivan 2010-12-16 08:42:23 UTC
        Partial fix ...
        A program renders alternative characters as follows. It first needs to
        select the desired font via "ESC [ <n> m". Where <n> ranges from
        10 to 19, 10 meaning "default" font. About the only
        interesting font in widespread use is 11, the so-called DEC graphics
        character set. See the right two columns in
                http://vt100.net/docs/vt220-rm/table2-4.html
        Then the program needs to emit the right character codes. These vary
        from terminal type to terminal type and are described by the 'acsc'
        entry in the terminfo DB:
                cd /usr/share/terminfo/a
                infocmp ansi
        The mapping described by 'acsc' is for _output_. E.g. to render
        the DEC graphic character '~', i.e. ACS_BULLET, the program
        has to emit 0304 for an ANSI terminals and '~' for xterm.

        To that end ...
        ... introduced support for alternative font character attributes.
        These are set by codes 10-19 of the ANSI "ESC [ ... m" escape
        sequence, 10 being the default font.

        These codes are normalized to a 0-9 range and stored in a new
        4-bit-wide FONT field in Attr using Attr.setAttribute() and setFont()
        and subsequently accessed via Attr.font().

        How are various fonts rendered?
        We work on the assumption that unicode will contain a glyph
        for any desired graphic. This reduces the problem to
        mapping a character coming into op_char() to the appropriate unicode
        "glyph". This is done by Term.mapChar().
        mapChar() only handles additional font attribute 1 (aka ANSI character
        attribute code 11). It does so in two steps. First it maps the
        incoming character to the Alternative Character Set (ACS) based on
                http://vt100.net/docs/vt220-rm/table2-4.html
        These are canonical VT-100 characters.
        It then maps the canonical VT-100 character to an explicit unicode
        character representing the desired graphic.

        Still to do ...
        - Term.mapACS() needs to be an abstract method of Interp.
        - Not all VT-100 graphic characters have high fidelity representations
          in unicode. A more sophisticated system would create it's own
          scaled glyphs.

        And all of this hasn't solved the original problem of the fancy
        zsh prompt. I can get almost everything right except that there
        are too many spaces on the second line of the prompt causing
        the "date" portion to wrap.
        But I think this has to do with the terminfo entry for 'ansi'
        or a curses bug on my FC12. Will have to check on some other
        platforms after pushing.
Comment 10 Quality Engineering 2010-12-17 06:19:01 UTC
Integrated into 'main-golden', will be available in build *201012170001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/a9740597c452
User: ivan@netbeans.org
Log: Partial fix for #187345 - wrong characters in Terminals
Issue #187345 - wrong characters in Terminals
Comment 11 Leonid Lenyashin 2012-10-30 21:09:34 UTC
There was no activity on this issue for quite a long time. We apologize that the issue was not addressed so far due to lack of development resources. We might not have time in near future to fix this problem, so it is closed as WONTFIX.
If the issue is still critical for you please do not hesitate to REOPEN it.
Thank you for using our product and reporting bugs. We are really sorry that we were not able to fix this issue timely.

Regards,
CND team.
Comment 12 ivan 2012-12-19 08:29:03 UTC
Update:
I've added ability to emulate "xterm" to Term.
But this functionality is not yet %100 ready to make "xterm" be the default instead of "ansi".

I've also gotten the Alternative Character Set (ACS) (aka graphics characters) rendering
working for "xterm"" emulation.

However, when $TERM is set to "xterm" all applications which use curses and the
terminfo database end up sending some sequences that Term isn't
prepared to handle yet. So it would be premature to make "xterm" emulation be
the default until we can handle all those sequences.

This _doesn't_ mean that we need to handle _all_ official xterm sequences per
    http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
but only those that are visible in "infocmp xterm".