Created attachment 7831 [details]
Full thread dump

Trung suspects jar filesystem.

JarFs issue 27416 was fixed.

I can not assure that it has eliminated this problem entirerly because
of issue non-deterministic nature caused by multiple thread contention.


Another observation from stackstace is that it was provoked by link
creation. We can try to share result of ClassElement.forName() query
among link creation methods.

Svata does it make sense? Should not it be cached at
ClassElement.forName() implementation level? This MRU cache need to be
invalidated during classpath content change.


Gabo perceived perfomance is close coupled with usability (and
vice-versa), what do you think about:

Do we really need to create hyperlinked content? Should not we
simplify the javadoc hint content by stripping out:

  - class name link in header

  - method signature (it's a part of completion view 
    anyway) which needlessly made user to scroll
    horizontaly.

  - other links that are accesible only by mouse
    which is unlikely to happen during typing
    code and asking for short hit

ClassElements returned from forName() are already cached at their
origin's DataObject (SoftReferenced). When someone asks for the same
ClassElement, the cached instance is returned, obviously.

The parser itself has some symbol caches, too, which are occasionally
flushed (precise cache management would be quite expensive). The
stacktrace clearly shows that either the source which held the
ClassElement asked for was not yet parsed at all or the caches are
empty - the parser seems to be in process of parsing referenced symbols.

Do you think there should be a specific MRU cache for CE.forName() ?

Sorry, I looked for caching into ClassElement code only. It starts
with a loop. I naively expected a cache check before the loop.

forName() caching does not solve first call problem that you
identified from stacktrace.

Besides is Java source parsing thread really handling the forName()
request? Is not it a "background" parsing? I do not know what other
requests the queue can contain because no other thread is waiting for
RP. Still it takes a long time to wait to this task.

I have been looking for the right place to introduce a new cache in
entire stack. There was added cache to JarFS that should speed up
parsing. java.ClassElementFinder is cached too. Then it does not make
sense to add a cache at editor.java.NbCompletionJavaDoc. Even if it
can ask for the same ClassElement multiple times.

I see two options. RP queue contains some higher/same priority
"garbage" or the full thread dump shows a case where editor javadoc
hint content contains a lot of links to unrelated classes.

The second option is not right for JFrame().| example.

For the first option I miss at the full thread dump a thread that
pollutes Java source parsing queue. However I'm able to reproduce
queue pollution by mounting new FS. The queue is then flooded by
"background" editor's JCUpdater requests.

Svata, I would like to ask you if I can somehow properly prioritize
parsing requests (indirect in this case)? 

RequestProcessor.Task priority inheritance came to my mind. It means
that RP with priority queue is needed. Posters to such RP can
explicitly adjust/ceil post priority assuming they know their own
priority (current thread priority can be used as it's the same as RP
task priority being currently executed). Java module is said to use
such a queue internally.

You are right, the parser might be processing some request posted
earlier. However background parses are posted with MIN priority, while
requests, which synchronous or are waited on have priority bumped to
MAX (or they should, this was the original intention, bugs possible).
It's implemented (now) so that SourceElement.prepare() produces
(better: should produce) a low-priority task, while
SourceElement.getClasses() will issue a high-priority one.

Are background parse requests available to clients? I'm affraid that
all client requests (issued via srcmodel) are MAX ones.

Created attachment 8060 [details]
Low priority JCUpdater blocks high priority JavaDoc

I have to adjust my guess. In JCUpdater thread dump case, the queue
cannot be flooded by JCUpdater requests. It also waits for Java
parsing results.

Two new hypothesis I'm going to investigate:
  - JVM level threading causes Javadoc thread
    starvation (I'd expect it on Linux where
    priorities are not mapped properly but on
    Windows?)
  - the queue is flooded by Java module internal
    "background" requests

The first case can be eliminated by using shared RP (with priority
queue) instead of two independent threads.

The second case also leads to RP with priority queue and assuring that
all queue posters use/ceil proper priority.

Created attachment 8144 [details]
JVM Thread Priorities Test

I inspected several JVMs for their background threading support:
                       
Linux Sun JVM 1.4.1             no
Linux Sun JVM 1.3.1             no
Linux Blackdown JVM 1.4.1 beta  no
Linux Bea JVM 1.4               --  
Linux IBM JVM 1.4               no
Solaris Sun JVM 1.4.1           no
MacOS-X Apple JVM 1.4.1-a7      yes
Windows Sun JVM 1.4.1           yes

no - behaves like simple round-robin 
     scheduler (ignoring priorities)

See also jre bug 4799163 on Thread.yield() behaviour.

Another Java bug database entry of particular interest:  4813310.

It blocks issue #27785.

Running on stock Linux 2.4.20 I observed proper Thread.yield()
functionality.

I think that we can add explicit Thread.yield()s to all threads that
are expected to run on background. It works on most platforms and JVMs.

Created attachment 9250 [details]
Intensive explicit yielding mini-benchmark

Shouldn't we implement the entire OS scheduler in NetBeans to
eliminate JDK thread functions breakage at all ?

Are you able to code preemtive timeslicing scheduler in pure Java 1.3
SE? I'm afraid.

Let's give Thread.yield() change. Correct Java app must use it anyway
because JVM does not guarantee preemtivity at all.

OK, let's pollute all java apps - not only NetBeans - with
Thread.yield because JDK team is unable to produce correct threading
mplementation on OSes that are officially supported and have thread
support (such as Linux happens to be). 

Will you also patch long computations in third-party libraries to use
Thread.yield, too ?

One additional suggestion (less sarcastic): Please add some text to
o.o.util.RequestProcessor documentation (and possibly to some
threading docs) that explain that a developer is obliged to
occasionally call Thread.yield() *in addition* to posting priroitized
tasks to a RequestProcessor.

JLS 17.12 Threads

Every thread has a priority. When there is competition for processing
resources, threads with higher priority are generally executed in
preference to threads with lower priority. Such preference is not,
however, a guarantee that the highest priority thread will always be
running, and thread priorities cannot be used to reliably implement
mutual exclusion.

I'm missing there modal verb "must" and implementations take
"advantage" of it. I'm also missing "preemptive scheduling" keyword so
it's not guaranteed.

I prefer to update NetBeans threading model document to cover
background thread coding requirements (or guidelines?) rather then
covering it by RequestProcessor comment update only.

I see one risk in yield()ing. Too excesive yielding is
contraproductive, it slows down whole system. Unfortunatelly proper
level cannot be deternined statically at design time. The faster
runtime the more penalization it gets (relative to slow systems).

May be:

  MAX_LATENCY = 50;
  last_yield = S.currentTimeMillis();
  while() {
    nextstage(); // on recommended HW < MAX_LATENCY
    slice = S.curremtTimeMillis() - last_yield;
    if (slice > MAX_LATENCY) Thread.yield();
  }

unfortunatelly it may miss preemptive switch. And it works only by
chance if thread priorities are ignored. Priorities are a must
(yielding may help with them). Preemptivity is nice (as JVM takes care
of "yielding").


Does anyone have an idea how to improve scheduling latency of JLS
constrained application? I can recall only heuristics approaches such
as above but no reliable solution.

Corrected background yielding pattern:

/**
 * Call occasionally from nice (portable) background
 * thread.
 */
public void backgroundYield() {  
  MAX_LATENCY = 50;
  // last_yield is a ThreadLocal
  last_yield = S.currentTimeMillis();
  while() {
    nextstage(); // on recommended HW < MAX_LATENCY
    slice = S.curremtTimeMillis() - last_yield;
    if (slice > MAX_LATENCY) {
       last_yield = S.currentTimeMillis();
       Thread.yield();
    }
  }
}

Hui Huang pointed out that yield is a double purpose method. He
uncovered that explicit yield is also required to signal that current
thread has nothing to do and really wants to give up CPU (before
timeslice expires). I need to test it however. It could play role for
tiny Runnables.

It's getting more hairy:


Doug Lea:

We don't, but upon checking out the code example on the bug site,
there definitely seems to be a problem in how at least Solaris and
linux don't deal at all with priorities.  Priority hints are the most
you should need/do in cases like this. As a rule of thumb,
Thread.yield is useful only in spin loops, and not even then if you
can avoid it.

Interestingly, on some runs of that code on linux, the background
thread runs FASTER than the foreground when the yields are used, but
not if they are commented out.  And setting priorities seems to have
no effect at all.



Hui Huang on Linux:

It's hard to believe missing thread priority can cause 30 sec delay
when loading some contents. How many background threads do you have?
And what about the CPU useage while you are waiting?

While Linux scheduler doesn't accept thread priorities, it does try to
give interactive threads a higher priority over cpu hogs. This also
means if your background thread spends most of the time sleeping, it
will be more likely to get scheduled on CPU once it becomes runnable,
because kernel will think it's an interactive thread. But I would be
really surprised if it can block other threads for 30 sec. After all,
kernel will figure out it's not interactive after a few slice, and
every time slice is only 10ms.

Another potential problem on Linux that could cause serious delay is
its mutex granting mechanism. Linux pthread uses FIFO queue to hand
an unlocked mutex to the next waiter. While fair, it's poor in terms
of throughput. If you have many threads in the program, you might want
to try the latest Redhat 8.1 beta (phoebe-3) that is now available on
many RH mirrors. It fixed several known issues, including the mutex
problem. 

Update: Thread.yield() is a workaround that is not guaranteed to be
platform neutral. We should solve problem roots rather that looking
for workarounds.

In this case the real problem is poor JVM implementation quality
(possibly caused by poor threading model in underlaying OS) - JavaBug
4813310.

Created attachment 9571 [details]
JNI package for linux nice(2)

You can try invoke attached Kernel.nice() from your background
threads. Microbenchmarks report better behaviour comparable to Windows
and MacOS-X JVM implementations. It works only with Sun's JVM 1.4.1
for linux.

I think you're way off track here.
Background processing works quite well even without priorities
and microbenchmarks will tell you nothing usefull here.

We should concentrate on the things that really block the editor.
One of them may be java.codesync support which likes to run in
background and excercises e.g. filesystems heavily, which can slow
down any other FS access.

Anyway, there may be other things like some ugly request to java
parser that will make it busy for several seconds, because priority
queue won't help you once it is already running (in one request).
 

Well, it was about threading. I spotted several times that some
insignificant thread was occuping CPU blocking completion threads.
Hence the microbenchmark.

You are I right that the microbenchmark does not cover the most
required quality parameter - latency.

BTW: completion internally uses *low* priority parsing, usinge the
prepare() call:
org.openide.src.SourceElement.prepare
org.netbeans.modules.editor.java.NbJavaSyntaxSupport.refreshClassInfo

Editor infrastructure is completely rewritten and improved since then.