Created attachment 110294 [details]
nps snapshot

Created attachment 110409 [details]
nps snapshot

Created attachment 110410 [details]
nps snapshot

What db is used if any?
Is it always reproducible for some use-cases? if I got it right persistence completion waits in performWhenScanFinished, but the only comment regarding the issue " no scanning"< I may miss something also at initial evaluation.
Is UI blocked/frozen?

Sorry, not just sitting and waiting for an answer, so after a week it's hard to say what happened.

While saying, there was no scanning active, I've noticed NB sometimes *seems* to scan without showing the progress bar.

BTW, cannot open npss files into the profiler, because it is not "Nimbus compatible".

ok, thanks, anyway, I saw unusual 'npss' also but after rename to nps was able to open in nb. also I hope other reporters can provide some more information.

(In reply to comment #6)
> ok, thanks, anyway, I saw unusual 'npss' also but after rename to nps was able
> to open in nb. also I hope other reporters can provide some more information.

npss is for IDE snapshots, while nps is for "regular" ones.

(In reply to comment #6)
> ok, thanks, anyway, I saw unusual 'npss' also but after rename to nps was able
> to open in nb. also I hope other reporters can provide some more information.

Well, while I cannot open the npss file (while I should be able to open it as IDE snapshot, which doesn't work), the nps works. In my case, it's waiting for about 33s in WinXP file system, just for testing, if a file exists.

What does Windows cause to be waiting there??? Probably there's some tool to detect that?

As I've just used code completion, I even doubt, that there's the necessity to test for file existance synchronously.

initially it's executed by request processor, and the entire task is asyncronous, it may not have any sense to split completion internally in more asynch tasks. I don't see yet 30 seconds to check file existence but in code completion it's 30 seconds to try to acquire lock in javac as I see, it may be caused by background scanning and may be valid (but may be not).
Main question (for me as it affect severity of the issue) if UI was frozen or not and if proper "scanning" message was shown in completion window etc.

on other side if there is any scanning it should enqueue task and quite the method quite fast and it there is no scanning it shouldn't wait for so long, may need some help from java team or it may be javac issue also.

Tomas, Jan, can you take a look if it's an issue in parsing etc.?

ok, found File.exists() call in Parsing&Indexing loop and it takes 32 seconds, still not sure if it's java/parsing issue but it's good to get some feedback.

(In reply to comment #11)
> ok, found File.exists() call in Parsing&Indexing loop and it takes 32 seconds,
> still not sure if it's java/parsing issue but it's good to get some feedback.

Probably it's a problem of cache design:
My var\cache\index contains 15305 files in 15384 folders
with a total size of 115 MB (121.235.609 Bytes)

A database (e.g. using javadb) might solve these locking problems.

Seems as OS problem, the fs dir operation does not depend on the size of nodes in subfolder.
>A database (e.g. using javadb) might solve these locking problems.
There is no locking it's optimistic. The javadb was something like 10 times slower :-)

See(In reply to comment #13)
> Seems as OS problem, the fs dir operation does not depend on the size of nodes
> in subfolder.
> >A database (e.g. using javadb) might solve these locking problems.
> There is no locking it's optimistic. The javadb was something like 10 times
> slower :-)

Seems a Windows-specific problem (when opening folders with many files in the explorer, it takes some time, too). That's why I'd prefer sth. like a big file containing everything, such as databases usually use it, so windows wouldn't know anything about opening files. Otherwise NB will be usable for Linux and similar OSs. :-(

It seem as the "getBooleanAttribute" issue. We have some issue where we duplicate these problems.
I don't know much about NTFS but it seems that it does not even depend on size it can happen even for folder with single file. It seems like some transaction update which sometimes blocks the FS metadata operations. But I don't know it's only my guess. I will try to find more about this problem.

Tomas, is it more parsing&indexing issue on nb side in your opinion or there still may be some improper usage from persistence code? If first I'm going to reassign to parsing or may be editor/completion.

It should be set as a duplicate of gBA umbrella.
I will try to find it and let you know.

Tomas, any news? What is gBA?

gBA = getBooleanAttribute

thanks. can't find any umbrella myself but see reference to getBooleanAttribute in issue 187715 but it's assigned to php.

I've asked about the umbrella issue and it's not yet created.
The policy is: If there are few calls of gBA which took long time (sec per one call) you can close the issue.

ok, just put to P4 before close, not sure what is better in this case.

Most latest reports seems to be caused by long time in completion during scanning. If it's "getBooleanAttribute" issue wonder why it's mostly jpa completion issue, may need to reevaluate.

at least back to P3 for now, 
and these is thread on netcat with "slowness on autocompletion" subj

ok, some more investigation, there is two query in code completionm one is JavaCompletionQuery and second one is JpaCompletionQuery, both are present in nps, both futuretask.get() usage and it wait for parsing to complete. 
what may be good is to optimize api to combine this tasks in one "get()" call instead of two separate, but not sure it will fix the issue, just jpa query may be first and it cause report to be jpa report instead of java. It seems out of my control to fix or at least improve right now, may be something can be done in parsing (longest reports have 1000+ calls to getBooleanAttribute) is there really nothing we can do?

Triggered by Sergey's suggestion the source of the slowness is in JPA I closed my model project (which is the only project with a jpa dependency and entities). Slowness in autocompletion seem to have disappeared after that.

It is very hard to reproduce this issue in a controlled project structure ( I tried with a pretty extensive dummy project)

I can reliable reproduce it with the AgroSense project
https://hg.java.net/hg/agrosense~core

Slowness does occur with the model module open, and doesnt with the model module closed.

In real it shouldn't depend on project opened with jpa usage or I don't see such dependency, I also tried something similar having slow completion (in this case long "please wait"), tried to disable jpa and restart and got it fast but after check if jpa is disabled I found jpa is enabled, so it's not proper test in my case. 
I'lltry suggested project, how to checkout/build it?

> I'lltry suggested project, how to checkout/build it?

Commandline (assuming installed mercurial and maven and both on path)

hg clone https://hg.java.net/hg/agrosense~core
cd agrosense~core
mvn clean install

You can also do a clone from netbeans.(Team->Mercurial->Clone Other)
or open the cloned the root project after commandline clone(+open required projects) in Netbeans and then clean build the root project (AgroSense - Netbeans Platform Application)

In replay to comment #25.
>nps, both futuretask.get() usage and it wait for parsing to complete. 
>what may be good is to optimize api to combine this tasks in one "get()" call
>instead of two separate
The tasks are waiting until some other priority task ends. Combining them makes no sense. If they operate on the same file none of these tasks do parsing.

>It seems out of my control to fix or at least improve right now, may be something can be done
>in parsing (longest reports have 1000+ calls to getBooleanAttribute) is there
>really nothing we can do?
The real problem is very slow IO (1 invocation of gBA on the first snapshot takes 33s). Obviously parsing api cannot help.

Does the JPA code completion correct cancelling of the Future when the CC is closed?

Can you point me to the nps which has 1000+ calls of gBA? I will look if it can be optimized.
But nothing can be done with 1 invocation of gBA on the first snapshot takes 33s.

Thanlks for your help, I'm looking into http://statistics.netbeans.org/exceptions/exception.do?id=549108
It's also Linux.
I can see 1165 calls to getBooleanA, but may be i'm not yet good in nps analisys, as If I looks from tree view i can see one only.

Thanks!
I will look at it and let you know.

haven't look at possible problem with cancelling, but there may be the issue even with first completion invocation and also as I assumed before, I test and see JPA query is called first and Java query is called second, I can't find any position attribute in CompletioProviders registration but it's likely if there is a way to change order the issue can be switcched from JPA into Java (of cause it have no sense, just some theory)

added one cancelled check, latest report is just 2 sec also, previous biggest one is with getBA > 13 sec, may be heavily loaded fs etc or just win issue, to P4 once again.

Integrated into 'main-golden', will be available in build *201204240400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/2ff6b4fbb3ec
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: #201476 check if cancelled is added

2 latest reports are different a bit and contain jpa completion + some other.
latest one have 4.7 in jpa, 2.7 in javadoc, 2.3 in websvc,2.2 in java completions from 12 seconds total. it seems dfferent from previous cases, I'm closing this one as suggested above + to get new statistic in new issue. also nps show more then 5 seconds in getBolean so it may be the same but in differnt context.

change to invalide as old snapshots are not valid fr new 7.3 code in my opinion. latest report is separated from duplicates list.