Created attachment 65510 [details]
threaddump

Created attachment 65511 [details]
threaddump

maybe the java task scanner should wait till classpath scanning is finished at startup - reassigning to java team for
evaluation

This is still the same issue.
1. Java Source module does not do any IO/CPU intensive things in parallel with classpath scanning.
2. This is JPAVerfication Task and the other J2EE tasks. Cannot recall their name.
3. Task list should not start anything in paralel with classpath scanning, no matter if it is related to java or not.
Classpath scanning is IO intensive operation and starting another IO intensive operation in parallel with scanning is
simply wrong. Simply don't do it :)

We would like to help you with it. Ask for help.

> We would like to help you with it. Ask for help.
please help:)

*** Issue 140312 has been marked as a duplicate of this issue. ***

Well. We have SourceUtils.isScanInProgress () and waitScanFinished() ...
   

thanks for the tip. is there any way to find out that class path scanning is about start or has just started?
another problem is that i don't want task list impl to depend on java...

isScanInProgress (), waitScanFinished() and runWhenScanFinished are the only methods we have.

The scanning done by task list significantly affects post-startup (or post project open) performance even if done 
after the CP scan. Already without task list scanning this was one of the biggest performance issue in the IDE. This 
is something we should not make worse - which is happening now when the task list is shown in the default IDE setup. 
Can you please turn off the addional scans in the task list's default setup (leaving on users to turn them possibly 
on)? This is a very minimum we can do to keep the performance acceptable. (Assuming the java errors/warnings for which 
this unadvised change was mainly done can be obtained from java infrastructure without additional scan.)

Yes. Java errors/warnings are available "for free".
As I already mentioned, the bad tasks comes (IMO) from JPAVerificationTask and EJBVerificationSomething.

i've turned all task providers off by default except for java tasks as tpavek suggested. so there won't be any task list
originated io during the initial class path scan: 973ce6c0aaf2


however i still think that the task list should pause itself when class path scanning is running (or any other io heavy
activity). but since there's no api for this it'll have to wait till the next release

Your change caused module tests to fail, so I rolled it back (#ce110b8d2259).

fixed again (with fixed test)

659c12fd7360
615b48465c93

I don't think this solution is okay for non-Java projects.

Here's what I wrote on a mailing list:

Hmmm.... Can we at least make this project type sensitive?

If you're using the Ruby distribution, and you open a Ruby project, you should get to see the Ruby errors in your
project.  Furthermore, for Ruby we show hints as well, not just errors as is the case for Java. I think this is a really
useful improvement (and I always build the contrib/javahints module as well to get similar behavior for Java).

Without this, you have to go and open each and every file in your project to see where your semantic errors are. Other
IDEs such as Eclipse manage to do this. I think we should consider building a cache into the tasklist such that it can
come back up at startup immediately populated with results. A refresh button can be used to cause it to flush the cache
and regenerate everything. I don't think anybody will be confused if they exit the IDE, delete/add/edit files, restart
the IDE and they see somewhat stale results in the tasklist - they'll just hit Refresh.

It's great that our IDE automatically detects edits outside the IDE and reloads the current editor etc., but for
something like the tasklist, I don't think anybody will fault us for caching results. Certainly doing lots of additional
disk processing at startup is not something people will prefer.   And I think a cache is vastly superior to just
disabling everything except Java errors.

-- Tor

Any comments on Tor's suggestion? The current state when tasklist is opened by default but does not show anything is not
acceptable. For now, let's reenable the scanning of errors (at least for GSF - that's the most important area), and then
deal with performance issues, if any. Either the way Tor suggests, or some other way.

Another solution is to hide the tasklist by default (as was the case in 6.1), but I'd rather not do that.

I agree with Tor and Petr. From PHP point of view, it doesn't have sense to open task list by default, if it doesn't
show anything from PHP files by default.

I agree with Tor and Petr. From PHP point of view, it doesn't have sense to open task list by default, if it doesn't
show anything from PHP files by default.

*** Issue 145815 has been marked as a duplicate of this issue. ***

> For now, let's reenable the scanning of errors (at least for GSF - that's the most important area),
> and then deal with performance issues, if any.

If any? This issue was reported because having all the scanning running made the IDE unusable. Now you suggest to 
return back to that state? I don't agree with this. First solve the performance issues, then reenable the scanning of 
errors.

Speaking about GSF only, I understand the current state is not good for GSF-based projects from usability point of 
view. If you can make the GSF scanning not affect java-based projects, then fine. E.g. by making it project type 
sensitive, as Tor suggested.

I have checked in a fix to GSF's task scanner, such that when the task-scanning scope is project wide, it will only scan
gsf-enabled projects.  In particular, this means that if you open a plain Java project, it will not scan for relevant
source files.  (Of course, that means that any Groovy errors or hints will not be scanned, since as I understand it,
Groovy uses plain Java projects).

If you however have a Rails project, a Scala project, a Java Web project, or a PHP project, the scanner will kick in as
usual.

Also, if you have set the scope to Current File, it will check with GSF and if the file is a GSF file, you will get
scanning errors. Thus, even if you have JavaScript file in a Java project, if you open it, you can see errors (for
scope=current file). But it won't look for JavaScript files at the project scanning level.

This was changeset 6b4081992fa8.

Can the GSF task provider be re-enabled now? Or is there something else I should do?

Also, Martin - is there a quick and fast way for me to check if a Java project is "Groovy" enabled? Does it add some
project level property I can check? It would be nice to also include GSF scanning for these projects if there's
something fast and simple we can check in the case of Java projects.

i'm working on a fix that will pause task scanning when classpath scanning is in progress. so all providers will be
enabled by default then.

> Also, Martin - is there a quick and fast way for me to check if a Java project is "Groovy" enabled? Does it add some
> project level property I can check? It would be nice to also include GSF scanning for these projects if there's
> something fast and simple we can check in the case of Java projects.

Only friendly used org.netbeans.modules.groovy.support.spi.GroovyFeature.isGroovyEnabled() which is used to enable Groovy refactoring only for Groovy-
enabled projects (Grails and Java SE with Groovy enabled in properties)

i've created a short wiki page describing my proposal for 'project type specific task scanning':
http://wiki.netbeans.org/TaskListScanningExtension

pls let me know if that solution is good enough for everybody. as there are some API changes needed, i'm turning this
issue to fast-track API review.

note: pausing the task list when classpath scanning is in progress is still part of this fix.

I wonder if it isn't best to just rewrite all the Task Providers to instead be project sensitive.

It's hard to write declaratively whether a task provider should be enabled for a project or not. If we use project
lookup instead, then a project can dynamically determine whether it gets task scanning.

This is roughly what I did in the GSF task provider on Friday: When the setScope() method is called on the TaskProvider,
it checks the projects in the scope, and decides whether they apply. This means the decision of whether a task provider
applies to a project type is made on the task provider side, rather than on the projects side. That seems like the right
place to have the logic (and as I understand it is also what you were proposing in the wiki, except you made it
declaratively rather than computed by code).

The advantage we have by using lookup, however, is that this should be more flexible. For example, take Groovy. A
standard Java project (which according to our XML files should disable everything except the Java task provider) can
also be used for Groovy. In that case, the Groovy integration can augment the project lookup with additional classpath
information (for GSF), which will make GSF's task provider kick in because it suddenly finds additional project context.

In short, I think we should make all Task Providers enabled by default, but we should visit each and every one of them
and MAKE SURE they perform very minimal computations immediately to decide if they apply. In most cases, this means
checking if the projects in the lookup are of the set of projects the task provider cares about. In case the scope is
file scope, it means looking up the project containing the file and making the same decision. But in some cases, a file
scope can be used to see if just the filetype matters. This means that for example in a Java project, which normally
doesn't have the GSF task provider running, if you open a JavaScript file, you -still- get tasks provided because GSF
knows that it should run the GSF task provider on JavaScript files that are explicitly opened, no matter what the
project type is!

I think this gives us more flexibility than something based on static project type registration, and I don't think it
has a performance downside if we perform the "appliesTo" logic to all the task filters. 

Some more comments on the proposal:
- I think the enabled state of the ToDo provider should be the same in all projects. If this provider is fast enough,
then it should be enabled all the time. If not, then it should be disabled all the time.
- The Default filter should be language-neutral. E.g. when I use the IDE for PHP and C++ development, then a Default
with only Java scanning enabled in not the sensible default. Tor's proposal addresses this - all providers are on by
default, so they are treated symmetrically. 
- Tor's proposal also has advantages for Maven project - I can imagine that the Maven project could expose the
information on whether it has the webapp archetype, and the GSF provider could enable/disable scanning of JavaScript,
CSS, JSP etc. based on this info.

And one question:
> note: pausing the task list when classpath scanning is in progress is still part of this fix.

Will this apply also to GSF scanning? I think the situation is analogous to Java classpath scanning, so should the
tasklist be also paused when GSF scanning is in progress?

I would extend pjiricka's concern about Maven projects to apply also to freeform and automatic projects. For these cases
it does not make much sense for the project to declare a particular filter. It would be better for scanners to decide
whether or not they apply to the project based on their inspection of the project's services, e.g. source roots and
their classpaths, or existence of a web module.

well, there are no API changes required if we want to let the task providers to decide whether they apply to project(s)
in current scope or not. the project already are in scope's lookup so it's up to the implementors of task providers to
add some project sensitive logic.

re 'pausing task scanning while classpath scanning is in progress' - it applies to all Java classpath scanning only and
as there's no public API for it there'll be some hack from task list impl in ide cluster to java cluster. if there's an
API or hook to gfs scanner, i can plug in there too.

One more thing - any comments on Tor's suggestion to cache results across IDE sessions, so the results come up very
quickly after startup? Is this desirable/feasible?

> One more thing - any comments on Tor's suggestion to cache results across IDE sessions, so the results come up very
quickly after startup? Is this desirable/feasible?

it is feasible but not in 6.5 time frame

after a bit of testing (http://wiki.netbeans.org/TaskListScanningExtension) it seems safe to turn all task providers on
by default except for EJB and JPA task scanners which slow the task list down the most. they are both file-based
scanners and basically compile each Java source file again when enabled (they use runUserActionTask()). when those two
scanners are turned off the task list is reasonably fast.

Thanks many times for the investigation, Stando. I filed separate issues to track the slow speed for EJB and JPA: issue
146890 for EJB, and issue 146891 for JPA.

So does this all mean that we can get the GSF task provider enabled by default again, or is there anything else I can do?

> So does this all mean that we can get the GSF task provider enabled by default again, or is there anything else I can do?
yes GSF task provider will be on by default (unless there are strong objections from somebody else). i'm just waiting
for java team guys to implement a hook into classpath scanning so that i can close this issue.

fixed - the task list will pause all file access when java classpath scanning is in progress. java, gsf and todo task
providers are turned on by default.


ebeac74c369d

Integrated into 'main-golden', will be available in build *200809250201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/ebeac74c369d
User: S. Aubrecht <saubrecht@netbeans.org>
Log: #141514 - pause task list when classpath scanning is in progress, turn java, todo and gsf task providers on by default

Hi Jan,

could you please verify this bug because performance team is really interested in fix results.

verified. The performance in the recent daily builds was improved.