Please use the Apache issue tracking system for new NetBeans issues (https://issues.apache.org/jira/projects/NETBEANS0/issues) !!
Bug 120582 - [60cat] Initial Subversion scan freezes whole machine
[60cat] Initial Subversion scan freezes whole machine
Status: RESOLVED FIXED
Product: versioncontrol
Classification: Unclassified
Component: Subversion
6.x
PC Windows XP
: P3 (vote)
: 7.0
Assigned To: issues@versioncontrol
issues@versioncontrol
: PERFORMANCE
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-10-31 08:39 UTC by tboerkel
Modified: 2010-09-28 12:40 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Stacktrace of OutOfMemory Exception (3.28 KB, text/plain)
2007-10-31 08:41 UTC, tboerkel
Details
ThreadDump (42.43 KB, text/plain)
2007-10-31 08:41 UTC, tboerkel
Details
dir-props after killing NetBeans during scan (2.77 KB, text/plain)
2007-10-31 10:47 UTC, tboerkel
Details
log (22.77 KB, text/plain)
2007-10-31 10:56 UTC, tboerkel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tboerkel 2007-10-31 08:39:27 UTC
[ JDK VERSION : 1.6.0_03 ]

Let me first say, that this is not an artificial test case, it's a
real life problem.

We have .class files and .java files in the same directory (like in
NB 3.x) and we cannot change that for now.

When a package with lots of classes is being opened the first time,
NB calls svn.exe very often (I guess to find out about the .class
files that are not in the repository, I see PROPGET in the command
line with Process Explorer) and this somehow causes the process
csrss.exe to need lots of CPU. NetBeans is frozen while doing this
and often even the whole computer freezes.

This whole process takes *very* long (10 minutes or more) on a very
large (500 sources) directory. Sometimes I even get OutOfMemory in
the log and I have to kill NetBeans (attaching Stacktrace and
ThreadDump of that situation).

After that, .class is being added to the ignores property of the
directory. That is supposed to be a feature, but I would say it's not
needed.

Outside of NetBeans, we work with Tortoise and there it's no problem
at all to have unversioned .class files together with versioned .java
files.

Workaround is to add .class to the global ignores list of the
Subversion command line client.

BUT: What if someone does not know about this (very likely) or
forgets it (likely)?

This issue is even more important since Subversion support is on by
default in NB 6.0.
Comment 1 tboerkel 2007-10-31 08:41:10 UTC
Created attachment 52091 [details]
Stacktrace of OutOfMemory Exception
Comment 2 tboerkel 2007-10-31 08:41:39 UTC
Created attachment 52092 [details]
ThreadDump
Comment 3 tboerkel 2007-10-31 08:43:17 UTC
This is Beta 2.
Comment 4 Tomas Stupka 2007-10-31 09:10:34 UTC
could you please attach your whole messages.log file. if possible the one with the OOME.

thanks
Comment 5 tboerkel 2007-10-31 09:20:43 UTC
In the meantime, I restarted NetBeans and the log is gone. The OOME is not in messages.log or messages.log.1 or
messages.log.2.

Should I try to reproduce the whole issue and then attach the log?
Comment 6 Tomas Stupka 2007-10-31 09:23:21 UTC
> Should I try to reproduce the whole issue and then attach the log?
that would be nice. however, even the messages.log you got would help. I want to now as much as possible about your setup.

thanks
Comment 7 Tomas Stupka 2007-10-31 09:47:31 UTC
tried a simple scenario on linux 

1.) versioned project, class set as ignored in system settings
2.) 1000 .java files in a package
3.) externally created 1000 .class files in the package
4.) started nb, opened the package and it worked for me:(

so i will give it a try on windows, but there seems to be a bit more to it ....

how exactly does your usecase look like?

- you have a versioned project with many (~500) .java and .class files in a package
- your System Settings/Ignored Files are something like ^(.*\.class|CVS|S ...
- how are the .class files created - is it a buildin nb or an external action
- ...?

> When a package with lots of classes is being opened the first time
- this means the first time after a nb start or the first time after the .class files where created?

- could you please attach or send over the .svn/dir-props file from the relevant folder
Comment 8 Tomas Stupka 2007-10-31 09:52:27 UTC
ok, i restarted nb and it took quite a long time to open such a package...

i still would like to see the .svn/dir-props file
Comment 9 tboerkel 2007-10-31 10:28:16 UTC
The usecase is exactly like you describe. The sources are compiled by NB (changed build.classes.dir to the src dir).

This is my ignore files system setting:
^(.*\.class|CVS|SCCS|vssver\.scc|#.*#|%.*%|\.(cvsignore|svn|DS_Store)|_svn)$|~$|^\..*$

Before opening the folder in NB, I do not have dir-props at all.

I tried again. This time I waited less than 10 minutes before killing NB. The OOME did not occur, nothing interesting in
the log. So I guess, the OOME was a side effect.

The main problem is that svn.exe seems to cause csrss.exe to use lots of CPU and freezing the machine.

I am attaching dir-props after killing NB, so only some of the .class files had been added to the ignore list at this point.

In essence, I think NB simply should not add each and every .class file to the ignore list. If it absolutely must do
something to the ignores list, then why not add *.class?
Comment 10 Tomas Stupka 2007-10-31 10:41:57 UTC
>Before opening the folder in NB, I do not have dir-props at all
- could you please send the file after it was created?

- i still want to see your messages.log file


Comment 11 tboerkel 2007-10-31 10:47:30 UTC
Created attachment 52101 [details]
dir-props after killing NetBeans during scan
Comment 12 tboerkel 2007-10-31 10:56:46 UTC
Created attachment 52102 [details]
log
Comment 13 Tomas Stupka 2007-10-31 16:08:42 UTC
the only way i was able to reproduce was:

1.) versioned project, many .java files, .class files in src.dir
2.) the files aren't in the svn cache yet - $userdir/var/cache/svncache was removed, or clean/build was invoked for the
first time on the project
3.) opening the package

> Workaround is to add .class to the global ignores list of the Subversion command line client
workaround exists

> This issue is even more important since Subversion support is on by default in NB 6.0.
this doesn't apply to no svn projects - they aren't scanned by the svn module at all and i also couldn't reproduce.
There is no problem just because the svn support is on by default and there is definitely no overall impact caused by
this. Again - the only way to reproduce is a svn versioned  project with many ignored files in a not ignored folder. To
make my point - i don't intend to downgrade the fact that this is very annoying for you, however with all the necessary
prerequisites it looks to me like a corner case. And even more - there is a quite simple workaround.

We shall see that we fix this. I'm sure there is a way how to optimize the involved logic, unfortunately it's in the
modules deeper layers at it will take some time ...

> BUT: What if someone does not know about this (very likely) or forgets it (likely)?
agree, we should at least add this into our faq

thanks
Comment 14 Tomas Stupka 2007-11-01 16:46:21 UTC
just realized that the OOME was fixed in scope of issue #120659 and the ignored property processing for the use case
described above runs now twice as fast...

However - it's still to slow, but shouldn't cause huge memory consuming or deadlocks anymore


Comment 15 tboerkel 2007-11-01 18:48:40 UTC
OK. IMHO, the whole issue under Windows with the csrss.exe process hogging the CPU is caused by calling svn.exe so
often. The way I see it, for each and every .class file a PROPGET is called and then each and every .class file is being
added to the svn ignore property of the directory.

When I already have *.class in the ignorable files system setting of NB, then why add it to svn ignores property at all?
And if this can't be avoided, then why not set *.class in the svn ignore property instead of every single .class file?

But in my opinion, NetBeans should not touch svn ignores property of the directory at all. I find this too magic.
Comment 16 Tomas Stupka 2007-11-01 20:13:33 UTC
first of all - i definitely agree that the relevant implementation is suboptimal and we should fix it.


> When I already have *.class in the ignorable files system setting of NB, then why add it to svn ignores property 
> at all? And if this can't be avoided, then why not set *.class in the svn ignore property instead of every single
> .class file?
unfortunately, there is more to the ignore logic than the mentioned "ignored files" setting... 

The thing is, that this issue tracks a specific performance problem when files are automatically ignored by netbeans and
discussing at this place how exactly and why at all would IMO run a bit over of scope. So, if you feel strongly about
this topic, you might write to the interest@subversion.netbeans.org mailing list or contact me directly. There is still
some work to be done and we are looking forward to any relevant input from our users.

thanks
Comment 17 Tomas Stupka 2008-02-27 13:49:42 UTC
fixing the horrible performance problem. 
69903:ef9e342dc850

With "*.class"  set in the global ignores i couldn't reproduce the slow behavior anymore -> P3.

However, there are still some issues in the is ignored logic. We should deal with it ...
Comment 18 Tomas Stupka 2010-09-28 12:40:27 UTC
couldn't find anything bothering in the ignore logic now. fixed.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo