This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Scanning projects is necessary for update the index therefore it should be triggered only when there is a reasonable evidence that the index is outdated. Currently the projects scanning starts without any real necessity of reindexing. Here is a simple exapmle: when I modify a single Java source file by inserting a single empty line in a package private class and then switch to another file in the same package the scanning gets triggered! Triggering scanning does not make any sense in this case! And for a really big project (thousands of classes) it takes hours! And for a really big project it consumes so much memory that IDE becomes almost irresponsive! Scanning should be smart enough to understand what modifications do require reindexing (like when signature of a public method in a public class is modified) and what do not. The mishandling of infamous scanning project issue raises a big question for the NB development team: do you really think that NB is just for a small entry level Java development? And any professional development should be based on Enclipse? Even if you don't think so, the way you handle this problem speaks for itself in favour of Eclipse.
(In reply to comment #0) > Scanning projects is necessary for update the index therefore it should be > triggered only when there is a reasonable evidence that the index is outdated. > Currently the projects scanning starts without any real necessity of > reindexing. The IDE tries to. After a save (or focus is moved from the currently modified file), the current file is reindexed, which for any reasonable file should be a very quick process. If no signatures change is detected, nothing more happens. If a signature change is detected, all dependent classes are reindexed. Other triggers, like classpath structure changes, force reindex of some or all files to ensure index consistency. > > Here is a simple exapmle: when I modify a single Java source file by inserting > a single empty line in a package private class and then switch to another file > in the same package the scanning gets triggered! Triggering scanning does not Sorry, but it seems we will need a case where we can reproduce this, so that we can find out what is wrong and fix that. I just tried to add an empty line to a package-private class having quite a few big projects open (~10000 Java source files in opened projects and their dependencies), and after save basically nothing happened. So a reproducible test case is likely crucial to find the cause and fix this. > make any sense in this case! And for a really big project (thousands of > classes) it takes hours! And for a really big project it consumes so much > memory that IDE becomes almost irresponsive! Might it be that the IDE is actually operating in an almost out-of-memory situation? The easiest way to spot this is on a thread dump which contains heap summary info, when the tenured generation is permanently filled to very high percentage. (The memory toolbar shows total heap consumption, but the new generation can be pretty big [~120MB], making it quite hard to detect this situation using the toolbar.) > > Scanning should be smart enough to understand what modifications do require > reindexing (like when signature of a public method in a public class is > modified) and what do not. > > The mishandling of infamous scanning project issue raises a big question for > the NB development team: do you really think that NB is just for a small entry > level Java development? And any professional development should be based on > Enclipse? Even if you don't think so, the way you handle this problem speaks > for itself in favour of Eclipse. Sorry, but to fix the problem, we will most likely need a reproducible test case. I develop NetBeans in NetBeans, and I am having lots of quite big projects open. So there is probably some bug that manifests in your case, but not in mine. And to find such a bug, I do not see any other option than being able to reproduce the problem and debug the IDE.
The memory is not a problem. When the scanning ends the very first time after the start of the IDE the memory usage for netbeans.exe is shown at 500MB, while I have 2GB on my desktop (Windows XP, 32-bits). Then after each rescan 200MB are added to the used memory value. There might be a memory leak in the scanning process, but it is not related to this problem. The IDE really gets slow only when the usage comes close to 1000MB. The problem is very well reproducible for me, also searching the Bugs Database for "scanning" I found so many similar issues filed by many different people. None of these cases were resolved and many are marked as "non-reproducible"! If you are unable to reproduce the problem on your side, then please, provide some instructions for additional logging which I could turn on to shed some light on the scanning details which cause such odd behavior. I don't believe that debugging the IDE is the only way to find the bug -- log file can be as well effective.
(In reply to comment #2) > The memory is not a problem. > When the scanning ends the very first time after the start of the IDE the > memory usage for netbeans.exe is shown at 500MB, while I have 2GB on my desktop > (Windows XP, 32-bits). > Then after each rescan 200MB are added to the used memory value. There might be > a memory leak in the scanning process, but it is not related to this problem. > The IDE really gets slow only when the usage comes close to 1000MB. > > The problem is very well reproducible for me, also searching the Bugs Database > for "scanning" I found so many similar issues filed by many different people. > None of these cases were resolved and many are marked as "non-reproducible"! If you know about bug reports that seem to contain enough information to find the problem and are closed as resolved/worksforme or resolve/incomplete, could you please list them here? Thanks. I would definitely take a look on the data in them. But, I believe that scanning related problems are being fixed regularly, if the cause is found: e.g. bug #183405, bug #183233, bug #198901. > > If you are unable to reproduce the problem on your side, then please, provide > some instructions for additional logging which I could turn on to shed some > light on the scanning details which cause such odd behavior. I don't believe > that debugging the IDE is the only way to find the bug -- log file can be as > well effective. Ok. A warning though: this is likely to be iterative process. I would start with these two command line options: -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=0 -J-Dorg.netbeans.modules.java.source.indexing.JavaIndex.level=0 Thanks.
Thanks to the reporter for sending the logs. The problem appears to be as follows. When Dependency Scanning is disabled and a class' signature is changed, all the source roots that depend on the current source root get marked with "dirty" attribute. And each time a dirty source root is "touched" (opened or any file from it being scanned), it is fully reindexed. Probably the flag should be split into two flags: -dirty that would be set when something fatal happens during indexing so that the cache is completely repaired on next indexing -dirty-through-dependency ("badges wrong") which would be set when the dependency scanning is disabled to know which caches need to be refreshed when dependency scanning is enabled. (Although the net effect on find usages is still a bit unclear.)
Lahvaci, could you please elaborate a little on what can be done ? If the affected root is not rescanned fully then up-to-date check might prevent some classes that should be rescanned from being processed: their files did not change from the last indexer run, but the referenced identifier may resolve differently Thanks.
(In reply to Svata Dedic from comment #5) > Lahvaci, could you please elaborate a little on what can be done ? If the > affected root is not rescanned fully then up-to-date check might prevent > some classes that should be rescanned from being processed: their files did > not change from the last indexer run, but the referenced identifier may > resolve differently Which is absolutely correct when the dependency scanning is disabled. The problem is that when dependency scanning is disabled, the roots are marked DIRTY in JavaCustomIndexer.findDependent on signature change, and JavaCustomIndexer.Factory.scanStarted then forces full reindex of the root on next reindex (even minor, e.g. one file) of the root. Ultimately, this may cause much more (and much more costly) reindexes when dependency scanning is disabled than when it is enabled, eliminating any possible advantages of it (but keeping all disadvantages). IIRC, the problem itself may not be reproducible for users in current versions (there surely were versions where it was not reproducible) - probably masked out in RepositoryUpdater, by effectively ignoring the result from scanStarted (in some cases) - not sure about this, though. > > Thanks.
I understand the issue with unnecessary broad scanning, but did not get the proposed solution outlined in comment #4
(In reply to Svata Dedic from comment #7) > I understand the issue with unnecessary broad scanning, but did not get the > proposed solution outlined in comment #4 Right now, the DIRTY flag is used for two purposes: 1. when something goes very wrong during indexing (uncaught exception, out of memory, etc) 2. when a signature is changed, and dependency scanning is disabled In the first case, fixing the cache as soon as possible is desired. In the second case, the cache only needs fixing when the dependency scanning setting gets changed. So the proposal is to use two different flags to record these two different cases.
This old bug may not be relevant anymore. If you can still reproduce it in 8.2 development builds please reopen this issue. Thanks for your cooperation, NetBeans IDE 8.2 Release Boss