This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Lucene 2.4.1 consumes more memory.
The new lucene may need a bit more mem while doing merge (event it is not seen in visual VM) but this memory is freed. I've added a test for it (commented by default as it takes long time). I've also added gc to the low momory handler to force the gc to free the non needed (unreachable objects) and not to fire the merge by adding next document.
Fixed in jet-main: d97576b593bd
Integrated into 'main-golden', will be available in build *200909181401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/d97576b593bd User: Tomas Zezula <tzezula@netbeans.org> Log: #172312:Lucene 2.4.1 consumes more memory
Is it right to invoke GC in method LuceneIndex.addDocument(IndexDocumentImpl)? I see that CPU about 100% in GC and very slow IDE. Heap allocated 508Mb. Heap used 415Mb. It seems a wrong policy is selected for calling GC. In any case avoid call GC (or minimize number of calls).
The GC is invoked only if low-memory condition is detected (currently 80% of whole heap). GC is (IIRC) needed in such a case, as the MemoryMXBean measures heap usage including garbage, and without GC the low-memory condition could be triggered again and again uselessly. 80% from 508MB is 406.4MB, 415MB is above that. If 415MB is a sustained usage, it seems a lot to me. Why is the memory consumed? What is held there? What will the IDE do with 384MB or 256MB of heap?
I also want to know who eats memory ;-) See: Bug 171672 Cannot create project for Chrome sources with -Xmx512m Bug 181684 NB platform consumes a lot of memory on big projects.
As Honza already explained, the GC is called only in case when memory usage is bigger than 80% of heap size. The GC is enforced to avoid expensive non needed Lucene index merges caused by postponed implicit GC. If the mem usage is bigger than 80% and GC is not able to free any memory (even after flushing parsing api indexes) it seems that the problem is somewhere else (in code holding the memory). From Alexander's comments it seems that it's caused by the recursive listener.
Problem is in number of called GC. Each GC results in not responsible JVM for some time. If you call GC too many times it results in slow IDE. My suggestion is: - call GC once in 20 seconds.
Yes, we can do something like this, not to call the GC for some time. But the GC is not the problem, the disk index merge is much more expensive. The code is like: if (noEnoughMemory) { store(false, null); System.gc(); } The GC just prevents non needed store to be called. The store is the expensive call as it does IO (is some cases lots of IO). We cannot defer the store for 20 sec or something like this as this will cause OutOfMemoryError as the indexers may generates megabytes of data during a time.
Hello Tomas, I'd like to reopen this issue after our last discussion. It is responsibility of JVM to manage low memory. Current threshold is too aggressive which cause unneded expensive flush of indexes
Yes, it should be reopened.
Fixed jet-main 8617b760ad44
Integrated into 'main-golden', will be available in build *201101090000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/8617b760ad44 User: Tomas Zezula <tzezula@netbeans.org> Log: #172312:Lucene 2.4.1 consumes more memory
*** Bug 188734 has been marked as a duplicate of this bug. ***