This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 172312 - Lucene 2.4.1 consumes more memory
Summary: Lucene 2.4.1 consumes more memory
Status: RESOLVED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: Tomas Zezula
URL:
Keywords: PERFORMANCE
: 188734 (view as bug list)
Depends on:
Blocks:
 
Reported: 2009-09-15 16:48 UTC by Tomas Mysik
Modified: 2011-01-18 12:58 UTC (History)
4 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tomas Mysik 2009-09-15 16:48:59 UTC
Lucene 2.4.1 consumes more memory.
Comment 1 Tomas Zezula 2009-09-16 13:22:42 UTC
The new lucene may need a bit more mem while doing merge (event it is not seen in visual VM) but this memory is freed.
I've added a test for it (commented by default as it takes long time).
I've also added gc to the low momory handler to force the gc to free the non needed (unreachable objects) and not to fire the merge
by adding next document.
Comment 2 Tomas Zezula 2009-09-16 13:29:48 UTC
Fixed in jet-main: d97576b593bd
Comment 3 Quality Engineering 2009-09-18 22:32:55 UTC
Integrated into 'main-golden', will be available in build *200909181401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/d97576b593bd
User: Tomas Zezula <tzezula@netbeans.org>
Log: #172312:Lucene 2.4.1 consumes more memory
Comment 4 Alexander Simon 2010-03-11 08:21:26 UTC
Is it right to invoke GC in method LuceneIndex.addDocument(IndexDocumentImpl)?
I see that CPU about 100% in GC and very slow IDE.
Heap allocated 508Mb.
Heap used 415Mb.
It seems a wrong policy is selected for calling GC.
In any case avoid call GC (or minimize number of calls).
Comment 5 Jan Lahoda 2010-03-11 09:12:12 UTC
The GC is invoked only if low-memory condition is detected (currently 80% of whole heap). GC is (IIRC) needed in such a case, as the MemoryMXBean measures heap usage including garbage, and without GC the low-memory condition could be triggered again and again uselessly.

80% from 508MB is 406.4MB, 415MB is above that. If 415MB is a sustained usage, it seems a lot to me. Why is the memory consumed? What is held there? What will the IDE do with 384MB or 256MB of heap?
Comment 6 Alexander Simon 2010-03-11 09:28:51 UTC
I also want to know who eats memory ;-)
See:
Bug 171672 Cannot create project for Chrome sources with -Xmx512m
Bug 181684 NB platform consumes a lot of memory on big projects.
Comment 7 Tomas Zezula 2010-03-15 12:54:42 UTC
As Honza already explained, the GC is called only in case when memory usage is bigger than 80% of heap size. The GC is enforced to avoid expensive non needed Lucene index merges caused by postponed implicit GC.
If the mem usage is bigger than 80% and GC is not able to free any memory (even after flushing parsing api indexes) it seems that the problem is somewhere else (in code holding the memory).  From Alexander's comments it seems that it's caused by the recursive listener.
Comment 8 Alexander Simon 2010-03-15 13:34:10 UTC
Problem is in number of called GC.
Each GC results in not responsible JVM for some time.
If you call GC too many times it results in slow IDE.
My suggestion is:
- call GC once in 20 seconds.
Comment 9 Tomas Zezula 2010-03-15 14:27:30 UTC
Yes, we can do something like this, not to call the GC for some time. But the GC is not the problem, the disk index merge is much more expensive.

The code is like:
if (noEnoughMemory) {
           store(false, null);
            System.gc();
}
The GC just prevents non needed store to be called. The store is the expensive call as it does IO (is some cases lots of IO). We cannot defer the store for 20 sec or something like this as this will cause OutOfMemoryError as the indexers may generates megabytes of data during a time.
Comment 10 Vladimir Voskresensky 2010-06-07 10:45:22 UTC
Hello Tomas,
I'd like to reopen this issue after our last discussion. It is responsibility of JVM to manage low memory.
Current threshold is too aggressive which cause unneded expensive flush of indexes
Comment 11 Tomas Zezula 2010-06-07 11:39:25 UTC
Yes, it should be reopened.
Comment 12 Tomas Zezula 2011-01-06 16:29:12 UTC
Fixed jet-main 8617b760ad44
Comment 13 Quality Engineering 2011-01-09 06:20:56 UTC
Integrated into 'main-golden', will be available in build *201101090000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/8617b760ad44
User: Tomas Zezula <tzezula@netbeans.org>
Log: #172312:Lucene 2.4.1 consumes more memory
Comment 14 David Strupl 2011-01-18 12:58:56 UTC
*** Bug 188734 has been marked as a duplicate of this bug. ***