It would be good to generate a heap dump next time this happens. Thanks Michael.

Can you please look into your ~/.netbeans/dev/var/log/ directory and look if there is a file heapdump.hprof or heapdump.hprof.old (it is generated automatically on OOME) and if so, zip it and somehow (using megashare.com or some other similar free service and paste link here) share it with us. Please reopen the issue when it is done.

And also if you have messages.log from that time please attach it here too.

It neither generates the heap dump (or thread dump) automatically or on demand. It gives: Not enough storage is available to process this command .

Ok, I was able to generate a heap dump:

http://www.MegaShare.com/3743311

(In reply to comment #5)
> Ok, I was able to generate a heap dump:
> 
> http://www.MegaShare.com/3743311

You don't need to 'generate'  heap dump. It is generated automatically by JVM. That's why Petr asked you to look for it in the userdir. Did you find it there?

(In reply to comment #0)
> Product Version: NetBeans IDE Dev (Build 20111122-b2ebb6bae35a)
> Java: 1.6.0_19; Java HotSpot(TM) Client VM 16.2-b04
> System: Windows 7 version 6.1 running on x86; Cp1252; en_US (nb)
> 
> After this is thrown, the IDE is unusable. 
[..] 
Please provide steps how to reproduce it. Thanks.

The heap dump shows that 500M is taken by org.netbeans.core.output2.HeapStorage. Assigning to platform/output window. The actual OOME is not OOME from heap, but from native Windows process.

Output data are normally stored in a file on the disk. Heap storage is used only if there is not enough space on the disk. It seems that you have very limited disk space and that your output is too big for your free memory as well.
Is this that case? If so, I'm afraid we can't do much about it.

I have 220 GB of free space in this disk. I can also do the same thing in my NB 7.0 installation with no problems. This OOME happens at least 3 times per day using the daily build, sometimes more. I can tell it is related to the amount of info logged in the output window.

The JVM couldn't generate the heap dump or the stack trace due to how constrained resources were. I've included the message I was getting in a previous comment.

That is quite strange. Could you please describe steps to reproduce (if any) and attach IDE log?
How many lines is in the output window when the IDE crashes? Could you write a simple program that generates some problematic output?
Thank you.

I can try to reproduce it consistently, but I can assure you there is no relevant info in the messages log file, just the normal initialization messages and *sometimes* the stack trace for the OOME - it seems it gets so bad sometimes it cannot even log that. 

I will attach the messages.log for the next occurrence though anyway. I am just building the IDE right now.

Created attachment 113467 [details]
messages.log

Steps to reproduce:

1 - Create the following class:


package test;

import java.util.Arrays;

public class OutputCrasher {
   public static void main(String[] args) {
      for (int i = 0; i < 10000; i++) {
         char[] c = new char[1000];
         Arrays.fill(c, (char)('a' + 25 * Math.random()));
         System.out.println(new String(c));
      }
   }
}

2- Shift + F6 twice (while the first one is running, just to make it happen quicker)

Shouldn't take long for the IDE to freeze and to throw the OOME (around 1 min here).

10000 lines of log is pretty common for any application.

I am sorry, I was not able to reproduce it.

The problem originates probably in file <netbeans source dir>\core.output2\src\org\netbeans\core\output2\FileMapStorage.java, line 352.
Method FileChannel.map throws an IOException, then FileChanell.read is called instead of it to get some data, but OutOfMemoryError is thrown. Then FileMapStorage.lowDiskSpace is set to true (to prevent further errors), which means that HeapStorage will be used onwards and that even more memory will be needed.

Can you please debug NetBeans to see what happens in FileMapStorage, near line 352?

Thank you for your help.

BTW, for some reason it is much easier to reproduce this using the standard cluster than the java cluster. java takes way longer and requires more parallel instances to be running (3 in my case).

I was able to reproduce using my Ubuntu workstation. I ran 6 instances of the
proposed test case and after some minutes (something like 5 minutes) the IDE
frozen. I am uploading the heap dump right now and will attach the link here
asap.

Created attachment 113472 [details]
Ubuntu message log

Created attachment 113473 [details]
Thread dump for Ubuntu

For the record, I tried it at 7.1RC1, with 6 instances running and it took 27 minutes and 3 seconds to finish without any problem, so looks like some change after the 7.1 branch is causing this problem.

You can download the Ubuntu heap dump at http://www.MegaShare.com/3745776

Thank you, Michel, for the heap dump. It seems it is "correct", i.e. I cannot see any memory leaks or incorrect values there. The memory was probably consumed by standard objects, e.g. swing components displaying the output. 
But in the first heap dump, there is a big HeapStorage object that should not be there.

>so looks like some change after the 7.1 branch is causing this problem
No changes has been made in output window sources since 7.1 branching.

I can confirm this behaviour started somewhat recently, maybe in the last 3-4 weeks, very likely after 7.1 was branched.

michel, misterm, can you please retest extensively with 7.1 RC1 so that we are sure this does not affect 7.1? (in case we don't find the cause soon enough it is important to know if this is a stopper for 7.1). Thanks.

I was able to reproduce on Windows 7 with 32bit JDK. 

The only recent change that affects memory consumption of output window is fix for bug 203739. I tried to revert this change. The performance improved, but the error still appeared.

Misterm also kindly tested reverted module and wrote:
> It improved a lot, but is not as good as before. It didn't crash at
> all with two, even though there were apparently some full GCs going
> on, since it would hang every now and then. Then I tried it with four
> and it still crashes.

The memory consumption was partly affected by fix for bug 203739 and partly probably by other modules. The solution is quite complex and quick fix would be dangerous. The plan is to show only tail of output (several thousand lines), and to store the rest of output in a file (this fix will be contained in version 7.2).

Decreasing the priority. This problem does not impact the majority of the users and is not a stopper for 7.1.

The solution will be complex and will be targeted into the next release.

I strongly disagree with this assessment. To me, this happens with a single compilation in my freeform project. It takes 6 or 7 minutes from the time I boot the IDE to reproduce this. It is just I couldn't send it to you. 

I'm pretty sure several tasks, such as starting an application server inside the IDE will produce enough log for the error to happen. This has nothing to do with multiple copies running.

We have a team of 10 developers here, many of which use Windows. We have been using NB in this project for 7 years. Are we really saying we cannot upgrade to 7.1?

I was able to reproduce it at 7.1RC1, take a look at
http://statistics.netbeans.org/analytics/exception.do?id=545783

Looks like it is a dup of #199744 which was closed as INCOMPLETE. Maybe Michael
was able to create a way to reproduce it now. Maybe it is a stopper anyway...

*** Bug 205538 has been marked as a duplicate of this bug. ***

*** Bug 199744 has been marked as a duplicate of this bug. ***

(In reply to comment #28)
> I strongly disagree with this assessment. To me, this happens with a single
> compilation in my freeform project. It takes 6 or 7 minutes from the time I
> boot the IDE to reproduce this. It is just I couldn't send it to you. 

Agreed, for now not a stopper, but still on radar for 7.1 ...

Tonda/Honza, could we add some logging or anything else ?

Yes, I was able to reproduce it with NB 7.1-RC1. However, I was also able to narrow down the issue: it has something to do with output produced by Ant.

We have a Hudson installation set up here that builds the project twice for each commit (because it uses two JDK installations). Therefore, its output is more than twice as big as a single execution (there is the CVS log and other small messages). I was able to "Show console" for the last 8 builds with the same NB 7.1 RC 1 installation that crashed for a single local build.

core-main/rev/f842abcd6ee9

Releasing of mapped byte buffers improved.

With this change, I was able to print whole output (10,000 lines; 10,000,000 characters) of two simultaneous invocations of code from comment 15. Before, I was able to print 1500 lines only (with 32 bit JDK 6 on Window 7).

Considering for integration into 7.1. Code review will be done. QA will also need to give go.

Considered by QA too risky for 7.1 this late.

Planning for the first 7.1 patch.

core-main/rev/891145f7c403

Improved exception handling.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/891145f7c403
User: Jaroslav Havlin <jhavlin@netbeans.org>
Log: #205437 - OOME with enormous output... - improved exception handling

*** Bug 204272 has been marked as a duplicate of this bug. ***

Workaround for this bug in NetBeans 7.1:
http://wiki.netbeans.org/Bug205437inNB71

*** Bug 207476 has been marked as a duplicate of this bug. ***

Michel G, Misterm, 
can you please try fix on dev build and mark as Verified, if everything is fine?
Without verification issue fix cannot go 7.1 patch 1. Thanks in advance!

V. Build 20120123-0a001538c65d

*** Bug 207737 has been marked as a duplicate of this bug. ***

Transplanted to release71_fixes:

http://hg.netbeans.org/releases/rev/7b8e21b6354b
http://hg.netbeans.org/releases/rev/c8d7ca9cba8f
http://hg.netbeans.org/releases/rev/0469b783f7d2

*** Bug 208172 has been marked as a duplicate of this bug. ***