This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 197905 - Fix MemoryValidationTest
Summary: Fix MemoryValidationTest
Status: VERIFIED FIXED
Alias: None
Product: ide
Classification: Unclassified
Component: Commit Validation (show other bugs)
Version: 7.1
Hardware: PC Windows XP
: P2 normal (vote)
Assignee: Petr Cyhelsky
URL: http://hudson4qe.czech.sun.com/view/C...
Keywords: PERFORMANCE
Depends on: 178071
Blocks:
  Show dependency tree
 
Reported: 2011-04-20 12:59 UTC by Jiri Skrivanek
Modified: 2012-06-07 06:00 UTC (History)
4 users (show)

See Also:
Issue Type: TASK
Exception Reporter:


Attachments
List of few references keeping the project in memory (10.69 KB, text/plain)
2011-04-27 13:40 UTC, Jaroslav Tulach
Details
Fix in Project UI BadgingNode (4.16 KB, patch)
2011-05-07 19:53 UTC, Jaroslav Tulach
Details | Diff
Log from test run. (232.93 KB, text/plain)
2011-05-10 12:12 UTC, Jiri Skrivanek
Details
Log from local test run on Windows XP. (265.00 KB, text/plain)
2011-06-13 09:46 UTC, Jiri Skrivanek
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jiri Skrivanek 2011-04-20 12:59:12 UTC
Please, fix MemoryValidationTest. It is failing and it is disabled in project.properties. To reproduce:

- open java.kit\test\qa-functional\src\org\netbeans\test\ide\MemoryValidationTest.java
- call "Run|Test File" (Ctrl+F6)
Comment 1 Jaroslav Tulach 2011-04-27 13:40:04 UTC
Created attachment 107973 [details]
List of few references keeping the project in memory
Comment 2 Jaroslav Tulach 2011-05-06 15:44:27 UTC
Performance is priority again for 7.1, making P2.
Comment 3 Jaroslav Tulach 2011-05-07 19:51:45 UTC
testGCDocuments is fixed. I have an improvements for testProjects, but one reference chain is remaining. Something from java.hints. I am not sure where to break the reference, passing to Jan Lahoda. Attach my patch and then just run java.kit/**/MemoryValidationTest.java with Shift-F6 to see the problem.

Checking if all projects are really garbage collected Project:
private static java.util.Map org.netbeans.modules.java.hints.HideField.allMembersCacheTo->
java.util.HashMap@18288f6-table->
[Ljava.util.HashMap$Entry;@2f2659-[3]->
java.util.HashMap$Entry@195321f-key->
com.sun.tools.javac.code.Symbol$ClassSymbol@3218ea-fullname->
com.sun.tools.javac.util.SharedNameTable$NameImpl@198315d-table->
com.sun.tools.javac.util.SharedNameTable@12a4314-loader->
org.netbeans.modules.java.source.TreeLoader@11b4dd2-cpInfo->
org.netbeans.api.java.source.ClasspathInfo@bdf8d3-srcClassPath->
org.netbeans.api.java.classpath.ClassPath@12ada00-impl->
org.netbeans.spi.java.project.support.ClassPathProviderMerger$ProxyClassPathImplementation@23cdb2-mainProvider->
org.netbeans.modules.java.api.common.classpath.ClassPathProviderImpl@8457a3-helper->
org.netbeans.spi.project.support.ant.AntProjectHelper@10f5a54-state->
org.netbeans.api.project.ProjectManager$ProjectStateImpl@8459ad-p->
org.netbeans.modules.java.j2seproject.J2SEProject@1be341e
Comment 4 Jaroslav Tulach 2011-05-07 19:53:28 UTC
Created attachment 108165 [details]
Fix in Project UI BadgingNode
Comment 5 Quality Engineering 2011-05-08 04:44:08 UTC
Integrated into 'main-golden', will be available in build *201105080000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/09f5f3325550
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #197905: Fixing testGCDocuments, enabling the test again in commit validation. Disabling testGCProjects for now as they continue to fail
Comment 6 Jan Lahoda 2011-05-08 09:40:19 UTC
http://hg.netbeans.org/jet-main/rev/88eba98b8b93
Due to #198426, it may take a while before you can actually see the changeset in other repositories. Reassigning back to origin.
Comment 7 Jan Lahoda 2011-05-08 11:15:36 UTC
Back to original assignee, too.
Comment 8 Jaroslav Tulach 2011-05-09 10:41:55 UTC
ergonomics#f9738d07cd6a
Comment 9 Jiri Skrivanek 2011-05-10 12:12:49 UTC
Created attachment 108209 [details]
Log from test run.

Still failing when run locally on WindowsXP. Details in attached log file.
Comment 10 Quality Engineering 2011-05-11 19:41:18 UTC
Integrated into 'main-golden', will be available in build *201105111436* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/88eba98b8b93
User: Jan Lahoda <jlahoda@netbeans.org>
Log: #197905: really clear HideField.allMembersCacheTo when the CompilationInfo is GCed.
Comment 11 Jaroslav Tulach 2011-05-23 15:42:01 UTC
Not important for 7.0.1
Comment 12 Jaroslav Tulach 2011-05-24 05:30:52 UTC
Failure on solaris related to spellchecker. Jan, can you use weak reference somewhere and pass to projects?

Are all documents GCed? TextDocument:
private static java.lang.ThreadGroup java.lang.SecurityManager.rootGroup->
java.lang.ThreadGroup@1b383e9-threads->
[Ljava.lang.Thread;@d8727d-[10]->
org.openide.util.RequestProcessor$Processor@12572b8-todo->
org.openide.util.RequestProcessor$Task@14d6e25-run->
org.netbeans.modules.spellchecker.ComponentPeer$1@131bdfd-this$0->
org.netbeans.modules.spellchecker.ComponentPeer@14b948b-document->
org.netbeans.modules.editor.NbEditorDocument@1af8ce9


and on windows and solaris related to "modified projects":

ssertionFailedError: Checking if all projects are really garbage collected Project:
private static final org.netbeans.api.project.ProjectManager org.netbeans.api.project.ProjectManager.DEFAULT->
org.netbeans.api.project.ProjectManager@2f5c-modifiedProjects->
java.util.HashSet@1b2060d-map->
java.util.HashMap@7725aa-table->
[Ljava.util.HashMap$Entry;@4fe956-[9]->
java.util.HashMap$Entry@337cfd-key->
org.netbeans.modules.java.j2seproject.J2SEProject@d9d2a4
Comment 13 Jan Lahoda 2011-05-24 06:51:57 UTC
(In reply to comment #12)
> Failure on solaris related to spellchecker. Jan, can you use weak reference
> somewhere and pass to projects?
> 
> Are all documents GCed? TextDocument:
> private static java.lang.ThreadGroup java.lang.SecurityManager.rootGroup->
> java.lang.ThreadGroup@1b383e9-threads->
> [Ljava.lang.Thread;@d8727d-[10]->
> org.openide.util.RequestProcessor$Processor@12572b8-todo->
> org.openide.util.RequestProcessor$Task@14d6e25-run->
> org.netbeans.modules.spellchecker.ComponentPeer$1@131bdfd-this$0->
> org.netbeans.modules.spellchecker.ComponentPeer@14b948b-document->
> org.netbeans.modules.editor.NbEditorDocument@1af8ce9

Sorry, but I guess this begs some investigation in RequestProcessor. I can see two possibilities:
-the task is still running: a thread dump is needed to find out why. WeakReference wan't help, as the task would need to keep the Document in a local variable for the span of the task, leading to a similar test failure, but way more difficult to investigate.
-the task is finished and RequestProcessor keeps it uselessly and incorrectly: fix the RP rather that look at all RP.Tasks and try to workaround the RP problem all over the IDE.
Comment 14 Jaroslav Tulach 2011-05-24 10:52:19 UTC
OK, let's ask Jesse for help with the modifiedProjects map.
Comment 15 Jesse Glick 2011-05-24 18:15:55 UTC
Tried to run the test locally, but testGCDocuments timed out, and it seems to be hanging in testGCProjects too - 100% CPU for many minutes. Anyway the last ref chain does not directly indicate any problem; if the j2seproject's metadata is in fact modified, then ProjectManager is obliged to keep it in memory until it is saved. The question is rather - who modified it (ProjectManager.ProjectStateImpl.markModified) without then also saving it (ProjectManager.saveProject)? When I run the test (killing it somewhere in the testGC*), there are a few calls to markModified, all followed soon after by saveProject.
Comment 16 Marian Mirilovic 2011-06-10 08:57:35 UTC
too old - starting with clean table
Comment 17 Marian Mirilovic 2011-06-10 08:58:48 UTC
v/c
Comment 18 Jiri Skrivanek 2011-06-13 09:46:00 UTC
Created attachment 108873 [details]
Log from local test run on Windows XP.

The test is still failing with latest dev build.
Comment 19 Jaroslav Tulach 2011-07-20 07:15:49 UTC
The failures are deep in AWT/Swing and will be hard to workaround:

private static java.awt.Toolkit java.awt.Toolkit.toolkit->
sun.awt.windows.WToolkit@1c7c9eb-eventListener->
java.awt.Toolkit$ToolkitEventMulticaster@4f82c2-b->
java.awt.Toolkit$SelectiveAWTEventListener@8811f9-listener->
java.awt.LightweightDispatcher@af1a01-mouseEventTarget->
org.openide.text.QuietEditorPane@89cf1b-model->

QuiteEditorPane is no longer isShowing(), of course...
Comment 20 Jesse Glick 2011-07-20 10:18:14 UTC
I thought the test already did some hacks to clear certain fields in AWT that were thought to be harmless?
Comment 21 Jaroslav Tulach 2011-07-21 12:26:31 UTC
Right it does, but the listener in this case is not static field, so cleaning it is not going to be easy. 

Btw. I've started a discussion at
http://mail.openjdk.java.net/pipermail/awt-dev/2011-July/001801.html
with the hope to make necessary fixes in JDK.
Comment 22 Jaroslav Tulach 2011-07-25 12:30:46 UTC
Reported as JDK bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070542
Comment 23 Jaroslav Tulach 2011-08-05 10:14:37 UTC
Another JDK bug reported as http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7072167
Comment 25 Jaroslav Tulach 2011-10-20 05:36:31 UTC
Clearly, this is a longer process which will not finish by 7.1. If there is a simple short term action (like disabling the test), tell me. I can do it for 7.1.
Comment 26 Jiri Skrivanek 2011-10-20 07:08:33 UTC
We do not run the test regularly now. As soon as it is fixed, we add it again to daily runs.
Comment 27 Jaroslav Tulach 2012-01-11 14:03:08 UTC
Petr, could you move this issue forward instead of me. I am busy with other tasks and I will not be able to work on this anytime soon.
Comment 28 Petr Cyhelsky 2012-06-01 11:25:51 UTC
fixed in ergonomics 481dbe593c43
potentially found leaks are to be dealt with in separate issues...
Comment 29 Jesse Glick 2012-06-01 15:09:06 UTC
The test might also need to be expanded to do more operations which are likely to create leaks - e.g. open output-area windows (Search, Find Usages, Test Results) with GUI controls that hyperlink to project files. Properly written code will hold only a URL or similar here, but buggy code might hold a FileObject or even a Project.
Comment 30 Jaroslav Tulach 2012-06-03 19:27:28 UTC
(In reply to comment #28)
> fixed in ergonomics 481dbe593c43
> potentially found leaks are to be dealt with in separate issues...

Amazing! If the test seems stable, we should include it again in some commit validation.
Comment 31 Jiri Skrivanek 2012-06-04 08:27:24 UTC
Actually the test reports some memory leaks but Petr Cyhelsky said it is now with proper heap dumps and so. He will run it regularly and file bugs appropriately.
Comment 32 Quality Engineering 2012-06-07 06:00:00 UTC
Integrated into 'main-golden', will be available in build *201206070001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/481dbe593c43
User: Petr Cyhelsky <cyhelsky@netbeans.org>
Log: #197905 - correcting memory validation test