This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
[ BUILD # : 200409221800 ] [ JDK VERSION : J2SE 1.5.0 ] RANDOM Something occurs in the ide which prevents my ant clean target from deleting jar/class files which are located in my WEB-INF/lib directory. Some notes about my setup: - I've only seen this affect Free form (with existing ant script) projects - Internal Tomcat server is turned off - The files that cannot be deleted are copied (via an ant task) from a directory (not part of my netbeans project) into my WEB-INF/lib directory (which happens to be inside the directory designated as the 'web pages folder' for my project) - I know that nothing else has a lock on these files (using ntfilemon and psexplorer from sysinternals). If I quit netbeans, I can then delete the file manually (or with external ant), but can rarely delete the file from within the IDE or with ANT from within the IDE. Additionally, I am able to delete other files and directories which are 'mounted' by the IDE, but there's something special about the WEB-INF/lib directory that prevents me from deleting certain files. I'd like to also mention that the behavior I'm seeing is very similar to Issue#: 48890 and Issue#: 47919. However, I have the internal Tomcat DISABLED and nothing else has the files locked. I've had some success in reproducing this. I created a non-proprietary test project that exhibits this behavior occasionally. A few things I've noticed which may help in recreating this: The size of the data being deleted and the time it takes for the script to run seems to impact the frequency of this error occuring. If I'm copying one jar file that's very small, the script runs quickly and I never see this behavior occur (the script time was always over 3 seconds when I've been able to reproduce this).
Created attachment 17864 [details] contents of my ant output
My project which can occasionally reproduce this behavior exceeds the attachment size limit, but I've given a copy to Karel already and will be willing to get it to anyone else who's interested
Thanks, Jason. From private emails with Jason: - WinXP SP1 - it happens with JDK 1.4.2 as well. - I've got a test project (17MB) from Jason which fails in 30% (but for me it works fine). BTW: More people saw this problem sometimes.
Isn't this a duplicate of 47919 ?
I would say that this is not a duplicate because tomcat is not involved in this case (I have internal tomcat disabled).
Jason, can you do one more test for me? I assume that you have the jar files in WEB-INF\lib added to your project's classpath - is this correct? If so, can you remove them from there and then try to reproduce the problem? Thanks!
The exact instances of the jars in WEB-INF/lib are not in my classpath. The ones in my classpath are in my ${buildroot}/lib directory (which is not part of my netbeans project) and are then copied to WEB-INF/lib (which is a subdirectory of my web folder and part of my netbeans project). Would you still like me to try removing them from my classpath?
Also, it's worth mentioning that the test project I gave Karel did not have any items specified in the project classpath and I'm still seeing the same behavior with that project (using build 200409281800)
Thanks. So it does not seem related to classpath then. If the files you are deleting are not on classpath then you do not need to test.
Created attachment 17935 [details] simple project which occasionally exhibits this behavior
I've attached a simplified project which occasionally exhibits the behavior I've described... Appropriate jars need to be added to the lib directory. The contents of my lib directory are as follows: Directory of C:\dev\test\lib 09/29/2004 05:55 PM <DIR> . 09/29/2004 05:55 PM <DIR> .. 07/13/2004 04:41 PM 45,386 activation.jar 07/13/2004 04:41 PM 52,379 aspectjrt-1.0.6.jar 07/13/2004 04:41 PM 1,213,897 classes12.jar 07/13/2004 04:41 PM 118,726 commons-beanutils.jar 08/04/2004 02:32 PM 559,366 commons-collections.jar 07/13/2004 04:41 PM 109,096 commons-digester.jar 07/13/2004 04:41 PM 22,379 commons-fileupload.jar 07/13/2004 04:41 PM 63,980 commons-lang.jar 07/13/2004 04:41 PM 31,605 commons-logging.jar 07/13/2004 04:41 PM 46,865 commons-validator.jar 07/13/2004 04:41 PM 45,386 Copy of activation.jar 07/13/2004 04:41 PM 1,213,897 Copy of classes12.jar 09/10/2004 05:15 AM 3,784,080 Copy of openide.jar 07/13/2004 04:41 PM 65,368 jakarta-oro.jar 07/22/2004 09:12 AM 795,231 jakarta-poi.jar 07/13/2004 04:41 PM 121,070 junit-3.8.1.jar 07/13/2004 04:41 PM 352,668 log4j-1.2.8.jar 08/23/2004 09:31 AM 280,984 mail.jar 08/23/2004 09:31 AM 153,117 mailapi.jar 09/10/2004 05:15 AM 3,784,080 openide.jar 07/13/2004 04:41 PM 75,849 servlet.jar 08/23/2004 09:31 AM 11,902 smtp.jar 07/13/2004 04:41 PM 498,051 struts.jar 07/13/2004 04:41 PM 927,669 xercesImpl.jar 07/13/2004 04:41 PM 123,705 xmlParserAPIs.jar 25 File(s) 14,496,736 bytes I've been able to reproduce this behavior with various combinations of jars by running the build and then the clean target repeatedly from the context menu of the included ant script.
Jason, since nobody else can reproduce this may I ask you to do 2 more tests? I want to see if it has anything to do with web projects or with proejcts at all. The first test would be close all projects, restart the IDE, create a project for your files using template Standard | Java Project with Existing Ant Script and try build/clean several times there. If this still fails then the second test would to close all projects, restart the IDE and choose menu Window | Favorites. If the files are not under the folers there use Add Favorites from its popup menu. Then find the build.xml and try build/compile. Thanks a lot!
Testing on Build 200409301800 The first test still fails... on the first try even The second test also fails (I added the entire project directory to favorites).
Thanks, Jason. I am reassigning to core. Apparently the problem is not related to web project and probably not related to projects at all. Radku, can you evaluate if this may be related to jar filesystem?
Is there any progress on this issue? People are still reporting the problem.
I'm not able to reproduce. But naturally this conflict may occure in Windows and this isn't surprising in Windows world. There is important to know if its just bad luck caused by two threads that share the same archive file (JarFile, ZipFile) or caused by wrong code that didn't close archive (relying on finalizer). JarFileSystem was rewritten to close jar as soon as possible with respect to performance (there is some small delay before archive is closed). I looked into package org.netbeans.modules.javacore.scanning where is FileInfo abstraction and its usage. There seems to be problem in ZipArchiveInfo because although input streams provided by archive are closed properly the whole archive is let open (please notice that ZipFile has method close that is crucial to be called on Windows).
[Windows XP SP1, JDK1.5 Beta 2, NetBeans 4.0 Beta 2] It happened with both Java Application and Web Application. I'm not using "With existing..." projects. Always the dist folder from Required Projects[Libraries]. Restarting the IDE may solve the problem, but not always --on those cases I couldn't delete the jar myself from W_XP Explorer--.
Due Clean & Build is the solution for things like delete/rename/move file synchronization between src and build folders, the issue becomes more and more common.
I've added closing of the ZipFile used during JavaCore's files scanning (as advised by Radek). So now this issue should be gone. Checking in src/org/netbeans/modules/javacore/scanning/FileScanner.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/scanning/FileScanner.java,v <-- FileScanner.java new revision: 1.10; previous revision: 1.9 done Checking in src/org/netbeans/modules/javacore/scanning/ZipArchiveInfo.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/scanning/ZipArchiveInfo.java,v <-- ZipArchiveInfo.java new revision: 1.2; previous revision: 1.1 done
FYI: I was not able to reproduce this behavior with the 10/10 daily build, but I am able to reproduce it (just as frequently as before) with the Oct. 11 daily build.
Reopening - Jason is still able to reproduce it with Q-build candidate (build 200410121800).
Are we able to monitor (and log) locking of those JAR files?
Java module now closes zip files properly, so I don't know what might be the cause. Jason, when you get this problem, does the jar file stay locked for the whole "life" of the IDE session? Or if you wait for a while and try to execute the operation again, the problem disappears? Jason, I would also like to ask you to perform the second Pavel's test (from earlier comments) once again to confirm whether it really fails. Please make sure you do exactly these steps: 1) Close all projects and restart the IDE (make sure that no projects are opened in the editor before restart) 2) choose menu Window | Favorites. If the files are not under the folers there use Add Favorites from its popup menu. Then find the build.xml and try build/compile several times If it is possible, please make sure that no directories containing java files are expanded in Favorites while performing the test.
When this problem occurs, the jar file which cannot be deleted stays locked for the lifetime of the IDE. The only way I've found to unlock the file is to shutdown the IDE and then manually delete the file from windows explorer or the console. Pavel's test also fails (Build 200410131800, this time). This is the Verbose output of relevance from ant: clean: Deleting directory C:\dev\test\output Deleting directory C:\dev\test\output\classes\common Deleting directory C:\dev\test\output\classes\META-INF Deleting directory C:\dev\test\output\classes\middle Deleting directory C:\dev\test\output\classes\test Deleting directory C:\dev\test\output\classes\web Deleting directory C:\dev\test\output\classes Deleting C:\dev\test\output\insight.war Deleting directory C:\dev\test\output\lib Deleting directory C:\dev\test\output\override Deleting directory C:\dev\test\output\props Deleting directory C:\dev\test\output Deleting directory C:\dev\test\web\WEB-INF\lib Deleting C:\dev\test\web\WEB-INF\lib\activation.jar Deleting C:\dev\test\web\WEB-INF\lib\aspectjrt-1.0.6.jar Deleting C:\dev\test\web\WEB-INF\lib\classes12.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-beanutils.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-collections.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-digester.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-fileupload.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-lang.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-logging.jar Deleting C:\dev\test\web\WEB-INF\lib\commons-validator.jar Deleting C:\dev\test\web\WEB-INF\lib\Copy of activation.jar Deleting C:\dev\test\web\WEB-INF\lib\Copy of classes12.jar Deleting C:\dev\test\web\WEB-INF\lib\Copy of openide.jar Deleting C:\dev\test\web\WEB-INF\lib\jakarta-oro.jar Deleting C:\dev\test\web\WEB-INF\lib\jakarta-poi.jar Deleting C:\dev\test\web\WEB-INF\lib\junit-3.8.1.jar Deleting C:\dev\test\web\WEB-INF\lib\log4j-1.2.8.jar Deleting C:\dev\test\web\WEB-INF\lib\mail.jar Deleting C:\dev\test\web\WEB-INF\lib\mailapi.jar Deleting C:\dev\test\web\WEB-INF\lib\openide.jar C:\dev\test\build\build.xml:137: Unable to delete file C:\dev\test\web\WEB-INF\lib\openide.jar at org.apache.tools.ant.taskdefs.Delete.removeDir(Delete.java:594) at org.apache.tools.ant.taskdefs.Delete.execute(Delete.java:498) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275) at org.apache.tools.ant.Task.perform(Task.java:364) at org.apache.tools.ant.Target.execute(Target.java:341) at org.apache.tools.ant.Target.performTasks(Target.java:369) at org.apache.tools.ant.Project.executeTarget(Project.java:1214) at org.apache.tools.ant.Project.executeTargets(Project.java:1062) at org.apache.tools.ant.module.bridge.impl.BridgeImpl.run(BridgeImpl.java:217) at org.apache.tools.ant.module.run.TargetExecutor.run(TargetExecutor.java:236) at org.netbeans.core.execution.RunClassThread.run(RunClassThread.java:125) BUILD FAILED (total time: 4 seconds) Interestingly enough, it doesn't always fail on the same jar file after you delete the initially problematic one manually. Let me know if there's anything else I can do to assist in tracking down this problem.
Try to invoke garbage colection using memory meter when this problem occures.
I am clueless. :( Since the Pavel's test I mentioned also fails, it does not seem to be a java issue. Given that Jason correctly followed the steps I described, there should be no scanning or background parsing done by Java module. So I don't see how this could be a java module issue. Could it be related to the build system? Maybe Radek has some idea how to debug this...
Invoking GC from the memory meter so far in every case has allowed me to successfully delete the file on a subsequent clean (occasionally I can do this without manually invoking GC, steps build -> clean -> GC -> clean or build -> clean -> clean). After doing another build however, the inability to delete returns. Doing a build -> GC -> clean, however, does NOT prevent this behavior from occuring. I'm just grasping at straws here, but one telltale sign of a clean not working, seems to be a slight delay between the time I invoke the clean and the time ant output appears in the output window. This delay does not occur when clean runs successfully.
> Are we able to monitor (and log) locking of those JAR files? Yes, naturally. I patched ZipFile.close and ZipFile constructor. I played with it relaitively long and I didn't find any close caused by finalizer. Everything worked fine. If I called build, clean, build,clean... quickly, clean task couldn't really delete some jar but even in this case all jars seemed to be closed correctly. There is important to know that JarFileSystem isn't closed at once but with some delay. I never noticed delay longer then 1 sec (mostly about 300ms). Although there might be some bug related to this issue its really hard to reproduce. Thats the reason why I let this issue open but I lower priority to P3. I would also like to ask Jason to patch its JDK in the same way as I did and watch what is going on. See my hint for patching ZipFile: 1/ some diagnostic (instead of "dist" use appropriate folder where you store your output jars) if (name.indexOf("dist") != -1) { System.out.println("Open: " + name + "["+ System.identityHashCode(this)+"]" + " " + new java.util.Date ()); Thread.dumpStack(); } 2/ paste it into constructor - choose the right one that is ever called 3/ paste it into close method somwhere after (jz != null) to filter useless finalizer calls and don't forget to replace word "Open:" with "Close:" 4/ build, patch your JDK, run NB IDE with this jdk and watch 5/ suspicious results add as attachment to this issue. Thanks. Seems to me that without your help we won't be able regularly fix this issue.
As a side effect of completely different problem I ran into this problem with deleting jar. Thus I patched JDK again and I found following (see attachment).
Created attachment 18327 [details] Possible culprit
After patching my JDK (painful in NB4 btw, I had to create a new project for the JDK. Is there really no way to open a java file and compile it without adding it to a project?), I see nothing from the patch printing out (except the static blocks denoting that the patch was actually applied).
Seems that javac we use for errors-underlining causes this (based on Radek's stacktrace). Tom, could you please check if ZipFiles are closed properly in javac?
Created attachment 18328 [details] Closed by finalizer. At least there isn't memory leak.
ClassReader.close() now explicitly called from parser and error checker, instead of waiting for finalizer.
Still occurs with about the same frequency as before with build 200410171800
Added explicit close to com.sun.tools.javac.util.Paths.addJarClassFile for jar files whose manifests are read during path computation. I need to pass this back to Jason for testing, as his test project has always worked on my system.
Tom, is there anything I can do in the meantime? I'm still seeing this in the latest NB build, but I guess it now seems to be more of a JVM problem than an IDE problem.
I tried to reproduce the bug with the custom build of Oct 29th sources on Win XP using the test case provided by Jason. I tried to build and clean the project repeatedly, fast and slow, but wasn't able to get it into the "unable to delete jar" state. Jason, can you still reproduce the problem with the 20041031 daily build? Does the test case you provided still work for you occasionally?
Also, is there anything special about your system config? Dual CPU, hyperthreading enabled, ...?
I can still reproduce this bug with the attached test case with about the same frequency as before with build 200410311900. Also, I don't believe there is anything special about my system configuration. I'm using a pretty standard IBM thinkpad Pentium M (single cpu, no hyperthreading). I believe Karel Zikmund and I already discussed this privately (Karel, do you have our email conversation by any chance? I seem to be missing this discussion from my archives. I recall that we were using the same processor.)
Ok, thanks. I'll contact Karel and see if he has any additional info.
I've been unable to reproduce this in 4.0 RC 1. The behavior still occurs however in the 1107 daily build.