Putting a build product into Additional Classpath is a very poor idea.
If you must, don't try to rebuild it with the IDE running. Probably
you don't need to do it at all, since the most common use cases for
A.C. are for third-party libraries needed by optional tasks in the Ant
distribution (which you should not need to build yourself), or
sometimes for special libraries needed by build scripts produced by
someone else which you cannot edit and are too lazy to use <taskdef>
correctly (in which case usually also the library is something
standard you would not be building yourself).

The Ant module does not hold a reference to the JAR files mentioned in
a previous version of A.C. (and only a soft reference to the libs in
the current version), but forcing GC does not always actually ensure
complete GC in my experience. Also if you ever use
Class.getResource(String) or similar methods to refer to resources
within the JAR, forget it - there is a JRE bug (#4405789) that will
prevent that JAR from releasing its lock, regardless of what the
application does. (The module system does work around this bug using
reflection if you are using test modules.) Apparently you can set
URLConnection.defaultUseCaches to false (only on Windows I suppose,
presumably during IDE startup) but I have no idea what effect this may
have on performance of unrelated code.

Problems with locked JAR files on Windows can occur in various
situations; it is pretty difficult to solve them all. However if this
is reproducible (is it?) it is probably fixable somehow. Assigning to
David just because he runs Windows and might be able to reproduce it.
If I know who the locker of the file is, I could suggest code changes
to remove the lock.

I assume you're correct in the first paragraph. I just needed to 
hack and test something quickly and tried to use the first path I 
came across.

And I think you're right in the second paragraph, too. I'm really 
using getClass().getClassLoader().getSystemResourceAsStream(String) 
and .getResource(String) to open xsl stylesheets packaged in the jar 
file.

So I'm afraid there may be nothing for David to solve. It's reliably 
reproducible, so if you want to investigate, I can provide you with 
all info. However, I've just probably run into #4405789 and have 
been locking the file myself.

Well please provide any info you have; perhaps there is an effective
workaround for #4405789.

For example, the Ant module might be able to use the same
JAR-cache-clearing hack that the module system uses, before starting
Ant. (This would not help in case you have a *single Ant session*
which creates, uses, and tries to delete the JAR; but then again, that
would probably not work in command-line Ant either.)

Or it may turn out to be tolerable for performance to just turn off
caching globally on Windows. Both the NB module system and Ant have
their own JAR-based class loaders which do not use the JRE's JAR file
cache, so class loading should not be affected, but resource loading
would be affected.

Or we might be able to supply a custom jar: protocol handler globally
in the VM which has some caching but does not hold a JAR file lock; I
think Radek uses some similar trick in JarFileSystem.

Reopen if you think I should do some workaround.

x

I am going to try the defaultUseCaches = false thing. I have noticed
that even on Linux if I look in the memory map for the NB process
there are several JAR files mapped which were only opened as part of
some Ant build. This is no good.

Also relevant: issue #21114 and BT #4646668.

Created attachment 14769 [details]
Proposed patch

Well the attached patch doesn't seem to break anything on Linux, at
least, and I don't notice any particular performance loss from it.
Would appreciate if someone could test it on Windows.

Note that besides hopefully resolving some Ant problems, it also
permits us to remove a really ugly hack for the JDK caching bug from
JarClassLoader, needed to make reloadable test modules work.

I tried it on Windows: started with new userdir; open a project;
rebuild and execute it couple of times; and did not see any problems.

OK, well I will commit it and if it causes some problems (worse than
it hopefully solves) it can be reverted easily. Would appreciate
specific testing on Windows that it fixes the originally reported
problem and similar problems - i.e. that you can load a task from a
JAR that uses Class.getResource(String) during its execution, and then
delete the JAR.

Please test some.

committed   * Up-To-Date  1.15       
core/bootstrap/src/org/netbeans/JarClassLoader.java
committed   * Up-To-Date  1.19       
core/bootstrap/src/org/netbeans/Main.java

I tested the originally reported scenario (build product used as Ant's
additional classpath) on latest continuous build (200405101153) + Win
NT 4.0 with the same result -- Unable to delete jar file. Build failed.

I'm not sure if the fix was supposed to fix this scenario, but it
apparently didn't. The code in the jar does call
Class.getResource(String). However, I'm not able to rebuild the
project even if I don't execute it at all. The reference to the jar in
Additional classpath property seems to lock it and removing it doesn't
help either. I need to remove the reference *and* restart the IDE to
get the project rebuilt.

Reopening. Feel free to close if you find this use case too obscure.

Uh sorry, I forgot that the particular usage of adding a JAR to the
Additional Classpath is not supposed to be changed by this issue - the
IDE loads the Ant installation incl. all JARs you specify, and holds
onto them for as long as you have that Ant configuration. (After that
the JARs should be released if you do a full GC - not feasible to
forcibly unlock them, I am afraid.)

That really is a pretty obscure scenario, though (and probably not one
I want to support). More interested in the semi-common case that you
make a JAR with some Ant tasks and run them and then clean the project
(deleting the JAR).

Ok. So I've re-arranged my Ant script a little bit and changed it to
define classpath element in taskdef that refers to the product of a
project build (a jar in the project's dist folder).

To recap, I have a project that implements a custom Ant task. I can
build it and produce a jar. The jar is used in a separate Ant script
(in a taskdef definition). The script is run from the IDE (using built
in Ant support and the All Files view). So far so good.

After the above steps, I try to Clean (or Clean and Build) the
project. I can't -- "Unable to delete file
C:\WINNT\Profiles\honza\Personal\nbprojects\izquery\dist\izquery.jar".

Please note that I didn't use the Additional Classpath property this time.

Does GC help? Some things like class loaders hold onto JarFile's and
release them only when they are finalized.

No, GC doesn't seem to help. I'm pressing the memory toolbar button
repeatedly before trying to clean the project, but the file still
can't be deleted. Should I reopen this issue?

Yes I guess so. Will have to get access to a Windows box and some
memory tools to find out what is wrong.

I did some experiments with Windows Process Explorer (from
www.sysinternals.com). I found out that:

a. When an Ant script with a custom task is run from IDE, the nbexec
process opens and holds a reference to the jar file containing the
custom task implementation. The reference *is not* cleared immediately
after the script is executed.

b. The 'taskdef' tag anywhere in the Ant script is enough to open the
jar file and keep a reference to it. I tried to execute a simple no-op
target in the script without any reference to the custom task, but the
jar file was still opened. In this case, I could clear the reference
by forcing the GC. Sometimes it takes up to 5-6 GC cycles, but the
file is eventually freed.

c. When I try to execute a target that calls my custom task, the jar
file is opened and the reference to it remains in place after the
script is completed (as in the previous case). However, this time the
 file reference is never cleared. GC doesn't help in this case. I'm
not sure if this is a regular behavior of Ant, or whether it has
something to do with my custom task implementation (yes, it does call
Class.getResource() several times), but the only way to Clean and
Rebuild the project that produces the locked jar is to restart the IDE.

IMO, (c) is bad, but it may be a special case triggered by the
implementation. However, (b) is not nice either as the user typically
doesn't have any way to force GC repeatedly. In theory, once accessed
jar might remain locked for some period of time and cause intermittent
problems when cleaning and rebuilding projects on Windows platforms.

Possibly relevant: #5010739, apparently fixed recently.

Re. (a) - I know, it has to wait for GC. This is true generally of
class loading from JARs.

Re. (b) - interesting. Even if the <taskdef> is in a target you did
not run?

Re. (c) - don't know why this would happen; would need to have some
memory tool to track the ref. In principle this ref should be cleared
after forcible GC.

I know depending on GC is unpleasant. However I am not sure what else
to do. AFAIK the same behavior can be observed in command-line Ant if
you try to clean the JAR in the same run as you built and used it.
Maybe AntClassLoader could be given some explicit instruction to close
its JAR, I don't know.

Aha, I see something here.... AntClassLoader has a cleanup() method
which *does* close JARs (not via GC), but only if it receives a
buildFinished event... which is currently fired direct to the NB build
logger, not thru the Project, due to an API bug in Ant which appears
to have been corrected in Ant 1.5! So maybe this really is solvable.
Will try it.

Re. (b) - actually, I want to refer to the custom task from multiple
targets potentially, so the <taskdef> is defined at the top level. I
guess that probably explains why the jar is always touched when the
script is run. I'll try the other scenario -- <taskdef> in a target --
and let you know.

OK, have a patch which may help. Hard to know for sure on Linux; I can
look at memory maps and see if a ZIP/JAR is mentioned.

I am testing with

package app;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
public class SomeTask extends Task {
    public void execute() throws BuildException {
        try {
            log("Hello... " +
SomeTask.class.getResource("SomeTask.class").openConnection().getContentLength());
        } catch (Exception e) {
            throw new BuildException(e);
        }
    }
}

with build.xml listing

    <target name="run" depends="jar">
        <taskdef name="sometask" classname="app.SomeTask">
            <classpath>
                <pathelement location="${dist.jar}"/>
            </classpath>
        </taskdef>
        <sometask/>
    </target>

Pressing Shift-F11 followed by F6 then should delete the JAR, build
it, and use a task loaded from it incl. use of jar: protocol.

On my Linux box the system memory map (Gnome "System Monitor", then
right-click the Java process and choose "Memory Map") does show one or
more copies of the JAR being referenced, but they disappear when you
do a GC.

If I replace the log(...) statement with:

            log("Hello... ");
            Thread.sleep(3000);

and keep the memory map open, then the JAR appears in the process' map
while the task is actually running, and disappears promptly after the
build completes, with no GC required.

I interpret this to mean that Ant cleans up its use of the ZipFile
from its class loader promptly, but that the JRE's impl of the 'jar'
URL protocol fails to close it immediately. (There is no apparent API
to dispose a URLConnection object, though you can close an InputStream
opened from it.) In fact JDK 1.4.2 sources show that
j2se/src/share/classes/sun/net/www/protocol/jar/JarURLConnection.java
retains a JarFile instance var but never calls close() on it - i.e. it
will be closed only when finalized.

Well I will hope that the Linux results correspond roughly to Windows,
i.e. that Windows would be unable to delete the JAR under the same
circumstances that the Linux memory map displays it. Will check in the
current fix shortly.

Proposed relnote:
---%<---
A JAR file locking problem applicable to Windows (or generally any
system with mandatory file locks) has been mostly fixed in NetBeans 4.0.

If you run an Ant build which loads one or more custom tasks from a
JAR file (using <taskdef>) you may not be able to delete (or modify)
that JAR during the same build (using Ant or any other means), but you
should be able to delete the JAR after the build finishes, e.g. by
running a clean target separately.

However if some code opened a 'jar'-protocol URL connection from that
task JAR (e.g. using Class.getResourceAsStream), you may still be
unable to delete the JAR after the build finishes, due to a limitation
in the JRE. To release the file lock, force a garbage collection -
e.g. by showing the Memory toolbar and clicking on the button, or by
adding "Garbage Collect" from Tools -> Options -> Actions to some menu
and selecting that menu item.
---%<---

Marking as PERFORMANCE since this fix may help reduce memory
consumption during and after a build, as well as solving the file
locking problem. TBD.

committed   * Up-To-Date  1.15       
ant/src-bridge/org/apache/tools/ant/module/bridge/impl/BridgeImpl.java
committed   * Up-To-Date  1.5        
ant/src-bridge/org/apache/tools/ant/module/bridge/impl/Bundle.properties
committed   * Up-To-Date  1.7        
ant/src-bridge/org/apache/tools/ant/module/bridge/impl/NbBuildLogger.java

Ha, it really seems to help, although the behavior is slightly 
different (on my Win XP box) from what I expected.

First, I tested with your test case on a build *without* the patch. 
I was able to reproduce the locking problem reliably. In both cases 
(with the getResource() call and without it), the jar handle 
remained open and the file couldn't be deleted via Clean. GC didn't 
help in any case.

Secondly, I tested on a build from freshly updated sources that 
included your latest changes. Surprisingly, I wasn't able to 
reproduce the locking problem! My understanding was that it would 
still manifest itself in the version with the getResource() call. I 
used Process Explorer again, but couldn't find the test.jar 
reference anymore.

Encouraged by these results, I went on to experiment with my custom 
task project. This time, the jar containing the task remained open 
after successful completion of my build script and I wasn't able to 
rebuild it. However, GC did make the reference go away (checked in 
Process Explorer) and I was able to clean and build the project 
after that.

So, there clearly is a big progress. Based on your description, I 
expected that getResource() would retain the jar in memory under all 
circumstances. It doesn't seem to be the case with your simple test 
case, although I'm able to reproduce it with my project. Since the 
latter is consistent with the proposed release notes text, I 
consider this issue fixed and verified. Please let me know if you 
want me to test some specific scenarios.

First of all, thanks for your help in testing this! Very much appreciated.

Using getResource() in and of itself doesn't necessarily open a
JarFile - you have to actually cause JarURLConnection.connect to be
invoked, e.g. opening an input stream or getting its length or something.

When a JarURLConnection is connected, it opens a JarFile and keeps a
hard reference to it that it will not close. However if the JURLC is
GC'd (as it hopefully will be), then ZipFile.finalize() closes the ZIP
file. So you would not be able to always reproduce the JAR lock even
if you connect a JURLC; depends on GC behavior.

Based on your results, it sounds like everything is working as
expected so the proposed release note item should still be OK. Note I
said "you *may* still be unable to delete the JAR".

*** Issue 45950 has been marked as a duplicate of this issue. ***

This is an interesting issue. I haven't read and understood all the
intricacies of this, but at first sight it looks somewhat related to
issue 43585 and issue 20384 (aka the infamous bug from hell).
JarURLConnection is evil.

It is likely that turning off default URL caching would have provided
a solution to the bugs you mention, Petr, but I am not sure.

Not really, the following JDK issue causes problems: 

http://dtsw.eng.sun.com/cgi-bin/bugtraq_showbug?bugid=5041014

One problem is that we don't have control over URL caching, as
URLClassLoader uses JarURLConnection implicitly and it does not turn
caching off. The fix would have to be in URLClassLoader.
Well, the fix or our side is to run the JSP compiler Ant task in a
separate process.

Petr, are you sure you mean that? In promo-D we have turned off
default URL caching. This means (AFAIK) that all code using jar: URLs,
including URLClassLoader/URLClassPath when using
getResource{,AsStream}, no longer holds a JAR file lock after the
stream is closed. It is likely that this fix would have solved the
problem you mentioned and that your workaround of running the Ant task
in a separate VM is no longer required. Please verify or supply
details why this is not so.

Thanks, will look at this.