Are you able to reproduce? If so, please try the following patch. If it works (or at least seems to do no harm), I will commit it on Thursday (and this might be a 70patch1-candidate):

diff --git a/openide.filesystems/src/org/openide/filesystems/annotations/LayerGeneratingProcessor.java b/openide.filesystems/src/org/openide/filesystems/annotations/LayerGeneratingProcessor.java
--- a/openide.filesystems/src/org/openide/filesystems/annotations/LayerGeneratingProcessor.java
+++ b/openide.filesystems/src/org/openide/filesystems/annotations/LayerGeneratingProcessor.java
@@ -134,11 +134,11 @@
             }
             return false;
         }
-        if (roundEnv.processingOver() && !roundEnv.errorRaised()) {
+        if (roundEnv.processingOver()) {
             Document doc = generatedLayerByProcessor.remove(processingEnv);
-            if (doc != null) {
+            List<Element> originatingElementsL = originatingElementsByProcessor.remove(processingEnv);
+            if (doc != null && !roundEnv.errorRaised()) {
                 Element[] originatingElementsA = new Element[0];
-                List<Element> originatingElementsL = originatingElementsByProcessor.remove(processingEnv);
                 if (originatingElementsL != null) {
                     originatingElementsA = originatingElementsL.toArray(originatingElementsA);
                 }

Of course I would prefer to make the maps intrinsically safe from leaks rather than rely on the processor being called with roundEnv.processingOver(), but that seems impossible without Java #6389107.

Created attachment 108334 [details]
Same patch but as an attachment so as not to lose formatting

Thanks. I guess it will be enough to edit a hint and check the number of javac instances - I will the testing soon.

A reference to a JavaC created by Java.Source+Parsing.API is a good candidate for showing in Timer&Counters window, I think.

The patch fixes the problem for me, thanks.

For files that are opened in the editor, we register their CompilationUnitTree in timers&counters. This works fine for things related to editing, but obviously not as a global status. I will try to add it to the global status too.

With test: core-main #5001a7862f5e

Integrated into 'main-golden', will be available in build *201105200400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/5001a7862f5e
User: Jesse Glick <jglick@netbeans.org>
Log: #198604: memory leak due to WeakHashMap values -> keys.

Something is wrong on MacOSX:

http://hudson4qe.czech.sun.com/job/platform-filesystems/jdk=JDK 1.6,label=OSX/lastCompletedBuild/testReport/org.openide.filesystems.annotations/LayerGeneratingProcessorTest/testProcessingEnvironmentLeak/?

junit.framework.AssertionFailedError: can collect ProcessingEnvironment:
public static final com.sun.tools.javac.code.Type com.sun.tools.javac.code.Symtab.byteType->
com.sun.tools.javac.code.Type@7a3a30-tsym->
com.sun.tools.javac.code.Symbol$ClassSymbol@354093-owner->
com.sun.tools.javac.code.Symbol$PackageSymbol@14bcb5c-completer->
com.sun.tools.javac.jvm.ClassReader@b6be69-sourceCompleter->
com.sun.tools.javac.main.JavaCompiler@121f4ef-context->
com.sun.tools.javac.util.Context@1464fee-ht->
java.util.HashMap@16f2b7f-table->
[Ljava.util.HashMap$Entry;@b536e6-[58]->
java.util.HashMap$Entry@1d9626b-value->
com.sun.tools.javac.api.JavacTaskImpl@22023a-processors->
java.util.Collections$SingletonSet@6bb78a-element->
org.openide.filesystems.annotations.LayerGeneratingProcessorTest$P@1e31f72-env->
com.sun.tools.javac.processing.JavacProcessingEnvironment@1b1fbf4
        at junit.framework.Assert.fail(Assert.java:47)
        at org.netbeans.junit.NbTestCase$4.run(NbTestCase.java:1361)

(In reply to comment #8)
> Something is wrong on MacOSX

Apparently so, but it is not the same reference chain as is implicated in this bug, so reclosing. I am not sure what the quoted chain means. The test has been run just once on each machine, so it is not yet possible to say that the failure is Mac-specific (especially as I have no access to a Mac to test on), but it could be a bug in some older (?) version of javac included in the Mac JDK. If the failure persists on Mac I will just disable the test on that platform, I suppose.

Backporting to 7.0.1: releases #b73e4016ae91

(In reply to comment #9)
> it could be a bug in some older (?) version of javac included in the Mac JDK

In particular,

public static final com.sun.tools.javac.code.Type com.sun.tools.javac.code.Symtab.byteType

is suspicious, since both JDK 6u22 and OpenJDK 7 sources list this field as nonstatic. Even Linux JDK 5u22 has it as nonstatic. Perhaps someone with a Mac could confirm that a nonstandard version of javac is bundled.

Older versions of javac had several static fields in Symtab, that kept an instance of javac in memory as long as Symtab class was held. Maybe the test could check if "com.sun.tools.javac.code.Symtab.byteType" exists and is static and stop if both these conditions are met?

Probably the simplest way to make the test pass on Mac. I'll find an Mac owner and force him to do the fix.

I do not think this has much to do with Mac - all that is needed is sufficiently old JDK. I was able to reproduce the test failure on Linux with:
$ java -version
java version "1.6.0-rc"
Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
Java HotSpot(TM) Server VM (build 1.6.0-rc-b104, mixed mode)

But I am quite sure that the test will also fail on 1.6u05 (untested). The test will most likely pass on update 14 (I do not have any updates between these two). So it should be easy to create and test the workaround on Linux.

Indeed fails for me in JDK 6[u0] on Linux. Working around it: core-main #113d79747ad9