This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 152943 - Editors recognizing *only* source-derived classfiles in dependent project jars
Summary: Editors recognizing *only* source-derived classfiles in dependent project jars
Status: REOPENED
Alias: None
Product: java
Classification: Unclassified
Component: Project (show other bugs)
Version: 6.x
Hardware: Macintosh All
: P3 blocker with 2 votes (vote)
Assignee: Tomas Zezula
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-11-11 12:25 UTC by cemerick
Modified: 2009-09-30 07:40 UTC (History)
0 users

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments
sample project, with nested dependency (80.05 KB, application/x-compressed)
2008-11-11 12:26 UTC, cemerick
Details

Note You need to log in before you can comment on or make changes to this bug.
Description cemerick 2008-11-11 12:25:11 UTC
I have two J2SE projects, A and B.  A depends upon B's output jar, as specified in A's Libraries panel in its project
properties dialog. A full clean and build of A works just fine -- it cleans A and B, builds B, and then builds A with no
errors.  However, NetBeans editors show that *some* packages and classes present in B's output jar cannot be found.

After some experimentation, I've determined that those packages and classes that cannot be found are those that are
being injected into B's jar (as part of B's build process, in a -post-jar ant target hook) from sources other than the
source code associated with B (some of those classes come from other jars, and we need to merge them into B's output
jar; other classes are generated by some code-generation processes that go straight to bytecode, such as Clojure's
gen-class facility, scalac, asm-generated code, etc).  So, it appears that the editors in Netbeans don't actually
resolve the classfiles present in output jars of dependent projects -- they only resolve classes that have corresponding
source files in those dependent projects.

If this is really the case, I'd say it's a bug -- there's absolutely no reason (at least that I can come up with) for
Netbeans not to use the actual output jar from dependent projects, as opposed to speculating on what will be in those
output jars based on the Java source files contained in those projects.

The only workaround I know of is to manually add references to interim jars we could build that contain those classfiles
that are injected into project B's output jar.  This strongly subverts the notion of a project-level dependency, and
breaks the encapsulation between projects.

I've attached a sample project that exemplifies the problem.  As described above, project A is dependent upon the nested
project B; the latter's build process injects some classes into project B's output jar (in a real-world project, this
could be from any of a dozen different entirely legitimate sources).  The com.foo.app.Main class in project A refers to
those injected classes, which the Netbeans editor flags as errors; however, the build of project A completes
successfully, and running that main class works just fine:

[catapult:Development/ProjectA] chas% java -cp dist/ProjectA.jar:ProjectB/dist/ProjectB.jar com.foo.app.Main
org.apache.commons.logging.impl.LogFactoryImpl@c672d0
[catapult:Development/ProjectA] chas% 

Interestingly, a run configuration to do the same fails with a compiler error, so it looks like this limited visibility
into what project B is actually producing goes further than just the editor:

run:
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.lang.RuntimeException: Uncompilable source code - package org.apache.commons.logging does not exist
        at com.foo.app.Main.<clinit>(Main.java:9)
Java Result: 1
Comment 1 cemerick 2008-11-11 12:26:19 UTC
Created attachment 73624 [details]
sample project, with nested dependency
Comment 2 Tomas Zezula 2008-11-11 14:20:42 UTC
The inter project dependency is always source to source, there are several reasons for this. One of this is that the editor has to be independent on the
resulting jar file as it may be cleaned or not even compiled.
You can solve the problem by copying the output jar out of the project and add it as jar file.

Comment 3 cemerick 2008-11-11 14:35:36 UTC
I understand that in the situation where the output jar file doesn't exist, the editor needs to go against the source
files present in the project dependency.  However, if the jar file *does* exist, shouldn't editors rely upon that jar,
rather than the source code?  That *seems* like an easy win-win sort of approach.  Is there a downside to *falling back*
on known source code, rather than referring to it exclusively?

Especially as other languages flourish that target the JVM by emitting classfiles, it's very counterproductive to assume
(mandate?) that a project dependency's jar only contains classfiles that map to known Java-language source files.

Yes, I can work around this problem by copying the dependency's jar outside of the project directory and linking to that
upstream.  That workaround is sub-par, if only because then I need to manually set up the chained build.
Comment 4 Tomas Zezula 2008-11-11 14:47:49 UTC
>However, if the jar file *does* exist, shouldn't editors rely upon that jar,
>rather than the source code?  That *seems* like an easy win-win sort of approach.  Is there a downside to *falling back*
>on known source code, rather than referring to it exclusively?
No, it's much worse. You modify something in project A and you don't see it in project B until you rebuild the project A,
this is unacceptable behavior.

>Especially as other languages flourish that target the JVM by emitting classfiles, it's very counterproductive to assume
>(mandate?) that a project dependency's jar only contains classfiles that map to known Java-language source files.
As far as I understand you use pure j2seproject which produces class files from java sources. If not the language should provide
the needed IDE hooks as for example groovy does.
Comment 5 cemerick 2008-11-11 16:57:52 UTC
>>However, if the jar file *does* exist, shouldn't editors rely upon that jar,
>>rather than the source code?  That *seems* like an easy win-win sort of approach.  Is there a downside to *falling back*
>>on known source code, rather than referring to it exclusively?
>No, it's much worse. You modify something in project A and you don't see it in project B until you rebuild the project A,
>this is unacceptable behavior.

My apologies, I misworded my proposal.  I am not proposing referring to the output jar file exclusively or primarily;
rather, refer to the Java source code first, and then fall back on the output jar file (if it exists) if the class being
searched for is not found in the Java source code.

>>Especially as other languages flourish that target the JVM by emitting classfiles, it's very counterproductive to assume
>>(mandate?) that a project dependency's jar only contains classfiles that map to known Java-language source files.
>As far as I understand you use pure j2seproject which produces class files from java sources. If not the language
>should provide the needed IDE hooks as for example groovy does.

There are dozens of languages that target the JVM, and most of their communities simply do not have the resources or
wherewithal to implement full-fledged IDE support.  Further, it is very common for users of these other JVM languages to
mix Java sources with source files of those other languages, as they can interoperate very readily in most cases.  

In the base case where the j2se project's jar contains only classfiles compiled from Java source files, the change I'm
proposing has no impact; however, for those who are using a non-Java JVM language, the proposed change would provide a
great deal of benefit (i.e. bringing functionality up to the basic level of all-Java projects).

Comment 6 Tomas Zezula 2008-11-11 17:05:07 UTC
>refer to the Java source code first, and then fall back on the output jar file
Doesn't work either, imagine you have deleted a class in sources without rebuilding the project A => the class, even it doesn't exist any more, is visible
to project B.

Comment 7 cemerick 2008-11-11 17:16:47 UTC
>>refer to the Java source code first, and then fall back on the output jar file
>Doesn't work either, imagine you have deleted a class in sources without rebuilding the project A => the class, even it
>doesn't exist any more, is visible to project B.

Seems like an edge case, especially since (a) removing classes is a relatively rare event and (b) the "shadow" classfile
would be eliminated on the next clean build.

Just to clarify: it is expected that any JVM language that generates classfiles must have a NetBeans plugin in order for
its classfiles to be usefully recognized by the IDE?
Comment 8 Tomas Zezula 2008-11-11 17:27:22 UTC
>Seems like an edge case, especially since (a) removing classes is a relatively rare event and (b) the "shadow" classfile
>would be eliminated on the next clean build.
Deleting a class is an edge case? It's a P1 defect! You force all the IDE features to create invalid code.
And there are another P1 related to this change, like rebuild of dependent project on each rebuild of the project A, jar file
opening, etc.

>Just to clarify: it is expected that any JVM language that generates classfiles must have a NetBeans plugin in order for
>its classfiles to be usefully recognized by the IDE?
Or it can create an ordinary library jar file.

Comment 9 Tomas Zezula 2008-11-11 17:34:40 UTC
BTW you don't need to add an IDE language support just implement VirtualSourceProvider (2 methods) is enough.

Comment 10 cemerick 2008-11-11 17:51:08 UTC
I'll look at VirtualSourceProvider, but working on IDE features one language at a time is a lot more work than adding a
few subant calls and a copy task to some ant builds.

Isn't this a broader issue?  There's lots of processes/tools that generate classfiles that are entirely unrelated to
programming languages; consider parser generators, tools like asm, MDA tools, etc.  It seems that gracefully handling
such generated resources in a general way would be extremely beneficial.  Perhaps my suggestions are off the mark
(obviously, mine being a user's perspective), but maybe there's another approach that would satisfy all use cases.

Two auxiliary issues:

(a) when I drop the project dependency, and set a jar dependency on projectname/dist/projectname.jar, the generated
classfiles are still not picked up.  If I move projectname.jar to projectname/projectname.jar, and set the jar
dependency to that path, all's well.  It seems that the IDE detects that projectname/dist/projectname.jar is the output
of a j2se project and decorates editors based on that project's source files, even though the dependency is explicitly
set on the jar, and not on the project.

(b) given the way that project dependencies work, perhaps it would make sense to not indicate in the Libraries panel of
a project's properties dialog that a project's output jar is going to be used to resolve classes, when it's that
project's source files that will be used again.  That UI hint is very misleading.
Comment 11 Tomas Zezula 2008-11-11 18:03:29 UTC
>It seems that the IDE detects that projectname/dist/projectname.jar is the output
>of a j2se project and decorates editors based on that project's source files, even though the dependency is explicitly
>set on the jar, and not on the project.
Unfortunately  yes, the project system uses Queries, in this case the SourceForBinaryQuery which provides sources for
the binary (jar). The query works with URL, the only information how it can find out the answer is a match with the binary jar file.
This causes a fact that even if you add the project output as a jar file it's handled as a project.

Can you attach more details which tool do you use to generate these class files? Maybe we can find some simple solution of your problem.
Comment 12 cemerick 2008-11-11 18:10:21 UTC
We're using Clojure pretty heavily (http://clojure.org), which uses ASM (http://asm.objectweb.org/) to generate bytecode
(usually at runtime, but optionally during a build process if one needs classfiles that can be statically linked).

Perhaps a new directory in projectname/build could be established?  We could configure our clojure compilation process
to drop all output there (and other tools could be configured similarly); that directory could then be monitored by the
IDE for editor decoration, and folded into the output jar file for normal ant builds.
Comment 13 Tomas Zezula 2008-11-11 18:19:37 UTC
Yes, this will work. I will download the closures and try.
Comment 14 cemerick 2009-09-29 17:33:09 UTC
A potential course of action was suggested after the issue was marked as WONTFIX, which tzezula indicated had promise. 
However, the issue was never reopened.

This is probably obvious to those who are familiar with the details, but the described behaviour is also present in NB
RCP modules.
Comment 15 Tomas Zezula 2009-09-29 17:54:51 UTC
The virtual source provider works fine used by groovy and others. Someone needs to create the implementation for clojures, which does not belong to the 
j2seproject.
Anyway you can try to use the -J-DCacheClassPath.keepJars=true command line option to force NB to use the jar files in addition to source files but with all 
the problems (delete file...) I've described above.
Comment 16 cemerick 2009-09-29 18:26:04 UTC
-J-DCacheClassPath.keepJars=true appears to do exactly what we want, thank you! (I was completely unaware of it.) 
Having to do a rebuild after deleting a file is a small price to pay to get the UI to report errors properly the other
99% of the time, IMO.

Clojure is simply our particular use-case -- this issue will affect anyone generating classfiles from non-java
sourcecode (such as when using certain parser generators, ASM in general, etc).  It seems that supporting an optional
subdirectory in build within projects where generated classfiles can be deposited is a good general (and default)
solution, regardless of the tooling being used to generate that code.  Do you consider that to be a worthwhile
enhancement possibility?
Comment 17 Tomas Zezula 2009-09-30 07:40:57 UTC
Yes, it's a valid enhancement.
We already have such generated folders for sources. Adding similar folders for classes sounds as a valid requirement.