Bug 47498

*** Issue 47163 has been marked as a duplicate of this issue. ***

*** Issue 46055 has been marked as a duplicate of this issue. ***

Configuring sources may also be tricky since you may need to provide a
SourceLevelQuery and ClassPathProvider for those as well, in case the
user steps through them during debugging and expects things like Go To
Declaration to work from those sources. I think j2seplatform handles
SLQ by checking for bytecode version in the binaries - not something I
really want to duplicate.

*** Issue 48025 has been marked as a duplicate of this issue. ***

Jesse Glick wrote:
> Currently in a freeform project, you can associate
> source and Javadoc information with classpath
> entries (JARs) only indirectly - by creating a
> Java library using the same JAR, and specifying
> source and Javadoc information for that.
Could you please tell me where I can specify the libraries my 
freeform project should use?

Tools -> Library Manager as usual.

One other issue that should be addressed is that if you (even in an
ordinary non-freeform project) add a jar file to the project
classpath, and later realize that you also have sources and javadoc
for this jar file, you have to define a library and specify again the
jar files with binaries, although you've already specified it once. It
should be more straightforward to do that. 

To Petr's last comment: unrelated to this issue, and probably WONTFIX
anyway, but should be discussed separately.

*** Issue 48994 has been marked as a duplicate of this issue. ***

I hope that project dependencies will also be handled by the
implementation of this enhancement.

lebedkov: project dependencies are a separate matter. See issue #49640.

*** Issue 49770 has been marked as a duplicate of this issue. ***

*** Issue 49769 has been marked as a duplicate of this issue. ***

Uh, I'm just a lowly user... 

"project service requests to SFBQ and JFBQ"?

"UI in java/j2seplatform"?

Could someone please explain in userspeak what this feature request
specifies?

does this correspond to the main thrust of 49770 (as opposed to the
alternatives), or is that what 49677 is about?

(trying to determine if this gets at the intent of the issues marked
as duplicate)

Basically this is about being able to specify, in the GUI as well as
project.xml, locations of sources and Javadoc corresponding to
classpath entries in a freeform project. Only meaningful for entries
which are binary JARs (libraries) - if they come from other projects,
you don't need to do this, because the other project would already be
advertising the locations of sources and Javadoc. (Which is only
currently broken in the case of Javadoc location from a freeform
project - for which see issue #49677.)

In 4.0, you can simply make libraries to match your project's
classpath entries, and associate sources and Javadoc that way. But the
association is not then sharable - it is not kept as part of the
project as such. That should preferably be fixed somehow, in case the
user is keeping a copy of lib sources and/or Javadoc inside the
project and versioning it - then a different user checking out the
project ought not need to reconfigure libraries. Also it is not
obvious to set up libraries to match the freeform project's classpath
entries. That is the main thrust of this RFE - provide a special kind
of library management for use in freeform projects.

TBD whether the existing library management GUI can be somehow reused
rather than duplicating it. Issue #44035 may be related in this regard.

Re. issue #49770 - this is only related to use #1 and sort of to
alternative #1. Use #2 and alternative #2 already work, modulo #49667,
e.g. they should work if the subproject is a j2seproject. Use #3 is
separate since only a parent project uses this project's Javadoc, and
is thus essentially the same as #49667.

*** Issue 46482 has been marked as a duplicate of this issue. ***

If information is stored in freeform itself then this is potentially
API change of project.xml format.

Hopefully for 4.1; published UI spec expected soon.

So for E it was suggested to not touch the project.xml, i.e. not make
the src/javadoc info sharable per classpath entry, as this is simpler
and probably enough for most users. (Would only make a difference to
people who keep src and/or javadoc inside a versioned area of the
project directory structure and wish to specify this info once for all
users. But probably most people do not version sources or javadoc of
external libs, even if they version the binary libs themselves.)

See URL for proposed user interface. Pretty simple; basically does the
same thing as creating a library using the Library Manager, but avoids
"polluting the library namespace", and is simply easier (no need to
reenter the JAR path, fewer things to click).

Tomas do you have any time to work on this? I could work on it as
well, but you are probably most familiar with how much can be reused
from the library manager impl. (It is not strictly necessary to reuse
*anything* from the library manager impl; could also be implemented
entirely in java/freeform.)

I am not sure If I will get to this before feature freeze (18/1/2005).
Firstly I need to integrate all the API enhancements.
I will let you know tomorrow.

I've looked at UI spec, it seems simple, but I have several
implementation questions. There are several ways how to implement it.
1) FreeForm specific - implement it in FreeForm, not nice - duplicates
code.
2) Using the Libraries module - it will create the j2se library. In
this case should be the library accessible from
LibraryManager.getLibraries() API (probably yes), should be visible in
the Library Manager (probably no)? It will probably need API
enhancements, since there is no public API for creating library. If
the library should not be visible it requires either extension to
library definition or to define new folder where the non visible
libraries are stored.

I agree #2 is probably the better solution for now; agree on all
points you mention. Some new kind of API is probably needed.

Created attachment 19671 [details]
Changes to project/libraries to support library creation.

I've attached a patch of project/libraries module to support adding
and removing of libraries. There is no API for library modification at
this time, since I don't want to make the Library API object mutable
or let the client to obtain the SPI LibraryImplementation object.
There should be probably something like LibraryManager.updateLibrary
(????).
But for now the the project customizer can update library by remove + add.

Created attachment 19685 [details]
Patch od java/freeform module

I've added patch of java/freeform. The UI is not complete - no
accessibility, no enabling/disabling of edit button, JFileChoosers
don't remember last used locations.

But I've found another serious problem, the queries which return
Result. The customizer creates library for an archive file which was
already on the project classpath (it was scanned by javacore and
SFBQ.Result was cached for it, the result is EMPTY_RESULT). Now the
librarie's SFBQI starts to provide new SFBQ.Result for it, but it has
no way how to let the javacore know about it. The IDE needs an restart
to refresh the javacore cache.
There is an solution which may result in big memory consumption, there
should be an SFBQI registered as a last one, which provides for each
unrecogized archive root SFBQ.Result listening on the LibraryManager.

My other suggestion (offline), which may also result in increased
memory consumption, is similar but slightly different - to make SFBQ
itself produce a proxy over all (non-null) SFBQ.R's returned by
available SFBQI's. Would still require the j2seplatform SFBQI to
provide a non-null result for every JAR, but does not require it to be
in any particular position in lookup.

I presume this problem existed before, in that creating a library for
an existing classpath entry would not refresh javacore's cache. The
difference now is that we are promoting a UI which triggers the bug.

Possible workaround: in the freeform customizer code, whenever a fake
library is registered (or any other similar change is made), remove
and then readd the same classpath item (according to the relevant
ClassPath), in an attempt to force an event to be fired and thus to
force javacore to refresh its cache. Could that work?

Yes, the workaround works fine. I am using it for now, but I was not
sure if it is not too ugly.
Thanks

This issue will be fixed in promo-D. It requires API changes.

Sorry, not promo-D but promo-F

Which API changes does it still require? Did the suggested workaround
not work after all? Please explain.

The addLibrary (), removeLibrary () methods in the LibraryManager. And
LibraryFactory class. See the libraries.diff.

*** Issue 95439 has been marked as a duplicate of this issue. ***

Tentative plan.

*** Issue 119790 has been marked as a duplicate of this issue. ***

I suppose the freeform project type can do what j2se project type does nowadays with declaring javadoc/source for jar
files (instead of libraries)

Changing the default component owner to tzezula.

This issue is already about seven years old! As infrastructure was greatly improved in the last years, it should be too difficult to fix.

It should also considered to categorize this issue as a defect, as some basic IDE functionality is missing, IMO.

In reply to comment #38.
Unfortunately it's not the case. The infrastructure has improved for projects which are based on J2SEProject but this is not a case of the Freeform.
Also the Freeform project is not a main focus now :-(

(In reply to comment #39)
> In reply to comment #38.
> Unfortunately it's not the case. The infrastructure has improved for projects
> which are based on J2SEProject but this is not a case of the Freeform.
> Also the Freeform project is not a main focus now :-(

Thank You for interpreting my comment correctly, as I've accidently dropped the important word "not"!

I've tried to implement a fix in Java freeform project, but it doesn't work at all, probably I'm just missing some dependencies. However, I'll attach my changes as a proposal (i.e. schema and basic impl.).

Created attachment 108687 [details]
Proposed schema

Created attachment 108688 [details]
Basic (non-working) changes to JavadocQuery for proposed change

Unfortunately the fix is not so simple.
1st) The schema cannot be changed as it's non compatible API change. Either the new schema has to be introduced and the upgrade from previous versions has to be added as there is for /2 and /3 versions. Or another persistence like AuxiliaryProperties (AuxiliaryCfx).

2nd) The Freeform JFBQuery cannot be used as it's called only for freeform artifacts.

I will try to save some time and do the infrastructure. Without changing the schema as the update code is really pain. I did it for annotation processing support and it was not very nice.

(In reply to comment #43)
> Unfortunately the fix is not so simple.
> 1st) The schema cannot be changed as it's non compatible API change. Either the
> new schema has to be introduced and the upgrade from previous versions has to
> be added as there is for /2 and /3 versions. Or another persistence like
> AuxiliaryProperties (AuxiliaryCfx).
> 
> 2nd) The Freeform JFBQuery cannot be used as it's called only for freeform
> artifacts.
> 
> I will try to save some time and do the infrastructure. Without changing the
> schema as the update code is really pain. I did it for annotation processing
> support and it was not very nice.

Thank You! But I don't understand, why You cannot extend the schema, as additional element would be optional, so I'd expect them to work properly with existing projects?

Here is an example when the compatibility will be broken:
User_1 has NB 7.0 with original schema
User_2 has NB de with changed schema

User_2 creates a new Freeform with the optional new element. IDE validates the XML according to schema and everything is OK.
User_2 shares the project and User_1 downloads it
User_1 tries to open it. As the schema is /3 the IDE (7.0) i happy because it's supported version and tries to validate the project.xml according the schema but now it finds an XML element which is not in the schema and the validation fails.

(In reply to comment #45)
> Here is an example when the compatibility will be broken:
> User_1 has NB 7.0 with original schema
> User_2 has NB de with changed schema
> 
> [...]

Well, I agree this might cause problems, if one user's IDE is upgraded but the other isn't. However, try to upgrade a project using Swing forms from NB 5.x to current version directly - AFAIK, it will fail. Had those problems earlier, could only resolve the upgrade problem using intermediate versions of NB. And I guess You won't be able to downgrade most project types.

However, it's a problem for me not being able to add libraries nor javadoc, as I need to use third-party libraries (different versions for different projects, so no chance to add them globally). Currently, I'm unpacking the javadoc and then I'm viewing it in my browser, but that's very inconvinient. :-(

In the projects infrastructure we tried to upgrade the project.xml only when it's needed. There is some support for it in J2SE based projects (UpgradeHelper) and something similar is in freeform project. The IDE does not do downgrade but it has to ensure at least that it generates the metadata according to schema. And the older version does not open project of newer version.
Also the change to schema will not pass through NB API review process as it's incompatible API change.

Yes I understand importance of this feature and agree that even when the Freeform is not main focus now it should be resolved.

(In reply to comment #47)
> Yes I understand importance of this feature and agree that even when the
> Freeform is not main focus now it should be resolved.

Thank You! Just because it is so difficult to do this later: Could You probably also consider to add some libraries locally? Doing this in the same working task would probably reduce organizational overhead ...

I've just noticed a related addition to http://wiki.netbeans.org/wiki/index.php?title=EditorPlan71:

Issue 199143 Javadoc & Sources for Freeform (Tomas, 2days) 

This entry seems to be related, but the bug id is incorrect.

Thank you for pointing me on it.
I've just added it into plan but used a wrong browser tab (the issue I am just fixing instead of this one).
I've fixed it. Thanks

No need to touch the freeform schema; could be a separate project.xml section with its own schema.

Anyway I do not think anything special in freeform is needed; see comment #19. Would be useful for autoprojects etc. as well. Regardless of project type, you ought to be able to immediately associate local source and/or Javadoc with a given binary classpath entry; this association would not be persisted in project sources, only in your userdir somewhere (NbPreferences?). UI to pick source/Javadoc roots, persist them, and global SFBQ/JFBQ impls interpreting these associations should all be trivial. I am just not sure about the entry points for these UIs. I can think of

1. You invoke code completion on something and it says no sources or Javadoc are available. There should be a button to add sources or Javadoc right then, associated with that binary root.

2. Help > Javadoc References > More Javadoc... should have some button to pick a binary root from among those in GlobalPathRegistry with no existing Javadoc association, then select a Javadoc root.

Customizing or removing the associations afterwards is another issue that would need to be addressed. Also there should be some SPI so that modules which have a preferred place to store this information (e.g. Libraries) can do so.

In reply to comment #51.
In this case the binding of source to binary is completely project independent. If you close the project it's still active. Which is a bit problematic, but maybe it does not mind (having 2 projects with same jar with different binary<->source mapping is stange anyway). Also when you copy the project to someone you lose the binding, but this is probably OK (you will need to resolve broken ref anyway).

Regarding the UI. I suggest the same behavior as in case of standard ant project "Edit" button which calls some api method attachSources(....), attachJavadoc(...).

So I propose 2 simple methods
 attachSources
 attachJavadoc

If we want to resolve the problem with independence of binding on project an additional factory method is needed.

createExtraSources which creates on OpenProjectHook which needs to be put into the project's lookup which activates the bindings stored for example in NbPreferences ProjectUtil.getPreferences().

(In reply to comment #52)
> In this case the binding of source to binary is completely project independent.
> If you close the project it's still active.

Indeed. (True even for project-defined associations so long as the project can be located somehow.)

> Also when you copy the project to
> someone you lose the binding

Right, this would just be for one-off associations that are tricky to add in more permanent ways.

> If we want to resolve the problem with independence of binding on project an
> additional factory method is needed.
> 
> createExtraSources which creates on OpenProjectHook which needs to be put into
> the project's lookup which activates the bindings stored for example in
> NbPreferences ProjectUtil.getPreferences().

Possible, though I am not sure this added complexity is worth it. Anyway you might want to make an association for a binary not owned by a project.

I've think about it a bit more and it seems that you are right.
Binding it to project is a complication which makes the thing worse.
It would be good if it will be generic enough to work out of the box for freeform, autoproject and NB library wrapper module (currently for lucene I have to have an additional j2seproject lucene with sources).

I looked at Eclipse and do something similar to what Jesse described. When you navigate to some class (CTRL+Click) and it has no sources a button "Attach sources" is shown. I will do something similar, the decompiler is under java.source control so it should not be a problem. In addition to this I will change the Javadoc Code Completion window to allow to attach Javadocs.

I will put the queries implementation + some small API attachSources(binRoot, sourceRoots*), attachJavadoc(binRoot, docRoots*) into java.project.

Any comments?

(In reply to comment #54)
> I will put the queries implementation + some small API attachSources(binRoot,
> sourceRoots*), attachJavadoc(binRoot, docRoots*) into java.project.

CC apireviews when there is a patch ready to look at.

Also consider last sentence of comment #51. If the user invokes this UI on a binary root which is already capable of making source and/or Javadoc associations by existing means, e.g. a library, then the NbPreferences-based storage should not be used. This probably means that there needs to be an API+SPI similar in concept to ProjectClassPathModifier, with the preferences SPI impl being in last position, and earlier ones such as for LibraryManager.

It may also be valuable to let an SPI offer to _retrieve_ the source or Javadoc association, rather than having the user _specify_ it. In particular, in the case of JARs in the local Maven repository, associations will be offered from [SJ]FBQ when the secondary artifact has already been downloaded. If the artifact is a dependency of some open project, it is possible to request this download from the Projects tab UI. But sometimes you are coming across a local repo artifact in another context (perhaps as a classpath entry in a j2seproject), or it may simply be cumbersome to find the dependency node in Projects when your attention is focused on the editor window. So in this case "Attach Sources..." (resp. "Javadoc") should simply trigger the secondary artifact download, not bring up a filechooser.

Given these use cases, I might suggest something like:

package api;
class SourceJavadocAttacher {
  static boolean attachSources(URL root);
  static boolean attachJavadoc(URL root);
}
package spi;
interface SourceJavadocAttacherImplementation { // for @ServiceProvider
  enum SupportLevel {NONE, EXPLICIT, IMPLICIT}
  SupportLevel supportsAttachSources(URL root);
  SupportLevel supportsAttachJavadoc(URL root);
  void attachExplicitSources(URL root, URL[] sources) throws IOException;
  void attachExplicitJavadoc(URL root, URL[] javadoc) throws IOException;
  void attachImplicitSources(URL root) throws IOException;
  void attachImplicitJavadoc(URL root) throws IOException;
}

where the API methods return true on success, and display a filechooser only when the support level is EXPLICIT. (The chooser could offer a text field for a remote URL in the case of Javadoc.) Probably the Maven impls of the attachImplicit* methods would block until the download completes and show their own progress bar.

Also I think api.java is the correct place for the new API; it does not refer to projects as such, and is closely related to the existing queries. (An SPI impl using LibraryManager would belong in java.project.)

It's much more complex then the "freeform" case but it will hopefully solve the attach problems for all artifacts in all project types.
When I will have a diff I will attach it I just wanted to collect all use cases (the maven use cases were completely new for me).

Fixed jet-main http://hg.netbeans.org/jet-main/rev/a01b7480064e
When you click (Ctrl+B) on element which has no sources the disassembled view provides Attach sources button which can be used to attach the sources. The similar link for Javadoc is in Javadoc Not Found window.

*** Bug 160485 has been marked as a duplicate of this bug. ***