Current Form on stavbicka branch:
---------------------------------
The global Form module classloader was temporarily modified to use
JavaPlatform.getDefault().getBootstrapLibraries().getClassLoader(false).
Now all JDK beans work. The InstallBean and InstallToPalette actions
were disabled till they are fully reimplemented.

What remains to do:
-------------------
* change persistence format of beans installed in palette. At the
moment bean installed to palette is persisted as empty
fully-qualified-bean-class-name.instance file on SystemFS. This is not
sufficient information to recreate bean and must be extend with at
least absolute jar filename which owns the bean or perhaps even the
whole classpath required by the bean.

* 3rd party beans can be installed in IDE as J2SE Library or perhaps
Form can has its own Java Beans Library type (TDB whether this is
really needed). The library approach will be necessary whenever a bean
consist of multiple jars, but this is already supported by J2SE Library.

* InstallBean and InstallToPalette actions must be fixed

Possible solutions outline:
---------------------------
Two solutions are possible:

Solution #1:
------------
The workflow in this solution is more or less similar to what we have
currently in NB3.6.

* InstallToPalette action can work as today and just store in addition
also filename of jar owning the bean.

* InstallBean action on Java files can be reimplemented to find
project owning that Java file, retrieve jar filename produced by that
project and installing into palette the Java class name + the jar's
absolute name. I guess that AntArtifactQuery could be used to find the
jar which will contain the bean?

Open issues:
* installing bean from palette to project requires project's classpath
modification and this is not currently supported by API. The action
would have to be implemented directly in project type.
* removing a bean from a form cannot update project's classpath. User
would have to remove unused bean jar manually.

Solution #2:
------------
In this solution the Component Palette has project specific content
and it reflects classpath of the project whose form is being edited.
There are no InstallBean and InstallToPalette actions. User's workflow
is that first they modify project's classpath and add any required
beans jar or Bean Libraries to it. The component palette when shown
for a form #1) find a project owning the form and #2) scans jars on
project's classpath for JavaBean entries in their manifests and #3)
automatically shows all available beans.

Open issues:
* there needs to be a way to find the manifest corresponding to a Java
source file (if any). This might be useful for other modules than just
form and is under consideration of buildsys team anyway
* if there is some programmatic need to edit manifest contents (e.g.
to mark certain files as beans), there probably needs to be some
VCS-safe structure editing API for manifests. Using
java.util.jar.Manifest freely reformats everything, rearranges
attributes randomly, etc. 

-----

The biggest difference is in user's workflow. The Solution #1 shows
user all available components in the IDE and by dragging them to form
the project's classpath is (automagically) updated. The Solution #2 on
the other hand presents to user only beans which are available for the
given project classpath and user has to explicitely modify classpath
to affect component palette content.

The second solution is easier to implement and is preferred from out
point of view.

Note: the "open issues" for solution #2 actually have nothing to do
with the form module and are not blockers for this solution. Actually
they are not specific to solution #2 either. Those considerations
apply more to the beans module - ideally you would be able to use the
IDE's GUI to update the manifest for a bean JAR you were creating by
marking certain classes as JavaBeans you wish to export. So best to
ignore those open issues here - AFAIK solution #2 should be
straightforward to implement with no special work outside the form module.

Form module classloader is returned at the moment in
FormUtils.getClassLoader() instead of default platform classloader.
Otherwise layout support classes defined in Form module could not be
loaded.
  
Form module also hides TimerBean which is registered by userguide
module because it is causing CNFE exceptions.

Solution #2 has some drawbacks - the most significant is that the user
cannot customize the palette, its content is derived automatically for
each project based on its libraries. BTW the project scanning seems to
be problematic to me - besides performance issues (you need to scan
each time a form is changed, cache results somehow, etc), it's rather
big restriction to look for marked beans in JARs only. In NB 3.6 the
user can have any class in palette, not only form JARs and not only
marked beans. But adding virtually everything to the palette is
neither desired. I think the auto-defining palette mentioned in #2 is
not satisfactory.

The basic requirement that should be met is:
User needs to customize the palete. They should be able to defined the
contenet and put anything to the palette - beana from JAR (library),
beans from another project, etc.

> you need to scan each time a form is changed

I meant "each time a form (project) is switched..."

First, what our competitors do in this area? Is there a common
expectation how the component palette should behave? 

The drawback you mentioned can be seen also as Feature. IMO, it is
just a matter of point of view. Opposite argumentation could be: user
is working on 10 projects, each is using different 3rd party bean
jars; user installs all beans from all jars into palette and is
dissatisfied because it is too cluttered now; user would prefer to see
in palette only beans which are relevant to particular project.

"it's rather big restriction to look for marked beans in JARs only" -
as Jesse suggested in off-line discussion the Java Library customizer
could have separate panel for beans customization and this UI could
allow even manual selection of beans.

Re. performance - I really don't see how that would be an issue. In
the simplest case, keep an in-VM (non-persistent) weak map from
ClassPath instances to a list of dynamic bean categories (each having
a list of bean class names or similar). No need to keep a persistent
cache on disk. How long does it take to open a couple of JARs and look
at their manifests (needed the first time a form from a particular
project is opened)? Probably on the order of 10-20msec, i.e. no more
than you would spend loading a persistent palette configuration. No
need to rescan when the user switches editor tabs - you just show the
cached palette list for the ClassPath corresponding to the active form.

I would argue that being able to manually customize the palette is
*not* a basic requirement. To the contrary, adding extra things to the
UI to be customizable is a small negative factor - any time the user
spends setting up the IDE (or deciding whether they need to set up the
IDE) rather than working is time lost, so it should only be offered if
the alternative is an unusable environment (e.g. missing beans they
need to use, or flooded by beans they will never use).

The second problem (too many beans) seems unlikely to me in practice.
After all, we show all the JRE "beans" and it is not a big deal. So
the main issue is ensuring that the user gets all the beans they
requested. Possibly the user could also care about the order of tabs
and/or beans in a tab, but I don't know if this is a serious issue or not.

Not clear why we should support beans packaged in JARs that are not
marked with the attributes required by the JavaBeans specification for
marking JavaBeans packaged in JARs, but if necessary it is possible
perhaps using Java library annotations, as David says. Seems better
than customizing the palette since you keep that info directly
associated with the library.

Main open issue with #2 is probably what to do with beans from the
same project or from its subprojects. Have already suggested that
there be a way to mark a class you are developing as a bean, which
would simply mark the project's manifest; that would be consistent
with the rest of the style of #2. This would probably be desirable
regardless of what solution is adopted for the palette, since if you
are developing a library of beans it ought to be made easy to package
them correctly.

Note another possible style: make the palette customizable on a
per-project basis, within the classpath available for that project
only. Could be stored e.g. using ExtensibleMetadataProvider. I don't
think this is a very good solution though.

Any feedback from HIE yet?

As for competitors, I've checked Idea 4, Eclipse with VE 0.5, and
JBuilderX.

Idea requires the user to manually write a xml palette descriptor file
if they want use custom beans and include manually the JAR with the
beans in lib install dir. Quite cumbersome, looks rather like a
extemporary solution.

Visual editor in Eclipse seems not to solve the palette customization
at all (yet - it's just 0.5 version). It offers just "Choose bean"
functionality, allowing the user to choose a bean from project
classpath (in a special dialog).

JBuilder manages the palette globally, the user may add items from
registered libraries, can fully customize the palette content in a
palette customizer. Items added to palette are always available, when
a bean is added to a form, coresponding library is automatically added
to project.

I've talked about this with Jano and Dusan. Guys, could you add your
opinion here?

So IDEA and Eclipse have no good solution and JBuilder uses style #1,
it sounds like.

From the user perspective, the palette is a global repository of visual and non-visual 
components. That is how it is presented in all of the IDEs supporting visual editing I am 
aware of (JBuilder, Delphi, VS.Net).

The user scenario is following:

1. The user downloads third party beans.
2. Puts them into the palette (into categories if necessary).
3. Creates a new project and starts visual editing by putting components into a visual 
form.

Please note that "UI Developer" is a bit different user type (persona :) than "Coder". JAR on 
the classpath is really just a packaging mechanism the UI Developed doesn't want to deal 
with directly. I believe that the user expects IDE to take care of this.

I admit that there might be certain portion of users who would prefer the project sensitive 
palette, but I find it a corner case.

So in that case the question remains how the magical updating of the
classpath should work, in several parts:

(1) Internally, i.e. what consolidation-private API java/j2seproject
should provide to form to permit it to add a classpath entry.

(2) Whether j2seproject is expected to add the right kind of dep, i.e.
plain JAR vs. lib vs. subproject, automatically.

(3) What happens - if anything - when the user removes that bean from
the form. Should the classpath entry be removed? Probably impossible,
which means that adding and then immediately removing a bean may cause
the project's configuration to be modified spuriously.

(4) Whether the user should be prompted about (2) and/or (3) or
whether it should be silent.

There is also then the need to produce a new format for the palette
which would encode not only bean name but owning classpath (for
3rd-party beans) or project (for developed beans). *.instance files
are not enough.

I think form editor should allow the user to add beans from either:
- external JAR,
- library defined in the IDE,
- opened project.

These three types of bean "sources" should be kept by the form editor
(i.e. stored in the palette items) for each bean in the palette. I
can't imagine a general way of describing where the bean comes from
and the j2se project adding the right type of dependency
automatically. What do you think? I think form editor needs three
(e.g.) methods in API - add a library, add a JAR, add a project. Each
method would have parameter of different type. Before the form editor
would really add anything to project, it would check (for given bean)
if it is not already available in the project (execution) classpath.
If yes, nothing would be added. Does this answer (1) and (2)?

As for (3) - this is a problem, I don't know a solution. The classpath
entry remains there. BTW JBuilder behaves the same way.

Ad (4) - I think the user could be prompted about adding classpath
entry and the consequences - with checbox "Don't notify me again".

I think that:

Ad (3): The user should take care of removing the unused items from the classpath.
Ad (4): Agreed with Tomas. The user should be prompted about adding an item to the 
classpath (with "Don't notify me again" check box).

FYI: UI spec available at:
http://form.netbeans.org/proposals/BuildSysIntegration_user.html

I'm going to sum up the impl. requirements.

Just setting a more useful summary.

Tomas, what is the status? 

The UI spec has not been reviewed yet. But anyway, it looks like the
following use cases need to be implemented:

- Choose a library
The user selects an installed library to choose some beans from it to
add to pallete.
Impl: I assume I use LibraryManager.getLibraries and take libraries of
type J2SELibraryTypeProvider.LIBRARY_TYPE

- Scan library for beans
The user selects some beans from the library. The library needs to be
scanned for available beans.
Impl: get list of roots (URLs) from Library.getContent("classpath"),
then scan them recursively testing each file if it is a bean. Don't
know yet how a JAR file will look like here (in such case the beans
could be identified just from the manifest).

- Choose a project
The user selects a project to choose some beans from it to add to palette.
Impl: using ProjectChooser.projectChooser and ProjectManager.findProject

- Scan project for beans
The user selects some beans from the project. The project needs to be
scanned for available beans.
Impl: using SourceForBinaryQuery.findSourceRoot, or querying for
source roots using Sources from project's lookup. Don't know how to
get the binary root for the first case, or how to filter out the test
root in the second one. Having the roots, all FileObjects can be
scanned if they are beans. I'm not sure yet about the way how to
recognize something is a bean - maybe take inspiration from
ClassDataObject.ClazzInstanceSupport.isJavaBean (using src model), or
just get the classpath, try to load the class and introspect it (not
very good from performance & memory point of view).

- Load a bean class
When a bean is used in a form, the form editor will load it using the
project execution class path.

- Add a jar to project classpath
- Add a library to project classpath
- Add a project to project classpath (as subproject)
These all are the operations that should be done automatically when
the user adds some components from palette to a form.
Impl: It seems all of these use cases can be achieved by just
modifying the J2SE project properties (i.e. javac.classpath property).
But would be nice if there was some (friend) API in J2SE project so I
would need not use AntProjectHelper (i.e. form would depend on Ant
project) and deal with the exact representation of jar, library and
project entries in the classpath property.

So could you please review my impl. assumptions if they are correct?
Any chance the API for modifying the project classpath to be implemented?

Re. finding a manifest from a lib root - simple, just look for url +
"META-INF/MANIFEST.MF" and read it if it exists.

Re. finding class roots in a given project - there is no API to do
this as such. AntArtifactQuery will however give you a list of JARs
produced by the project (if any). Or you can use Sources.TYPE_JAVA if
you are more interested in the source files, but in that case you have
no reliable way of finding what the output JAR is.

Re. using AntProjectHelper from the j2seproject - you can't anyway,
there is no way to get it.

Re. making modifications to the j2seproject's classpath safely using a
friend API - one simple impl would just be to add the ReferenceHelper
to the j2seproject's lookup (and document in arch summaries that this
is intended only for use from the form editor for now). This would
allow form to manipulate the classpath of the project in a pretty safe
fashion (adding JARs and subprojects). It would also allow form to
*remove* items, which we probably do not want to have happen, but I
guess that is not a real problem. I think currently ReferenceHelper
does not deal with libraries directly but that should probably change
anyway.

Generally, whatever APIs you think you may need added in order to
support your use cases, please file individual RFEs in the proper
component, marked with the API keyword, with any preliminary
suggestions for the form of the API, as well as your exact
requirements and intended use cases; and make sure your manager
explicitly lists that RFE as a requirement from the build system team
in your official plans for promo-D.

Thanks for the answers.

Issue 42631 filed.
Implementation plan updated: http://form.netbeans.org/plans/promoD.html

The J2SEProject provides implementation of ClassPathExtender interface.
The Form module should use its addLibrary, addArchiveFile and
addAntArtifact methods.

All project related functionality and main features from the UI spec
were implemented:
- form editor is aware of project classpath a form belongs to,
  components for that form are loaded from the project classpath
- palette manager allows the user to install beans from a JAR file,
  from an installed library, or from a built project,
- any JavaBean component in the project explorer can be added to
  palette using contextual Tools | Add To Palette,
- adding a component to a form automatically updates classpath of
  the project the form belongs to, so the form's source code is
  compilable with the component