I don't understand what you talking about unfortunately. Reassigning to JPA support

the same for j2se project, but not sure if it's a defect. current realization do not add driver library to the project and I don't know why it was done this way.

and the same for web/ejb etc, application will fail if there is no driver in server classpath.

Ok, I'll explain. The truth is that out of box driver setup does not work in the following scenarios:

* Java Persistence in Java SE Ant project
* Java Persistence in Java SE Maven project
* Java Persistence in Java Web Maven project
* Swing Master/Detail form in Java SE Maven project (does not work at all, see bug 178976).

but it does work in the following scenarios:
* Java Persistence in Java Web Ant project (for servers that support data sources and driver deployment, the driver is copied to the server)
* Swing Master/Detail form in Java SE Ant project (driver is added to compile classpath)

I don't insist this is a defect, we could reclassify it as an enhancement, but as you see, the experience is currently inconsistent in various scenarios.

if in some cases the driver is copied to the server or added to classpath it looks inconsistent with other cases, will verify.

Part fix. I'm not sure but hope it's good. Default behavior for j2se ant/maven is changed and jdbc driver is added now on pu creation. EE behavior isn't changed neither in maven nor in ant. As I know persistence do nothing to deploy a driver on server, may be it's in deploymen module, need more investigation.
http://hg.netbeans.org/web-main/rev/54a40911c096

Integrated into 'main-golden', will be available in build *201004060201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/54a40911c096
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: fix#178970 all SE projects now add jdbc driver if able to find one after pu addition, EE projects are not modified with this commit

Petr, can you provide complete test case/steps for 
"Java Persistence in Java Web Ant project (for servers that support data
sources and driver deployment, the driver is copied to the server)"?

Thanks Sergey, I will test this. To your question, the way to test this for Ant-based Web projects, is the following:
- Create a web projects with GlassFish v3 or Tomcat  as the target server
- Set up MySQL in the IDE and create a connection against MySQL (MySQL driver is not present in Tomcat or GFv3)
- Add some persistence stuff (Entity from DB, persistence unit etc.), using the MySQL db connection set up in the previous step
- Deploy the application

=> You will see that the driver files were copied to the server installation (the exact location is server-specific).

Just tried web app+tomcat 6 and can't find where mysql connector was deployed will try gf3 also a bit later.

Ok, In gf3 case I see connector in domains libs after deployment.

I tried the Java SE Maven project in the latest build, and I am getting a UI freeze for a long time:

"AWT-EventQueue-1" prio=6 tid=0x00000001018fb800 nid=0x14ae95000 runnable [0x000000014ae90000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x00000001238093e0> (a java.io.BufferedInputStream)
	at sun.net.www.MeteredStream.read(MeteredStream.java:116)
	- locked <0x0000000123809ab8> (a sun.net.www.http.KeepAliveStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:116)
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2512)
	at org.apache.maven.wagon.AbstractWagon.transfer(AbstractWagon.java:480)
	at org.apache.maven.wagon.AbstractWagon.getTransfer(AbstractWagon.java:327)
	at org.apache.maven.wagon.AbstractWagon.getTransfer(AbstractWagon.java:299)
	at org.apache.maven.wagon.AbstractWagon.getTransfer(AbstractWagon.java:276)
	at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:97)
	at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
	at org.sonatype.nexus.index.updater.DefaultIndexUpdater$WagonFetcher.retrieve(DefaultIndexUpdater.java:605)
	at org.sonatype.nexus.index.updater.DefaultIndexUpdater.loadIndexDirectory(DefaultIndexUpdater.java:209)
	at org.sonatype.nexus.index.updater.DefaultIndexUpdater.fetchAndUpdateIndex(DefaultIndexUpdater.java:139)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl.indexLoadedRepo(NexusRepositoryIndexerImpl.java:401)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl.loadIndexingContext(NexusRepositoryIndexerImpl.java:296)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl.access$300(NexusRepositoryIndexerImpl.java:137)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl$12.run(NexusRepositoryIndexerImpl.java:842)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl$12.run(NexusRepositoryIndexerImpl.java:837)
	at org.openide.util.Mutex.writeAccess(Mutex.java:433)
	at org.netbeans.modules.maven.indexer.NexusRepositoryIndexerImpl.findBySHA1(NexusRepositoryIndexerImpl.java:837)
	at org.netbeans.modules.maven.indexer.api.RepositoryQueries.findBySHA1(RepositoryQueries.java:247)
	at org.netbeans.modules.maven.indexer.api.RepositoryQueries.findBySHA1(RepositoryQueries.java:231)
	at org.netbeans.modules.maven.CPExtender.checkRepositoryIndices(CPExtender.java:188)
	at org.netbeans.modules.maven.CPExtender.addArchiveFile(CPExtender.java:166)
	at org.netbeans.modules.maven.CPExtender.addLibrary(CPExtender.java:158)
	at org.netbeans.modules.maven.CPExtender.access$000(CPExtender.java:86)
	at org.netbeans.modules.maven.CPExtender$3.performOperation(CPExtender.java:367)
	at org.netbeans.modules.maven.CPExtender$3.performOperation(CPExtender.java:363)
	at org.netbeans.modules.maven.model.Utilities.performPOMModelOperations(Utilities.java:333)
	at org.netbeans.modules.maven.CPExtender.addLibraries(CPExtender.java:376)
	at org.netbeans.modules.maven.CPModifierLookupMerger$Extender.addLibraries(CPModifierLookupMerger.java:164)
	at org.netbeans.spi.java.project.classpath.ProjectClassPathModifierImplementation$Accessor.addLibraries(ProjectClassPathModifierImplementation.java:422)
	at org.netbeans.api.java.project.classpath.ProjectClassPathModifier.addLibraries(ProjectClassPathModifier.java:92)
	at org.netbeans.modules.j2ee.persistence.wizard.Util.addLibraryToProject(Util.java:563)
	at org.netbeans.modules.j2ee.persistence.wizard.Util.addLibraryToProject(Util.java:548)
	at org.netbeans.modules.j2ee.persistence.wizard.library.PersistenceLibrarySupport.addDriver(PersistenceLibrarySupport.java:330)
	at org.netbeans.modules.j2ee.persistence.wizard.Util.addPersistenceUnitToProject(Util.java:511)
	at org.netbeans.modules.j2ee.persistence.wizard.fromdb.RelatedCMPWizard.instantiate(RelatedCMPWizard.java:274)


Maybe similar to bug 172930?

Yes may be related. What db have you tried?

JavaDB.

I discussed with Petr Hejl who owns the driver deployment. How it is supposed to work is that in a managed JavaEE environment, you use a datasource when creating a PU. When you choose to create a new datasource, the datasource definition will be created in the setup directory in your project. When NB deploys the project, it will deploy this datasource to the server, and it will also check which driver is required for this datasource and copy it to the server. This actually already works for both Ant-based and Maven-based web projects.

What does not work is the fact that for Maven projects, target server is not set by default. So it does not offer the user to use a datasource (or create a new one), rather it uses a straight database connection, so the deployment code has nothing to work with. 

What is the solution? Should we encourage the user to set the server when creating a persistence unit?

There is/are some  issues already filed for maven (as I remember) regarding jta/local recognition in pu creation. It may be good if it's possible to find if project is EE but do not have server set(hope i's possible to detect from base cluster), then wizards may suggest to select type of transactions/datasources instead of automatic detection. What a problem there will be no available datasources anyway but user willl be able to select new one creation or enter name manually.
Als if it's true it shouldn't work for TomCat, but current commited realization will addd driver to project itself in case of tomcat.

> There is/are some  issues already filed for maven

Can you please find them? They are potentially related, so this should be considered all at once.

> but user willl be able to select new one creation or enter name manually

This is not true, as the datasource file format is server-specific. So you would have to know the server instance (or at least type).

> current commited realization will add driver to project itself in case of tomcat.

I can confirm this is the case - works well.

>This is not true, as the datasource file format is server-specific. So you
would have to know the server instance (or at least type).

then it looks hard to support persistence before know server :(

>Can you please find them? They are potentially related, so this should be
considered all at once.

178824 is related with the same problem, unknown server, but I have changed it to enhancement.
161295 this one was moved to maven category

and may be more.

Regarding freeze in maven, I can move addition of jdbc driver to separate RP call, it have some sense as driver isn't used directly in generated code and is required only later at runtime. It will still work as fast as it's in ant based projects, but driver will apear in maven based project with significant delay, after repository index download, I'm not sure if any other way exist as javadb is not part of nb libraries(as I know and url point to one in java distribution on my system) and some other drivers may not be part too. Progress bar in wizard may not be best solution also as it will require index download anyway and will take the same time to complete.

In real transfer repository index:central repository takes always forever.
Is there any way to add jdbc driver to maven project without downloading main index?

in real this call ProjectClassPathModifier.addRoots(new URL[]{u}, fo, ClassPath.COMPILE); wait download main repository.

Integrated into 'main-golden', will be available in build *201004090201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/d648a7484c20
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: fix#178970 some cleanup and move jdbc driver addition to RP.post, do not work well in maven anyway.

Created attachment 96963 [details]
Sketch of a patch

I am attaching a sketch of a patch (needs to be polished). Basically, if the server is not set (and it seems that this could also happen for Ant projects with broken server reference), the wizard that create a persistence unit should display a warning about this (the wording of this warning needs to be improved). To propagate the information about missing server, I made an incompatible change in JPAModuleInfo, but since this is friend API, I believe this is better than keeping things compatible at all cost and making the code more complicated. Next, I added a new method to J2eeProjectCapabilities (another friend API). What do you think?

Cc Ken for the wording and David (general review). Petre H, you may also want to review.

I don't worry much about new methods,
should missed server block wizard from finishing or just display warning message?

I think just warning, plus we may want to provide an easy possibility to set the server from within the wizard, but we should not require the user to set it.

(In reply to comment #23)
> Created an attachment (id=96963) [details]
> Sketch of a patch

I do not think you need to add JPAModuleInfo.isServerInstanceSet() - there is already org.netbeans.modules.j2ee.persistence.provider.ProviderUtil.isValidServerInstanceOrNone(aProject). Perhaps Maven project must just implement org.netbeans.modules.j2ee.persistence.spi.server.ServerStatusProvider and place it in its lookup?

What should the warning communicate? "Set deployment server to see and manage server data sources" ?

Hi David, thanks a lot for the tip, that did it! I pushed the change on the Maven side.
http://hg.netbeans.org/web-main/rev/8c24c40f8d5e

After this change, the appropriate wizards already display an error telling the user to set the target server. However, there are still a few problems:
- this is an error which disallows the user to continue, I would somewhat prefer a warning
- the message tells the user to use the "Resolve Missing Server Problem" dialog, which does not apply to Maven project. So the message needs to be changed
- There is an inconsistency betweem the PU wizard and the Entity Classes from DB wizard - one of them displays a "JDBC connection" combo box, the other one displays a greyed out "Datasource" combo box. I guess both should display a JDBC connection combo (as the user should be allowed to continue)

Sergey, can you please look into these? Thanks.

thanks, will look.

Is it good "Wizard use server specific data, you need to set server"? Will it cover case of not resolved server properly?
Also is it good to enable finish in ant based projects with missed server?

My suggestion is to enable creation from connections and next/finish in entity from db wizard but keep disabled in pu wizard and also disable comboboxes.

For the wording, I would put in something like this:
"It is recommended that you set the target server for this project, as the behavior of the Java Persistence features is server-dependent." Maybe the "set the target server" part could be bold. But it would be great to have Ken's input on this.

> Also is it good to enable finish in ant based projects with missed server?

Not sure, I think either way is fine.

> My suggestion is to enable creation from connections and next/finish in entity
> from db wizard but keep disabled in pu wizard and also disable comboboxes.

Why? Shouldn't they all be consistent?

Because entities are server independent when pu is server dependent and it may be useless to create pu if server isn't properly set.

i am not clear about the message yet. where and when would it appear?
This message would only apply to web apps, but the same wizard is used in SE
apps.

I am running into a couple problems:

for me, it looks like that in a maven web app, to use the entity>db wizard you
need to set the server first before creating the persistence unit and entity
classes or else the IDE hangs. setting the server will make sure that the pu
uses JTA instead of RESOURCE-LOCAL. This is not exactly obvious, so probably
there should be a warning to set the server first when using the wizards in
Maven web apps to say that setting the server first is the recommended route
for using the IDE wizards for generating files if you want to use JTA.

For a Maven SE app (in this case a platform module), there is no server. It
seems to work if i first create a persistence unit (which adds derbyclient) and
then entity>db. If i try to use entity>db directly, the IDE hangs when trying
to extract data from the db, probably because the derbyclient jar is not there.
The IDE creates the pu but doesnt add the derbyclient.

isnt the only difference between setting the server the type of available transaction in the pu?

Do you use latest build?
Yes, main one is transaction type, and from transaction type other content is determined(connection/datasource and properties).

using Build 20100412025

ah, yes, you are right. The datasource data is also determined by server info

just tried, there is a real problem in entity from db in maven. if I remember right it used to work before.

the wizard worked with Maven apps a week ago.
the maven app needs the derbyclient as a dependency.
the user can either set it explicitly or use the pu wizard to add it, though there might be different versions.


so does the IDE treat a maven SE or web app the same as an Ant SE app because no server is set?

i think that for a Maven web app, there should be a warning about setting the server first, but i am not sure the best way to allow them to do that. stopping the wizard with a warning is annoying, esp for people not using glassfish.

So i see this problem in the entity>db wizard:
Maven web app - need to set server first to create pu if they want to use JPA

Maven SE app - no server is necessary because it will be resource local

the issue with the library not added in time is a separate issue

So for Maven web app, there could be a message:

You need to specify a Java EE server to use Java Persistence.

or maybe the Create Persistence Unit checkbox is disabled for maven web apps when the server is not set and the user will need to create the pu with the pu wizard.

Yes, in entity from db wizard pu can be deselected and disabled,  but the same message need to be shown in case of new pu wizard, and the same may be shown for ant web in case of unresolved server.

Looks like I need to rolback initial issue  fix "out of the box jdbc", it's based on master/detail wizard behavior and I just tried it in fresh ide in maven j2se and it's already 10 minutes since I click finish but the wizard window is still here. Fix require much more work to be compartible with maven projects.

... finally I got outofmemory

(In reply to comment #41)
> ...I just tried it in fresh ide in
> maven j2se and it's already 10 minutes since I click finish but the wizard
> window is still here.  ... finally I got outofmemory

Possibly unrelated note: Sounds very much like problem we had with JPA annotation processor and a bug which TomasZ fixed couple of days.

canonical processor is switched off now, at time out of memory was thrown toplink libraries was added but derbyclient wasn't yet added.

http://hg.netbeans.org/web-main/rev/f512bf664c4d disabled, 
is it stopper for beta?

So does the hang happen in the beta clone? When did it start happening? Is it a regression? If it affects the beta clone, then let's file a new P1 issue with the exact description of the behavior.

yes, it was integrated on 20100407. and it's a regression for entity from db, and isn't a regression for master/detail form.

Integrated into 'main-golden', will be available in build *201004131450* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/8c24c40f8d5e
User: pjiricka@netbeans.org
Log: #178970 - implementing ServerStatusProvider on the Maven side (partial fix).

with ServerStatusProvider implementation it's not allowed now to edit persistence.xml in visual editor, should it be allowed along with allowed creation of persistence unit?

> with ServerStatusProvider implementation it's not allowed now to edit
> persistence.xml in visual editor, should it be allowed along with allowed
> creation of persistence unit?

If it was allowed, how would the datasource combobox behave? I think if it's enabled, it would need to display a warning about server not being set, so the user knows something is wrong. But IMO disabling the whole editor is fine as well.

Integrated into 'main-golden', will be available in build *201004170515* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/893f2de2f443
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: #178970 - do not block creation of entities from db if server isn't set, message isn't changed yet, warning is shown only after all errors are resolved

"resolve missing server" part is removed from message, only "Server isn't set properly" remains, will it be enough?
Also entities can be created now, pu can't be created and can't be edited with visual designer.

Integrated into 'main-golden', will be available in build *201004210200* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/e6302d94d3a9
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: #178970 - "Resolve.." is removed as it's not valid for maven project.

Integrated into 'main-golden', will be available in build *201005010200* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/
User: 
Log:

Can it be considered as fixed?

Sure, this should be closed now, although I think it would still be good to enhance the message to make it clear *why* is it bad that the server is not set - the user will not be able to work with server datasources.

will it be enough "Java Persistence features are server-dependent."?
Also what to do with "out of the box jdbc driver" issue?

> will it be enough "Java Persistence features are server-dependent."?

Yes.

> Also what to do with "out of the box jdbc driver" issue?

What is the issue?

this one was initially "No out of the box support for JDBC drivers" and see first comments.

message is modified. as it's the issue for maven support now, may be jdbc driver addition issue can be refiled once more time.

You are right of course. I filed the driver problem with Java SE as a separate bug 185500. 
For web projects with Tomcat as the target server (both Ant and Maven), note that the user experience is also influenced by bug 185497.