I have also just checked this on my version of 7.3(201302132200) and the problem is reproduce-able in that version.

reassigning to javaee/maven, but likely the source of the problem is elsewhere.
The unknown binary is added to pom when all other ways of identifying the maven GAV for a jar fail. That's apparently the case here. We add the unknown binary dependency to keep classpath consistent but user manual intervention is necessary to find the correct GAV (or create it)

GAV =-= GroupId/ArtifactId/version

Reassigning to Hibernate

Why it's P2? Does it mean project fail to compile, any other broken functionality except may be portability of the project?

I'm not sure yet which way hibernate add these jars and if it use data from connection registration, it's hard to do anything as connection data contain no information regarding maven artifacts and there are only two options in this case  - add nothing or add incomplete reference. And it my opinion add smth may be better even for maven.

The reason I made it P2 is because I failed to notice the addition of the repository that allows for this dependency to be loaded, and had initially assumed that the derby jars couldn't be located.

But after noticing this locally added repository, I do believe that P2 is a valid priority.  As part of the adding the frameworks process Netbeans automatically updates the pom file more then any user would expect, thus deciding that the project is now tied solely to the netbeans IDE.  

I understand that the connection URL, and even the dialect chosen wouldn't maintain GAV information, so it would be hard to get the appropriate dependency to be used.  But maybe netbeans in this regard goes too far then?


Here is two options that I can think of that might help: 
1. Why not provide additional configuration options to the Hibernate Framework to allow for a user to provide a DB provider, or specific GAV information?
2. Inform the user with a dialog that lets them know that the Hibernate Framework has made changes to the pom file that need to be reviewed by the user.

Of cause it's not what user may consider, but hibernate framework just don't know if it's maven based or ant based application and just add new jars to project root, in case of maven it's handled by nb as add dependencies. If it really better to not add these dependencies it's better to treat maven as unsupported case and add db driver jars for ant only, but in this case I'd prefer more feedback from users as somebody may complain it's better to have this local dependency rather then nothing. In general most wizards are platform (ant/maven) independent and maven specific configuration shouldn't be included here and hibernate shouldn't know about maven except some critical cases).

I think that this bug is actually quite important. What code is responsible for adding the Derby dependency? I see that under Tools -> Libraries, there is "Java DB Driver" library - is this library used during this scenario? If it is, then this library needs to define a Maven volume - I guess this is Jarda Havlin's task. 
Would that fix this problem?

(In reply to comment #7)
> What code is responsible for adding the Derby dependency?
I guess it's somewhere in JavaEE/Persistence.

> I see that under Tools -> Libraries, there is "Java DB Driver" library
This Ant library is registered in module "derby".

> - is this library used during this scenario?
I don't know.

> If it is, then this library needs to define a Maven volume - I guess this is 
> Jarda Havlin's task.
If it's the case, Milos, could you please briefly explain how to do it, or is there some similar definition that I could use as a template?

> 
> > If it is, then this library needs to define a Maven volume - I guess this is 
> > Jarda Havlin's task.
> If it's the case, Milos, could you please briefly explain how to do it, or is
> there some similar definition that I could use as a template?

most libraries define maven-dependencies property with the maven GAV format for the dependency to be added to the pom.

eg. 

<library version="3.0" xmlns="http://www.netbeans.org/ns/library-declaration/3">
    <name>groovy-all</name>
    <type>j2se</type>
    <localizing-bundle>org/netbeans/modules/libs/groovy/Bundle</localizing-bundle>
    <volume>
        <type>classpath</type>
        <resource>jar:nbinst://org.netbeans.modules.libs.groovy/modules/ext/groovy-all.jar!/</resource>
    </volume>
    <properties>
        <property>
            <name>maven-dependencies</name>
            <value>org.codehaus.groovy:groovy-all:2.0.1:jar</value>
        </property>
    </properties>
</library>

if such definition is not available, all jars in the library are SHA1 checked against the know maven indexes if any at that time. Since initial downloading and processing is time expensive, we cannot wait for it when triggered by the addition of library..
Then there's a few other heuristics and when these fail as well, we add the unknown binary dependency.

> see that under Tools -> Libraries, there is
"Java DB Driver" library - is this library used during this scenario?

no, connection are bind to drivers registred on services tab and these drivers have jar files registered for specific db driver, and these jars have no maven related information, it may either match or do not match jars from Ant Libraries, may be default(sample) connections use the same jars but as I know there is no binding from drivers to ant libraries, and also users are free to register own driver with completely different set of jars.

Ok, in that case we will need to think about how to insert the Maven pom somewhere in the process. If we are currently using ClassPathModifier.addRoots, then we may want to change it to addLibraries. And the db module should provide some API that exposes the driver as a Library - possibly a in-memory implementation created by o.n.m.java.api.common.util.CommonProjectUtils.createJavaLibraryImplementation (in java.api.common).

The db module would need to hardcode the information about the required Maven pom for each driver supported by NetBeans - that way this would work also for drivers that the user adds by herself and which are not distributed with the IDE.

- all fixes seems to hit usability in other areas, currently user just point to a jar and have driver registered and can start using if from within ide, what if we will force uses to create a library for each case and request pom information? most users may not want these extra clicks, but we can not require to specify it and will get 'unknown.binary' in these cases and anyway it may mean 1-2 more clicks.
- if we will hardcode some pom to specific drivers, I see we either will limit number of supported jdbc drivers as now we may support some 1.2.0.1,1.2.0.2,1.3.0.1 etc should we hardode all links? does it mean user will not be able to use fresh release of jdbc driver with previous nb version(and also fresh javadb after java update) and how much efforts it will require to maintain each release and keep these list up to date? I don't see it as a good to solution to hardoce only "tested" drivers from release notes. We have 8 vendors mentioned in release notes, in my opinion we can't hardocde versions/links for all, also users may be happy to use others db with generic jdbc support and we may know nothing about versioning at all.
- currently I can get roots for a driver from db support, we  may need to rewrite a lot of api to get some library like or real library.
- in my opinion there are to many cons for fixing this and also I have no infrastructure to fix it on hibernate side and it may require full release cycle to design on maven support/db support/and hibernate usage, in case of hibernate fix will be trivial, just use 'addLibrary' after 'DBDriver.getLibrary'(or similar) if it will be available, but I'm still confused if it worth to do.
- also in my opinion it's not the issue, at most P3 as it's not critical usability issue in my opinion, but most likeley it may be enhancment request but it will require a lot of work in different areas and I'm nor sure it worth to do. We just do not support this kind of functionality and information for db driver yet and user if he know can update 'unknown' easily if move project to some not portable location.

Ok, but I still think that this leaves 2 choices:

1. You could keep the existing functionality as is, bu just provide the user with a dialog to tell them that their pom file has specific netbeans related changes, and will need to be reviewed if they intend to use a CI server, etc...

2. Don't add the dependency on Derby or any other database to the pom file.  If you can't make changes that are not portable, and don't make hidden changes to the pom(using a local repository that magically gets the a jar) then don't.  I would much rather prefer to be told by a dialog mentioning that I still need to add the dependency for the database used for my selected connection string.

As a user I would prefer option 2.  This way I can specify a specific dependency myself, depending on my needs.

(In reply to comment #12)
> - if we will hardcode some pom to specific drivers, I see we either will limit
> number of supported jdbc drivers as now we may support some
> 1.2.0.1,1.2.0.2,1.3.0.1 etc should we hardode all links?
It will be difficult to maintain dependencies for various versions of the same driver. Maybe we could use some heuristics, or show a dialog where the user can select the correct version.

> - in my opinion there are to many cons for fixing this and also I have no
> infrastructure to fix it on hibernate side and it may require full release
> cycle to design on maven support/db support/and hibernate usage
Agree, it would be a lot of work, new APIs, a DB-Persistence-Maven Bridge, etc.

> [...] user if he know can update 'unknown' easily if move project to
> some not portable location.
I suppose that the biggest problem is that the user doesn't know that the project is not portable (until it's started on different computer, which is too late).

I think it's a general Maven issue, that can happen with other libraries, not only with JDBC drivers.
What about showing an info message when "unknown.binary" dependency is added to the POM, as the reporter suggests:

(In reply to comment #5)
> 2. Inform the user with a dialog that lets them know that the Hibernate
> Framework has made changes to the pom file that need to be reviewed by the
> user.

This solution is simple and seems to be sufficient, at least for now.

>2. Don't add the dependency on Derby or any other database to the pom file.

I prefer this one too, it may be done relatively easily in my opinion, to check add some check if project support "good" extending with addRoots i.e. usual jars or not.
The only item need to be tested how some features like hibernate wizards and especially hql editor will work before user register jdbc driver manually.

>What about showing an info message when "unknown.binary" dependency is added to
the POM, as the reporter suggests:

And this one may be even more preferable but I don't know what module should be responsible to show this warning and if it's ok to have general handling to disable/enable this warning if users are happy with 'unknown.binary' and don't want to see these kind of warnings any more. It may be nice to have it on maven project support level. 
But  another way is to show warning (in several places - web project wizard, in web project properties framework configuration, in hibernate configuration wizard) but still add a dependency.

> what if we will force uses to create a library for each case and request
> pom information? most users may not want these extra clicks,

I agree, we should not add new UI and require the user to specify moe information. That's not a good option.

> - if we will hardcode some pom to specific drivers, I see we either will limit
> number of supported jdbc drivers as now we may support some
> 1.2.0.1,1.2.0.2,1.3.0.1 etc should we hardode all links?

I don't think we need to hardcode all these. It's enough if we just hardcode the latest version. Especially for the drivers that are present in NetBeans out of the box (Derby, MySQL, Postgres) this should be easy.

> does it mean user will
> not be able to use fresh release of jdbc driver with previous nb version

No. The user can change the version manually in the pom file any time. It's still much better to add a slightly outdated version of the library to the project, than to add unknown.binary.

> We have 8 vendors mentioned in release notes, in my opinion we can't 
> hardocde versions/links for all

If we only consider the latest version, the it's 8 links - is that too much? We already have many more links to pom in the IDE.

> I think it's a general Maven issue, that can happen with other libraries, not
> only with JDBC drivers.

Yes, but for other libraries this is being addressed. We had several cases before where the IDE was adding unknown.library, and these are fixed. Note that if there is a bundled library in NetBeans, in most cases it specifies the pom location. The MySQL and PostgreSQL library declarations under db.drivers don't do that - this part should be VERY easy to fix.

> Agree, it would be a lot of work, new APIs, a DB-Persistence-Maven Bridge, etc.

Why would we need a DB-Persistence-Maven Bridge? I think it's just the matter of one new API method to return library rather than jar of the driver, no? While agree that not adding anything (satory's option 2) is better than the current state, I don't think that fixing this for the mainstream usecases should be that much work.

With know dependency for a few latest or bundled jdbc drivers, will it mean the issue is resolved or we will start to consider it P4 after this part fix and will never resolve completely.

In my opinion, we shouldn't just use o.n.m.java.api.common.util.CommonProjectUtils.createJavaLibraryImplementation for jars registered in JavaDB driver in the ide and match it to some hardcoded pom values as user can update this driver and point to another version and use will get mismatch between what is expected and what is created in pom. Use can create one several more JavaDB driver registrations also. In my opinion we either should switch to usage of real libraries or keep the state as is.
In my default 7.3 installation, javadb driver registration use jars from stand alone javadb installation,  it's not bundled one and I can't find any derby*jar in nb installation directory.
Part fix for a few limited use cases(and may be with wrong result after additional configurations made by usrs) can be done with just one more api in db but I'm not sure we may be happy with this fix.

regarding other libraries,
Just check icefaces solution, I see only one hardocded item here for one icefaces version (and one version for rickfaces etc).
Also it's regarding framework where even ant part is usually stick to specific version only, jdbc drivers are not so limited.

> The MySQL and PostgreSQL library declarations under db.drivers don't
> do that - this part should be VERY easy to fix.

I've just checked it and the libraries registered in the IDE already contain Maven info,
e.g. PostgreSQLDriver.xml contains:

    <property>
        <name>maven-dependencies</name>
        <value>postgresql:postgresql:9.2-1002.jdbc4:jar</value>
    </property>

(Thank you, Milos, for the hint.)

But JDBC drivers are registered differently into the DB support, without any Maven-related metadata.

> > Agree, it would be a lot of work, new APIs, a DB-Persistence-Maven Bridge, etc.
> Why would we need a DB-Persistence-Maven Bridge?
True, if we return the Maven GAV as a string, DB wouldn't need to depend on Maven.

> I think it's just the matter of one new API method to return library rather than jar 
> of the driver, no?
We could check if the driver has corresponding library registered in the IDE, and if so, return the library. (But it could be done in Persistence module, without any API change at all.)


> With know dependency for a few latest or bundled jdbc drivers, will it mean the
> issue is resolved or we will start to consider it P4 after this part fix and
> will never resolve completely.
Maybe this solution is not so bad.
If we find registered library for the driver, let's use its Maven info (e.g. 70% users will be happy).
If there is no library for the driver, show some notification (remaining 30% users will be informed).

(Problem are drivers that contain no version info, so they can be matched to incorrect library. This is probably not critical).

>Problem are drivers that contain no version info, so they can be matched to
incorrect library. This is probably not critical).

If we'll decide it's not critical or If I'll be able to match just roots and find if these roots are present in a library and if match use it, it can be done on hibernate side and I can implement it based on a decision. 
But even with this solution I worry if I'll add some library with hundreds of jars/dependencies just because it contain the same derby.jar, but we can also consider 99.9% of cases are not so bad, but if 1 user will file an issue "I got 100 of new dependencies" what we will do with this issue? It looks like a serious one. Or even 10/few extra dependencies may broke a project.

And one more with libraries already registered, these libraries may contain pom information, but there may be others with the same jdbc driver jar and without this information, I need to hardcode preferred libraries names for bundled libraries and check only preferred. In this case there will be no problem with extra dependencies in 99.9% of cases(if user will not replace/update precreated library), and what to do with remaining 30%? 

Jaroslav, is there any maven information for derby?

(In reply to comment #20)
> > Problem are drivers that contain no version info, so they can be matched to
> > incorrect library. This is probably not critical).
> [...] It looks like a serious one. [...]
Indeed, there is still a lot of possible problems.
If you find it too risky, I'm still fine with solution "2" (notification).

(In reply to comment #21)
> Jaroslav, is there any maven information for derby?
No. But I can add it. Here's again the problem with correct version, I'll have to use latest stable one (currently "10.10.1.1").

> I've just checked it and the libraries registered in the IDE already 
> contain Maven info, e.g. PostgreSQLDriver.xml...

I you are right, sorry I was looking at the wrong file.

It's true that we may be able to solve this on the Hibernate side without an API change, but as Sergey says, there are several risks and potential problems.

To me it would sound cleaner and more logical if the db module could provide the API, because it knows which libraries it is adding, so it can provide this information accurately without these problems Sergey mentions.


But there is one other question I just thought of. Should the driver be packaged to war+deployed at all? Shouldn't it be copied directly to the server? I am not a Hibernate expert, so I am not sure what's typical workflow.

for project execution it may be better to copy to a server, but we have also j2se maven project without any servers. also server is easy to change especially for maven. from these points we may not need to review current approach withing this issue scope.

on other side regarding hibernate support, it may be possible to avoid jdbc driver addition for framework functionality as it's done in case of jpql editor where jdbc driver jar is added virtually only for overloaded classloader and it works independently on existence of driver on project's classpath.
the question if we need to add jdbc driver "out of the box" as we have also old request for jpa to have this functionality we may need to review this request and decide "to add or not to add".

> we have also j2se maven project without any servers

You are right, for j2se it should be added. Still, for web project I would say let's not add it.

> it may be possible to avoid jdbc driver addition for framework functionality
> as it's done in case of jpql editor where jdbc driver jar is added virtually

Yes, this sounds it would be useful to do.

> we have also old request for jpa to have this functionality

Do you have a pointer? Is there a bug number?

Integrated into 'main-golden', will be available in build *201306181109* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/83c5ec7f94f3
User: Sergey B. Petrov <sj-nb@netbeans.org>
Log: #230841 add driver urls to classloader directly(no need to register jdbc driver on project classpath for framework supprt), need to fix addition of driver in j2se environment as for now it's always skipped.

jdbc driver isn't added in case of framework configuration on last step of ant web project wizard. this step isn't supported in maven. from now jdbc driver isn't added also in case of hibernate support added with hibernate configuration wizard except j2se projects where jdbc driver is added and still with unknown.binary in case of maven, more work need to be done to support proper dependencies.

Created attachment 136024 [details]
Patch - changes in JDBCDriver and Derby library

Patch with changes in DB modules that could be used in the future (API review needed):

 - db:    new method JDBCDriver.getDriverLibrary()
 - derby: add Maven info for library Java DB Driver
          (version is hardcoded, better solutions are welcome)

This old bug may not be relevant anymore. If you can still reproduce it in 8.2 development builds please reopen this issue.

Thanks for your cooperation,
NetBeans IDE 8.2 Release Boss