Bug 133451 - address upgrade path for libraries shipping with IDE
address upgrade path for libraries shipping with IDE
Status: NEW
Product: projects
Classification: Unclassified
Component: Libraries
6.x
All All
: P2 with 1 vote (vote)
: 6.x
Assigned To: Tomas Zezula
issues@projects
: PLAN, UI
: 140959 (view as bug list)
Depends on:
Blocks: 119363
  Show dependency treegraph
 
Reported: 2008-04-21 13:03 UTC by Milos Kleint
Modified: 2010-07-30 15:16 UTC (History)
8 users (show)

See Also:
Issue Type: TASK
:


Attachments
Library Manager - obsolite library detection (69.17 KB, image/jpeg)
2008-10-01 08:52 UTC, Milan Kuchtiak
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Milos Kleint 2008-04-21 13:03:27 UTC
the shared libraries effort for 6.1 created a situation that we didn't have in previous incarnations of the IDE.
the libraries provided by the IDE's modules are identified by the string identifier only. when the user updates the IDE,
the libraries in user's projects are automatically used from the new ide installation.
For shared libraries since 6.1 this "autoupdate" feature will not longer work.


This issue is an umbrella to collect potencial and real  problems with users having old library binaries in their
projects and to discuss and come up with a solution on how and when to upgrade users to new binaries.
Comment 1 Milos Kleint 2008-05-19 08:00:33 UTC
I keep wondering if we actually need any sort of APIs and UI. Maybe writing up some guidelines how to tackle the problem
(when it arises) should be enough. Let me explain in more detail.


The owner of the library knows the SHA1 checksums of the old version, new version and knows how severe the problem is. 
Computing the SHA1 checksum of the actual user's bits is a trivial task. (The only one who's interested in the checksum
of a particular library is the library owner most probably so setting up infrastructure to cache the checksums somehow
is not adding value)
These days any module can submit a ProjectOpenedHook (via LookupProvider) so it should be easy get the foot in the door
to  start the checksum calculations.

So maybe this task boils down to providing a utility method that calculates a SHA1 checksum?

Comments?
Comment 2 David Konecny 2008-05-20 04:08:31 UTC
I have a tendency to agree with you Milos. The only thing I afraid is that module owners realize they need to do
something two days before NB6.5 RC2.

How would this work for 3rd party modules: eg. form module when generating JPanel to a project checks (apart form other
things) presence of a class from swing-layout-1.0.3.jar on project's classpath. If such a well known class is not there
ts adds the jar to project classpath via classpath modifier. If in NB6.5 form comes with swing-layout-2.0.jar what
should they do? Are you saying they should impl ProjectOpenHook?
Comment 3 Milos Kleint 2008-05-21 09:44:42 UTC
depends. The solution is different for jars and libraries.
for jars that get added on classpath, the proper process is:
1. at the time you would add the binary to the project, check for class from the binary.
2. that way you figure the jar it resides in, calculate it's SHA1.
3. if not good, remove the old one via the ClassPathModifier api and add the new one.


for libraries
1. at the time you would add the library, check for class from the binary. Not sure if it's better to check for the
class or got library name. AFAIK 3rd party modules cannot figure out the correct LibraryManager instance as the library
path for the project is accessible by AntProjectHelper which isn't exposed outside of the project (correct me if I'm wrong)
2. again after you figured the actual binary(ies), check the SHA1 token.
3. if not good, either replace the definition of library in librarymanager or add the new library under different name.
None is currently possible I think. removing a library from the project will just nuke the reference and adding the
library later with new content one will only delegate to the old one (with old binaries).

So I guess we actually need to do something about libraries upgrade from 3rd party modules. Any ideas?
Comment 4 Milos Kleint 2008-05-26 08:57:40 UTC
a possible solution for libraries is to create a new library named string2.6 for example, remove the old library
reference from project (spring) and add the new library (spring26). That way the new library gets added to shared
location and project references the new library.
Comment 5 Antonin Nebuzelsky 2008-05-26 12:30:10 UTC
> remove the old library reference from project (spring) and add the new library (spring26)

If this approach is chosen, we need to communicate this in advance (before FF of 6.5) to all developers.

An FAQ should be created in the Developer FAQs area (http://wiki.netbeans.org/NetBeansDeveloperFAQ) and announced loudly.
Comment 6 Antonin Nebuzelsky 2008-05-26 12:30:24 UTC
> remove the old library reference from project (spring) and add the new library (spring26)

If this approach is chosen, we need to communicate this well in advance (before FF of 6.5) to all developers.

An FAQ should be created in the Developer FAQs area (http://wiki.netbeans.org/NetBeansDeveloperFAQ) and announced loudly.
Comment 7 Antonin Nebuzelsky 2008-07-16 17:32:37 UTC
Milos, David, this issue should be resolved now, with some conclusion and a follow-up for the parts of IDE which provide
libraries for projects...
Comment 8 fommil 2008-09-04 12:17:29 UTC
*** Issue 140959 has been marked as a duplicate of this issue. ***
Comment 9 David Konecny 2008-09-10 05:32:52 UTC
Four months ago I said "The only thing I afraid is that module owners realize they need to do something two days before
NB6.5 RC2" and funnily enough I just realized I need to fix <copylibs> tasks. See separate issue 146736 for more details.
Comment 10 Milan Kuchtiak 2008-09-29 10:17:27 UTC
We have a similar problem with upgrading JAX-WS 2.1 library from 2.1.2 to 2.1.3 (implementation version)

Renaming library name for this purpose doesn't seem to be a conceptual solution.
Looking at library-declaration DTD I found there is a library:version attribute that should be used to identify the
version of that library:
http://www.netbeans.org/dtds/library-declaration-1_0.dtd

As a module developer, I'd like to have ability to detect the version of library used in project regardless the library
is in shared libraries or not.

Another option is update the library (in shared libraries) automatically when this version attribute is changed, or -
better - notify user that the library was upgraded, in a confirmation dialog, asking if jars in shared libraries should
be upgraded or not.
Comment 11 Milos Kleint 2008-09-29 12:41:08 UTC
mkuchtiak: unfortunately the version field is meant to denote the version of the dtd document, rather than the version
of the actual library. We don't have a version of the library encoded anywhere. I agree that it is useful, I can't think
of why we don't have it. Having just the name of the library as identifier seems wrong to me, but I might just be too
maven centric here.

"As a module developer, I'd like to have ability to detect the version of library used in project regardless the library
is in shared libraries or not." 
Unfortunately  you will never have the capability, unless we explicitly start declaring versions in the libraries. And
eventually we will need to leak the "version" concept to the user as the user is allowed to change the content of the
library. So if the IDE ships with 1.2.1, the user changes to 1.2.3 manually and we ship the next version of netbeans
with 1.2.2 for some reason, how will we know what is the old version in user's space and if we shall upgrade? It's
impossible without user interaction either at time when the library is changed or when the user upgrades the library.

Always silently upgrading the user's libraries is a big non-no for me, no matter that we did it for a few releases
anyway. The asking for upgrade can also be tricky, as the user neither the libraries infrastructure knows what are the
consequences of upgrading/not upgrading. The only one who knows is the module providing the library.


Comment 12 David Konecny 2008-10-01 00:18:51 UTC
Perhaps we should just fix libraries versioning. If a library does not have a version it would be considered being
"0.0"; any user defined/altered library is special "USER" version. That would let module providers to version their
libraries and upgrade them but not override libraries customized by user. Change like this is backward compatible,
should be relatively simple to implement and does not force users to deal with versioning in the UI. Advanced users
could tweak versions directly in definition files if they really want to do that. Do you think a solution like this
could work? Milos?
Comment 13 Milos Kleint 2008-10-01 07:36:17 UTC
so far we had only issues with libraries used in build script. 

The usecase I'm worried with the silent upgrade is this: 
people use 6.1 on their production projects, they beta test 65, we upgrade the libs and build script, they go down to
6.1, we regenerate the old build script (do we?) but the new library stays. Actually we regenerate the build script in
some situations only and therefore potentially a shared library location can serve both old and new build scripts.

That's a reoccuring problem and IMHO one of the reasons some people pick freeform over the IDE maintained scripts.

There basically only 2 usecases when we ever need to upgrade the user forcefully anyway (IMHO).
1. it's needed by the buildscript (our 2 usecases we already had)
2. it's needed by some of IDE's code generation features. I would argue that in this situation the IDE should bend, not
the user and we should generate different code based on the classpath of the project. (We do in case of api support file
wizards for example)

On the general level I'd rather go a way of allowing the user to *manually* synchronize/upgrade their shared lib content
with the content from the IDE, rather than doing a silent/forced upgrade of everything based on the criteria we pick.
However the IDE will never be a full scale software library repository, so not sure how complex the UI shall actually be..

Comment 14 Milan Kuchtiak 2008-10-01 08:50:21 UTC
We discussed our use cases with Milos this week. In our case, we use JAX-WS 2.1 library for purposes :

1. it's needed for build script only (ant task puts the library on classpath)
2. it's needed for project (project's classpath) only
3. needed for both

At least we(plugin) should be able to detect that library located in shared libraries is different from IDE version
Also user should be able to upgrade the library easily. See the picture.

I noticed, for now even you remove the library from shared libraries, you only remove the reference. Then, if you import
the library again, another folder (Library_Name-2) is created.
Comment 15 Milan Kuchtiak 2008-10-01 08:52:07 UTC
Created attachment 70952 [details]
Library Manager - obsolite library detection
Comment 16 David Konecny 2008-10-02 06:13:16 UTC
Re. "people use 6.1 on their production projects, they beta test 65, ... they go down to 6.1" - I wonder if in practice
this really is an issue. Your project and sharable libraries are in VCS. If you try out newer version of IDE you can see
all the changes IDE did (VCS diff) and you can always revert/ignore them and commit them once whole your team migrated
to newer version of IDE. 

Re. "they go down to 6.1, we regenerate the old build script (do we?)" - yes, we do. genfiles.properties keeps CRC of
stylesheet which will be different in older version

Previously I was talking only about fixing libraries infrastructure by adding a version to library definition. How such
an infrastructure would be used by IDE does depend on usage scenario. In three scenarios Milan described:

1. it's needed for build script only - in this instance upgrade should happen automatically otherwise IDE will not work;
it is internal detail/behaviour and user should not be involved; it is in our interest to make change like this backward
and forward compatible (two versions should be enough) that is if a jar gets upgraded eg. in NB6.5 and user later runs
NB6.1 it should still work; if that's not achievable and newer version is significantly incompatible than library name
should be changed so that two versions of the same library could coexist

2. it's needed for project - in this case user should be in charge; an "Upgrade" button could facilitate the process in
UI based on knowledge of a version of current sharable library and presence of a newer version in global libraries
catalogue; in some situations when a module knows that old version of a library is buggy it can suggest user upgrade
from project open hook(?)

3. needed for both - this is a variation of #2; user should be in charge but module can request/suggest/insist on an upgrade

Side question: if we enhance library infrastructure with a version should we allow module to provide one library in
multiple versions? Any usecase in which that would make sense?

Does it make sense? Did I miss something? I'm just brainstorming a solution.
Comment 17 Milan Kuchtiak 2008-10-02 11:10:15 UTC
I think, the plugin should be able to detect the version of library used in project, for individual use cases, and
notify user that the version is old.
User can upgrade or not. 

Note:In case of build script, we are able to customize Ant tasks regarding to the library version.
Comment 18 David Konecny 2008-10-03 03:07:44 UTC
I compared libraries from NB6.5Beta (all clusters build)(more changed possible since then) with libraries in NB6.1. I
compared that number of jars in a library has not changed and that file size of jars in a library has not changed. Later
I excluded J2ME libraries as they do not libraries sharability. Following is list of changed libraries which we should
properly upgrade in NB6.5 (I tried to group libs logically or by cluster):

- IdentityAntTasks

- restapi
- jaxws21
- restlib

- jsfsupport-runtime
- jsfsupport-designtime
- jsf12-support
- exceptionhandler-runtime
- jdbcsupport-designtime
- jdbcsupport-runtime

- spring-framework

- CopyLibs - this one is already handled

The list is alarmingly long!
Comment 19 Milos Kleint 2008-10-03 06:38:30 UTC
dkonecny: no, it's not long. And you included all that are changed. I'd like to reiterate that I don't think we should
automatically upgrade all libraries (eg. not sure the old version of spring framework will cause any trouble to the IDE
integration)

"Side question: if we enhance library infrastructure with a version should we allow module to provide one library in
multiple versions? Any usecase in which that would make sense?"
hardly ever going to happen, it would just increase the download size without any clear benefit.
Comment 20 David Konecny 2008-10-03 08:58:01 UTC
Re. "I'd like to reiterate that I don't think we should automatically upgrade all libraries" - agreed. What needs to be
done though is to talk to owners of changed libraries and made them aware of potential issues in sharable libraries
scenario and help them solve problem if them have one. And we should do it today and early next week so that they can
still fix it before code freeze. CopyLibs library problem I fixed is good example; IdentityAntTasks could be the same
case. Perhaps modules should just detect that older version is presented on project's classpath and tell user to upgrade
it if necessary.
Comment 21 Milan Kuchtiak 2008-10-03 09:21:17 UTC
< Perhaps modules should just detect that older version is presented on project's classpath and tell user to upgrade
it if necessary.

It's only possible when library is on project's classpath. In case of ant script - it doesn't need to be.

BTW. What are the best steps, for user, to upgrade library in shared libraries ?
     I guess first remove then import, right ? Another option is to replace library manually in shared library folder.
Comment 22 David Konecny 2008-10-06 02:13:39 UTC
Re. "What are the best steps" - see
http://hg.netbeans.org/main/diff/3b86a600922a/j2ee.common/src/org/netbeans/modules/j2ee/common/project/classpath/ClassPathSupport.java
for how I detect version in CopyLibs case and upgrade it. If there would be more time I would add to AntProjectHelper a
new method removeLibrary(Library). 

dongmeic, potingwu - could you look at all other libraries (jsfsupport-runtime, jsfsupport-designtime, jsf12-support,
exceptionhandler-runtime, jdbcsupport-designtime, jdbcsupport-runtime, spring-framework) and check if upgrade is
necessary or desired? We have last week before feature freeze so it needs to be fixed this week.
Comment 23 Milos Kleint 2008-10-06 07:10:03 UTC
well, since there already is a  removeLibrary() method in refHelper.getProjectLibraryManager().removeLibrary(lib)
I'd suggest to use a different name, eg. upgradeLibrary(Library) where the parameter will be library with the same name
but from global library manager.

anyone who copies David's code, please point to your changelog here, so that we can replace it with the api method
created for the netbeans.next release. thanks.
Comment 24 Antonin Nebuzelsky 2010-07-30 15:16:52 UTC
Reassigning to default owner.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo