Btw. another possible enhancement would be to ask user whether to
disable to module, not start at all or "pretend that the module is of
newer version". Last is dangerous but in our current situation would
work...

I don't see any reason to prevent a user from using an older version
of a module! There may be very good reasons for a user to want to do
that. Issue 19001 just demonstrates that
SharedClassObject.findObject(Settings.class,true) is safer than
Lookup.gD().lookup(Settings.class). As far as I can see the core is
behaving correctly here, there was a bug in the Java module's code.
What is the justification for this?

Ok Jesse, here are the consequences:

File UI/Services/java.settings has been changed in newer version as
such it is not recognized by InstanceDataObject in old version. As
such it does not have InstanceCookie nor it has properties that would
allow user to modify it. (There is an obscure way how to get the
original option - tools/options/thesetting + expand layer info and
move the setting from info to the most right column -> this should
delete the settings from the newer version).

Anyway from above I conclude that it is not good to run the IDE with
downgraded version of a module.

I don't draw that conclusion at all. The problem is not with running
an older module as such, it is that the structure of settings requires
that when a .settings file is presented in the layer, the user have an
opportunity to modify it. Apparently InstanceDataObject is not
currently doing this usefully. If there are stored settings with a
newer module version than is currently installed, rather than skipping
the instance cookie altogether and just showing "Broken Settings", it
should IMHO do either of the following:

1. As now, but with an obvious way in the UI to discard the unusable
state and reset it to the older module version's pristine state (i.e.
delete from system folder). Perhaps a menu item on the IDO, or just a
less cryptic UI for the Settings dialog.

2. In this situation, provide an IDO consisting of the pristine state
immediately, and permit it to be edited (clobbering the unusable newer
settings).

I somewhat prefer #1 I guess. Note that the IDO does not really know
where it is coming from (what layer etc.) so in fact the IDO itself
cannot reasonably handle this. However the Settings dialog could, I
think. There may need to be some way to communicate from a DataObject
to the dialog that it wants to be reverted somehow, i.e. that its
current state is bogus and that it ought to be restored to a pristine
editable state if that is possible, which would permit the dialog to
e.g. prompt the user to discard unusable settings in a batch, or
something like this. One easy mechanism would be to retain our current
system of providing a dummy InstanceCookie when there are broken
settings, but to standardize what the broken settings instance name is
- then the settings dialog could look for instances claiming to be
broken and attempt to revert them or otherwise warn/guide the user.

As with issue 19001, it ought never be the case that the module does
not work when some of its settings are broken. The module should
always have some backup scheme: it should not assume that objects
provided in its layer which are placed in user-editable areas
(settings, ....) will be in any particular state. Thus it should use
SharedClassObject.findObject for options, recover gracefully when
there is no applicable executor available from lookup, and so on. Such
robustness gives the user a chance to correct the settings without
getting exceptions all over the place first.

Jesse I would agree with (1.). It is already possible to reset broken
settings by deleting its node and UI could be improved. But what
settings which are e.g. hidden (not all are presented in Options
window)? Moreover  it requires user to find out all these broken
settings and reset them.

1. For settings which are hidden and not presented in Options, how
would they have gotten set at all? I can only think of a couple of
examples, like AutoUpdate's "last connected to server" etc. Probably
the module writing out such settings needs to decide whether it wants
to clobber old data or not. Better yet, it should not use versioning
stamps on such settings to begin with. Needs more thought.

2. Finding all the broken settings - yes this is probably a needed UI
improvement. See my suggestion about a dummy InstanceCookie. Or
something like this.

I am not sure if it should be _possible_ to downgrade, but it is IMHO it is unacceptable to let the user 
downgrade modules and just silently ignore all their settings and let it up to the user to find it out, 
go to Options and revert them to defaults. I gues there are multiple options which can be done:
1. disallow the downgrade
2. inform the user the they are using an older version of module and may have problems and
 a) disable the module
 b) revert settings to defaults
 c) ignore this fact and try to read new settings with old module - it may work, espacially in the case 
that it is similar version (like a beta and main trunk at the same time)
 - we can choose some of these options or let the user to choose

I guess 2b. sounds most attractive (tho a confirmation from the user
is needed first to avoid possible data loss - if rejected, the module
should continue running but stored settings will not be available and
it may not be possible to make changes to existing settings).

Unfortunately I don't see how to do it. There is no trivial way to
find all customized settings originating from a given module. You can
look for settings files matching those in the layer, but this will not
cover newly-created services and so on. It might be possible to:

1. Store all customized settings in a special system folder named
according to the originating module (TBD how to tell which module this
is). All these would then be merged together by SessionManager. Might
be desirable in its own right anyway: cleaner management of different
modules.

2. Keep a file attr. associated with layer-installed settings
indicating its originating module and perhaps version (the module name
would be needed for pnejedly's SystemFileSystem.localizingBundle idea
anyway). Let it be copied to new files created in the SFS by changing
contents of the layer file (e.g. options) or by copying/creating from
template (e.g. services). Then later on if you want to revert all
customizations from a module en masse, you can do so by just searching
for this file attribute. It might also be used, if this were helpful,
to perform the logic currently handled case-by-case in
InstanceDataObject and window system XML files to ignore files from
uninstalled modules.

3. Some sort of combination of 1. and 2.: a simple attr provided for
every layer file which can be used for various purposes, and
SystemFileSystem when creating a writable layer for a file switches to
some subdirectory according to this info.

Of course these suggestions are significant architectural changes, but
I don't think any less so than deciding "users can never run an older
version of a module again" which does not sound too popular.

Probably the thread should be transferred to dev@openide, I have not
heard any suggestion which is trivial to do and obviously a good solution.

Pavel, probably you meant to be CC'd here...

Jesse, your suggestion to store module settings in special module
directory is good, and I would like to suggest it to module writers.
Will we suggest - /system/Data/org.netbeans.modules.java/... this
could clean the mass that is on SFS...

Anyway we need something simpler right now.

The simplest is likely to modify InstanceDataObject load downgraded
settings anyway and if it fails present a reasonable description -
"You have tried to load settings from stored by a newer version of
module XYZ (2.1.43) but it does not seem to be readable with your
current version (2.0.3), please upgrade the module."

Second simplest and IMHO nicer solution is to modify the module system
to at least notice that the system runs older version of the warn the
user. But I can see that it is not Jesse's favourite.

1. Yes we need something simpler than per-module dirs for the short term.

2. Per-module settings cannot be left up to module authors. The system
expects certain SFS folders to be used for certain purposes. The
merging has to be done by the session manager.

3. A warning from IDO might work, though this would be ugly in non-GUI
mode I think, and could be very annoying if there are dozens of such
dialogs.

4. A warning from the module system that an older version of the
module is being run and that some settings may not be available would
be OK. (I mostly object to *preventing* downgrades.) It does not help
the user much, however, if the user wishes to really use this version
of the module, and wishes to discard the old settings. So it would
only be a partial solution.

Jan Chalupa made a suggestion in off-line discussion, let me describe it. I know that it may not be 
possible in current implementation, but I think that it should first be clear how settings updates 
*should* behave, then we can look for impl. It questions the statement in the beginning of discussion 
that we cannot support future compatibility.

1. In number of cases (I did not check the percentage) there is actually no change in the module 
settings. At least in the content :-)
2. In addition there are many cases when settings are just extended by newer version and if this 
extension would be ignored then settings could be used by an older version.

At least in these cases the settings _could_ be fw compatible. For both cases we do not make it possible 
to downgrade module and use settings because we do not know it. We are using the spec. version of module 
as a version of settings, which means that with every release we assume that settings have changed (case 
1). This could be changed by introducing a separate spec. version for settings which does not need to 
change if the settings format did not change.
The second case would require the format of settings to be specified and to make it possible to keep 
ignored data untouched.

opinions?

For Pavel's #1 - yes, this is surely true, but the IDE should have to
be told by the module that its format has not changed, as currently
the expectation is that the format may change without notice. I.e.
default should be to assume it is not compatible.

For #2 - I am against introducing a special spec version for settings,
it seems ugly. However if data were just stored in a different format
to begin with, e.g. XML files with versioned DTDs, it would be up to
modules to handle their own upgrading and downgrading, and the core
would not have to deal with it. More work for module authors, but
would probably result in better reliability in the long run, because
it is clear that the core knows nothing of your format, so you do not
expect too much and have to think about what will happen. We would
have to compensate by providing a usable API to help modules manage
the timing: when to load settings, when to save, when to revert, etc.

You say that module should be able to tell that settings have not changed. That is my point. Whether it 
is by a separate spec. No. or in other way (how then?, by version of DTD?) does not matter that much from 
my point to view.

Ok, so we can change the issue as wontfix and concentrate on issue
17924 - custom format for IDO...

May I ask a question? If I understood it well the whole problem is 
caused by spec version of the module which is stored in .settings 
files. But why is the spec version stored in .settings files at all? I 
don't understand its purpose.

Let me summarize my understanding how .settings file works and correct 
me please if I misunderstood something:

There are two types of versioning. First, the structure of the 
.settings file itself. This is handled by used DTD. Second, versioning 
of the module's settings stored in the .settings file. This is 
module's task - module must take care about this versioning and about 
backward compatibility.

Storing module's settings in .settings file in XML form is better than 
serialized form, but even the serialized form should not cause so much 
troubles. It is possible I'm missing something as I'm not expert in 
serialization. But if module:
* uses the same serialVersionUID
* and did not remove the class
* and did not change the semantic of the properties of serialized 
class
then they should be able to deserialize newer and older versions of 
the class. If module deserializes the old version of the class, all 
new fields must be initialized by module in readObject() method. If 
the module deserializes some newer version of the class, all new 
properties are ignored (right?). (if the settings are stored in some 
human readable format, the situation is almost the same - module 
should be able to recover from the missing/new fields)

So, where is the problem? Why we need to have spec version in 
.settings files. Something like spec version can be handy, but only in 
case that module did some really big incompatible changes in almost 
all of its settings. Otherwise they should use the mechanism above and 
recover from minor changes.

The purpose of the spec. version in .settings files is to allow the 
settings framework to ignore settings from disabled/missing modules 
otherwise CNF would be still thrown.

Jan, but for this you need just name of the module and not the spec 
version.

I do not think so. If a setting is introduced in the newer version of 
a module and you would downgrade to the older one just the module name 
is not sufficient.

Spec version in settings file can also be used to indicate that a
setting is not reverse-compatible. To David K.: yes unused fields and
missing fields are handled somehow by Java Serialization, but the
result may be an object with inconsistent state. E.g. you had:

public class Foo implements Serializable {
    private static final long serialVersionUID = 123L;
    private String thingy;
    // ...
}

and you change it to:

public class Foo implements Serializable {
    private static final long serialVersionUID = 123L;
    private String[] thingies;
    // ...
    private void readObject(...) ... {
        // getFields, look for "thingy" and for "thingies",
        // set thingies instance var appropriately
    }
}

This works fine. But the new state will not work in the old version of
the class, and it is too late to make it work, probably.

If you are careful in your module, and design all serializable classes
to be bidirectionally compatible for the future (using a map of
properties and so on), and design some system for handling problems
gracefully in both directions, great. Remove the specification version
from your .settings file in the layer, and IDO should let the user
downgrade and keep your settings. Just as if you were storing data in
XML files using some custom versioned DTD and wanted to manage
upgrades/downgrades yourself.

to: Yarda (this is really becoming a discussion forum:-)
please leave this issues open so that we make sure that the fix of 
http://openide.netbeans.org/issues/show_bug.cgi?id=17924 also fully resolves this issues, then close it.

Yes, Jesse, you are right. The scenario you have described will fail 
even in case we have nicely human readable XML. And in this case the 
increased spec version would make perfect sense. But shouldn't it be 
then the module owner who should decide when it is necessary to change 
the "version" of the setting? Because I tend to believe than in most 
of the cases the settings are compatible. The module owners just must 
be aware of the compatibility issue and design the changes according 
to it. Marking the settings automatically as incompatible after each 
release just causes a lot of problems. (I think this is what Pavel and 
Honza proposed - to have separate "spec version" for settings)

IMO, the problem with the current approach is that the decision about
compatibility is being made at the core level, although it should be
made at the module level. The spec version number in the settings file
is just a hint to tell core that things have probably changed, but
there's not enough information available to deal with potential
incompatibilities at that level.

I think that the format of the settings and dealing with compatibility
issues should be left up to the module developer. The settings format
(preferably human-readable) should be carefully specified, versioned
and treated as an API. There have to be good reasons to change the
format and the module developer should take responsibility for
handling all issues resulting from that change. The module should be
able to read all older versions of its settings. (In a perfect world,
it should also be able to save in older formats :-).

Last but not least, I think that we would be facing far less
compatibility problems if the module developers were forced to think
about the data they really need to persist in the first place.

David K. - no one is forcing you to mark settings as incompatible! If
you put a spec version in your *.settings, it will be used. If you
don't, it is assumed that any version of your module can read that file.

Jan Ch. - yes there is a need for better formats and more module
involvement in the process, we're trying to come up with something;
but I think it is out of scope for this issue, which should
concentrate on properly handling downgrades of settings as we have
them now, i.e. plain serialized SystemOption's, ServiceType's, etc.

I think this is another issue that needs to be moved to dev@openide
and probably marked RESOLVED LATER since there is no single option
settled on here.

Jesse, I did not know that module can say whether they want spec 
version in settings file or not. This is good. But still, once they 
have put it there (e.g. in version 2.0 of my module I did some 
incompatible changes in settings), it will be automatically updated 
according to module spec version, right? So although I did not change 
the settings and they are still backward and forward compatible for 
all >=2.0 version, the spec version is automatically increased for all 
2.1, 2.2, 3.0 releases and that causes problem reported in this issue.

*** Issue 19995 has been marked as a duplicate of this issue. ***

The implementation is based on the decision that each major release 
(3.3, 3.4, ...) introduces own user dir. So, it is possible to safely 
downgrade in the scope of that release (e.g. 3.4.2 -> 3.4.1).

The current settings implementation handles the setting versions in 
following manners:

old serialdata format:
significant is just the code name of the module. Remaining attributes 
like the spec. version or the release version are ignored.

custom convertor format:
convertors do not use the ModuleInfo any more. The setting's 
availability is given by its dtd's public identifier registration in a 
module layer. The content of .settings file is not dependent on a 
setting object class either. 

Now, It is module writers job to provide proper registrations using 
convertors and classes suitable for the given version.

For more details see the core/settings module.

Ok, different directories will work.