This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 31663 - vcsgeneric has no redirect of filesystem mkdir
Summary: vcsgeneric has no redirect of filesystem mkdir
Status: STARTED
Alias: None
Product: obsolete
Classification: Unclassified
Component: vcsgeneric (show other bugs)
Version: -S1S-
Hardware: PC Solaris
: P3 blocker (vote)
Assignee: Martin Entlicher
URL:
Keywords:
Depends on: 34113
Blocks: 35330
  Show dependency tree
 
Reported: 2003-03-04 19:19 UTC by pswisnov
Modified: 2011-09-19 23:28 UTC (History)
4 users (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments
The contextual diff, that adds the necessary functionality to VCS attributes. (3.61 KB, patch)
2003-04-01 13:59 UTC, Martin Entlicher
Details | Diff
Here's the binary patch of VCS attributes. Put this into <NB-install>/modules/autoload/patches/org-netbeans-modules-vcscore/ folder. (10.03 KB, application/octet-stream)
2003-04-01 14:05 UTC, Martin Entlicher
Details
Here's a test profile (for UNIX), that I used to test the functionality. Put into <NB-user-dir>/system/vcs/config/ folder. (5.87 KB, text/plain)
2003-04-01 14:08 UTC, Martin Entlicher
Details
Diff for the WebApps module (5.51 KB, patch)
2003-04-01 14:24 UTC, Petr Jiricka
Details | Diff
Patched ClearCase profiles with MKDIR_CMD command. Extract in <NB-user-dir>/system/vcs folder. (5.78 KB, application/octet-stream)
2003-04-02 15:04 UTC, Martin Entlicher
Details
Binary patch for WebApps (24.29 KB, patch)
2003-04-02 15:16 UTC, Petr Jiricka
Details | Diff
Also the context patch for ClearCase profiles - the MKDIR_CMD command which is already added in the attached ClearCase profiles above. (2.33 KB, patch)
2003-04-03 15:23 UTC, Martin Entlicher
Details | Diff
New patch for WebApps. Input provided by Ann Suhnachawee. (8.39 KB, patch)
2003-04-04 16:48 UTC, Petr Jiricka
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description pswisnov 2003-03-04 19:19:44 UTC
In systems like ClearCase, directories containing
versioned objects must be created by a specific
command. In clearcase a directory cannot be
converted to one that contains versioned objects
after creation as a local directory. There seem to
be a number of approaches to dealing with this:

- remove directory creation from all module templates

- change the filesystem mkdir command in the
vcsgeneric module's filesystem (or is it in
vcscore?) to check if there is an override to
mkdir, and prompt the user if they want to create
the directory versioned or unversioned

- ?

I'm logging this as a placeholder here to make
sure we track the issue.
Comment 1 pswisnov 2003-03-22 00:20:34 UTC
I should amend this. This is only true of DYNAMIC views, but is
apparently not true for SNAPSHOT views, where directory creates can be
added to source control after the mkdir is executed.
Comment 2 Mikhail Romanov 2003-03-27 08:39:28 UTC
I've raised the prioroty of this bug because it blocks fixing of bug
#4830260 which must be fixed for Nevada.
Comment 3 Martin Entlicher 2003-03-27 08:56:33 UTC
I'm afraid this can not be fixed for NB 3.5 / S1S 5.0.
Because:
1) This is a feature, not a defect,
2) It's not clear to me how to fix this at all.
Comment 4 Petr Jiricka 2003-03-27 10:41:17 UTC
My impression is that it is not clear at all what the user 
view of this behavior should be. For example, do we know 
what should happen if the user simply creates a directory 
using New -> Folder ? How will that get added to 
ClearCase ?

I don't see how we could fix this issue by "removing 
directory creation from all module templates". For Web 
Module, if we don't create the WEB-INF directory, then we 
are not creating a web module. Also, most templates have 
the second panel ("Target Location") provided by the core. 
This panel has the capability to create new folders. Are 
you saying that this capability should be removed for all 
templates ?
Comment 5 Sergey Soldatov 2003-03-27 12:35:14 UTC
I think it should be clear for end user. Possible we just 
need to document it. If user uses ClearCase->mkdir command 
he want to create a folder in the Source Control. If he 
uses New->Folder than he want to create a private view 
directory. The problem is what exactly IDE should do if we 
convert the filesystem to a web module. Should it add all 
necessary directories like WEB-INF, lib, classes to Source 
Control ?. Possible we need an additional option for 
Convert to Web Module dialog which will check is it source 
control filesystem or not and give user the chance deside 
want he add all this stuff to Source Control or not. 
Moreover whithout this feature it's not possible to use 
WebModules with ClearCase's dynamic views. 
Comment 6 pswisnov 2003-03-27 15:31:10 UTC
Regarding removing directory creation...

I really only put this out to try to define the space of alternatives.
However my understanding of .war files is that while the .war file has
to have a web-inf dir, this does not necessitate that the development
source be formatted this way. I don't think the ant war task has this
requirement, for example.

As far as creating template dirs...yes, this will need to be trapped
or removed. Project natures in 4.0 will also have to route through
this mechanism (I dropped Pavel a note about this a while back).

We can't simply replace new->dir with cleartool dirs, since one also
needs to create versioned directories. How about having two commands
new->versioned dir and new->unversioned dir if we see that the
filesystem is versioned, or defer to the filesystem for these commands.

Also, I think its overstating things to assert that the web module is
unusable without this. By definition the web module only gets created
once in the lifetime of the web application. There is a workaround for
this -- recreate the directories by hand and move the web.xml file
into the tree. Its ugly, but it works.

As to whether this is a feature or a defect, I dunno. It seems like a
defect in the design to me...or is that an "unanticipated feature"?
Comment 7 ssffleming 2003-03-27 16:40:53 UTC
changing target to S1S 5
Comment 8 pswisnov 2003-03-27 17:59:51 UTC
As Martin has already stated, this cannot be fixed in S1S5 unless we
are prepared to slip again. It requires pervasive changes in the
filesystems API, vcscore, vcsgeneric, the clearcase profile and in all
modules that create directories.
Comment 9 Martin Entlicher 2003-03-28 08:32:30 UTC
According to http://www.netbeans.org/issues/notargetmilestone.html the
Target Milestone is set by the developer.
I'm reverting it back to TBD, because this problem is not yet
evaluated and it's not clear how to fix this. Because of this I can
not promise it will be fixed in S1S 5.0.

This problem is IMHO caused by a deficiency in ClearCase. And it's
very risky to change the architecture of NetBeans/S1S due to that
deficiency. It was found too late in the development cycle.
Comment 10 Martin Entlicher 2003-03-28 09:55:52 UTC
I've had a long brainstorming with Yarda about this problem.
We've come up with 3 different solutions:

1) Define a special command in ClearCase profile (e.g.
name="CREATE_FOLDER"), which will be called from
VcsFileSystem.createFolder(). This command will execute something like
'cleartool mkdir <folder name>'. The command must not show any dialog
to the user.

The consequences will be, that *all* folders, that will be created on
the filesystem will be automatically created in ClearCase. If the
command fails for whatever reason, the user will not be able to create
folders. The user can not create a folder, that will be only local
(not in ClearCase). It depends on the users workflow whether this is
acceptable or not.

2) Modify the ClearCase MKDIR command so that it can add existing
folders. Before running 'cleartool mkdir' move the original folder to
a folder with a different name. Then run 'cleartool mkdir', after it
finish, move the content of the renamed original folder to the newly
created folder. Then remove the empty renamed original folder.

The advantage is, that this can be implemented fully in the ClearCase
profile without any changes in the rest of modules.

3) VcsFileSystem.createFolder() will throw UserQuestionException or
some modification of it. There would have to be a mechanism which will
catch it and present to the user. The user will decide whether to
create the folder locally or in ClearCase.
UserQuestionException.confirmed() will perform the directory creation.

This is nice from the users point of view, but almost unimplementable.
It's not only hard, but almost impossible to find all places where the
UserQuestionException would have to be catched and presented to the
user.

IMHO the best solution it 2). It will workaround the ClearCase
deficiency and the rest of the functionality can stay untouched.

Peter, can you please write your opinion about these solutions?
Comment 11 Martin Balin 2003-03-28 10:23:09 UTC
In the QA we prefer #1 then #2. It will be good if Misha 
can express his preference as well.
Comment 12 Martin Entlicher 2003-03-28 15:53:11 UTC
If #1 is preferred because of the fact that it's automatic, then I
have not against it. If there is no problem that users will not able
to create local folders, it's O.K. from me. But now it looks like a
half solution to me, because:

After you convert a FS to web module, all created folders will be
automatically added to ClearCase, but files won't. web.xml will stay
as a local file. Is this acceptable? Why we do not introduce a fully
transparent mode instead?

So I'm proposing solution #4:
Add an option, that will turn the VCS FS into a "transparent" (or
whatever it will be called) mode. In this mode FS.createdData() will
be mapped to ADD command, FS.createFolder() to MKDIR command,
FS.delete() to REMOVE command and FS.rename() to RENAME command.
The transparency would still not be 100% however, mapping save action
to CHECK_IN is too dangerous and even not desired IMHO.
Every profile could specify whether it wants to be in the transparent
mode or not.
There might be also an option to switch the transparency mode on/off.
Comment 13 pswisnov 2003-03-28 15:55:41 UTC
Hmm...regardless of whether we think its a deficiency in clearcase its
an issue we will have to resolve: this is the way clearcase worked
back before there was a netbeans.

Solution one is not acceptable since there are cases where one wants
to create view-private directories.

I had thought at one time about moving the contents, but I'd was
concerned about editors with unwritten content in the directory. Are
there any modules which hold onto the file descriptor of the directory?
Comment 14 Martin Entlicher 2003-03-28 16:05:51 UTC
AFAIK there should nobody hold file descriptors for some long period
of time. But if there will be compilation going on, or Editor will be
saving it's content in the mean time, it can break.

It's possible to detect whether there are some modified files, but I'm
not sure it's possible to detect saving tasks or compilation.

But theoretically this could be managed by the VCS filesystem. It
knows when someone wants to read or write data. So it can just wait
till the moving process is over. And the moving process can wait till
all streams are read and written. This can be a little risky change,
but it should work.
Comment 15 Mikhail Romanov 2003-03-29 00:16:34 UTC
I think that we need some simple workaround for Nevada release. One of
possible chices is to use clearfsimport command, which adds private
directories to VOB (exactly what we need), but requires "root"
priveleges for this VOB. 

For the future releases we can probably use combination of #1st and
#2nd approaches, when user can specify whether she wants to add new
directory to the repository once it's created (recommended), or add it
later on (not recommended because it may require moving of the
directory and saving/closing of all unwritten files).
Comment 16 pswisnov 2003-03-29 01:30:06 UTC
Clearfsimport won't really work since it doesn't run in-place,
disregarding even that it requires root/vob admin access.

I would return to the question of why, among all modules, the web
module depends on creating a set of directories with special
semantics. I don't believe this is a requirement of the Javasoft
web-app specification: only that the war file produced has a
particular layout. If, instead, we had a metadata file separate from
the web.xml file that worked the way the jar packager does, I suspect
this whole issue could be avoided.
Comment 17 Petr Jiricka 2003-03-31 13:11:46 UTC
Regarding the proposed approach #2: I have concerns that 
this will not work well. For it to work, none of the files 
and directories in the subtree of the directory being 
renamed would have to be used. Also, there is one long-
running task in the IDE which locks files for a long time: 
the JarFileSystem. When a jar file is mounted as a JarFS, 
the filesystem holds the lock on that file for a long 
time. It is common for the Web Module to contain jar files 
in WEB-INF/lib, and these are mounted as JarFileSystems 
(see also issue 20384 for all the intricacies of this).

Comment 18 Petr Jiricka 2003-03-31 14:15:03 UTC
Regarding the web module behavior - I don't think this is 
the right place to discuss WM user view (or even suggest 
extensive changes for NB 3.5), but nevertheless:

The WebApps module creates WEB-INF, because it is the most 
sensible default (even though it is not strictly required 
in the source structure, as Peter points out). The DD is a 
part of the user's application source (unless we consider 
XDoclet, which is a technology that not many people use in 
the context of WebApps). Also it is better to put under 
VCS a standard-based file, rather than some metadata file 
with a proprietary format. Additionally, having a 
proprietary metadata file an a non-standard location would 
make the build process more complicated (which is 
unnecessary), and would disallow integration with command 
line tools.

Also, the web module case is analogous to the Java case: 
if you create a package and a Java file in this package 
using the New -> Java Class wizard, then directories will 
be created locally. Could one argue here that no 
directories should be created by this wizard ? What should 
the ClearCase behavior be in this case ?


BTW, it would be interesting to know what guidelines 
ClearCase has about handling existing directories in 
general - sure users must be running into the need to add 
an existing directory in other scenarios as well (import 
existing sources, large integration which creates 
directories but wants to remain atomic etc.)

Comment 19 Martin Entlicher 2003-03-31 15:19:51 UTC
Thanks Petr J. for your comments about web module and for mentioning
the problem with Jars (opened streams for a long time).

If the solution of this problem will be, that the user will have an
option to add already existing directories into ClearCase (it seems
like there is no other option if the transparent mode is not desired),
then the only way how to solve it is to move the existing directory.

It's good, that VCS support was built as a filesystem, because it can
completely hide this movement from the rest of the IDE till it's done.
So if this is done correctly, opened files in Editor will remain
opened and all I/O operations will be blocked till the move is done.
Only external tools (like external compiler) can have problems.

My initial idea was, that we'll wait for existing I/O operation to
finish before the move, but because of issue #20384 this is not
possible. So, no need to wait, we'll just close all opened streams and
open them again after the move is done. This is a sort of crazy
solution, but I don't think we have another option.

Therefore I'm proposing a more detailed version of solution #2:

- VcsFileSystem will remember all opened input and output streams,
- VcsFileSystem will return an implementation of FilterInputStream and
FilterOutputStream, which enables to block I/O operations and switch
the underlying streams. This will likely not have a noticeable effect
on performance.
After the user will run "ClearCase -> Add" on a directory (or whatever
the command will be called), the move operation will be done in
FileSystem.runAtomicAction(). This itself would not be enough, we must
assure, that I/O operations will be blocked. So all opened streams
will be closed (we'll remember the position of these streams), but our
FilterI/OStream will remain opened so that the clients will not notice
anything.
After cleartool mkdir and move back finish, for all streams that are
blocked we create RandomAccessFile objects, that permits to
efficiently skip all previously read/written bytes. We will then use
RandomAccessFile in FilterI/OStream to complete the I/O operations.
Then all streams are unblocked and AtomicAction is finished.

This is not an easy fix, but it should work. Considering that we're in
high resistance mode, the fix must be very well tested.

The only problem can be with external tools. In this case either the
move process will fail and the user will have to try it later (after
the external tool will finish), or the external tool will fail. If the
external tool is compilation, it can be easily run again.

We need to agree on the solution ASAP, so please write your comments
or other suggestions. It would be good if we have a solution
implemented by this Friday, but since the "move solution" was found to
be quite complicated it looks like it will take more than a week.
Comment 20 pswisnov 2003-03-31 16:45:56 UTC
Petr, you are right, the web module design should be discussed
separately. But this does generally suggest that in the context of 4.0
we should carefully consider the impact of VCS systems and their
discontents.

Martin -- I think you are right, this solution is crazy. 

So here is a thought. How about we add a ClearCase->Create Versioned
Web Module command? Its ugly, but it could work. We could create the
directories in clearcase, then call the stuff which creates the
web.xml and any other gubbins.

Nice thing is it requires no last minute architectural changes.
Comment 21 Martin Entlicher 2003-03-31 16:56:48 UTC
> ClearCase->Create Versioned Web Module command?

Yes, this should be easy and simple to implement. But are we sure,
that there are not other cases when the user will need to add existing
directories to ClearCase? Is it really only the one action "Convert
Filesystem into Web Module" that brings this problem?
Comment 22 pswisnov 2003-03-31 17:11:08 UTC
Well, there undoubtedly are other cases, but I wouldn't think they are
as visible as the web module or as painful to work around. Certainly
no others have come up from QA with anything like the same visibility.
Switching from 

New->X->{where? (make new dir Y), name, ...}

to 

CC->makedir Y
New->X->{where? Y, name, ...}

is not that painful. But creating the web module dirs and moving the
sources in manually is not very nice to the poor user.
Comment 23 Martin Entlicher 2003-04-01 12:36:56 UTC
O.K., so if we all agree, that we must solve mainly convertion of
ClearCase FS into Web module and in other cases the user will need to
create the directory in ClearCase in advance, we can describe the
solution in more detail:

We definitely need some cooperation of VcsFileSystem with the web
module, because web module will ask us for creation of the necessary
folders.
With Petr Jiricka we agreed, that the possible approach is to use
FileObject attributes. That way we will not introduce direct
dependency between the modules.

1) Web module will recognize ClearCase filesystem by
FS.getRoot.getAttribute("FS_DISPLAY_NAME"). The value is "ClearCase"
for ClearCase VCS filesystems. This will be a simple add-on to
VcsAttributes.
2) Web module will then ask the user whether they would like to create
the web module directory structure in ClearCase. If the answer is yes,
then:
3) Web module will call FS.getRoot.setAttribute("VCS_MKDIR_ACTION",
<directory-to-create>) for every directory it needs to create. This
action will map to the appropriate ClearCase command. This method will
block until the command finish, so that when setAttribute() is done,
the folder is created.
"VCS_MKDIR_ACTION" attribute must be implemented in VcsAttributes.
4) Web module will finish the FS conversion process.

Comments/opinions?
Comment 24 Martin Entlicher 2003-04-01 13:59:26 UTC
Created attachment 9626 [details]
The contextual diff, that adds the necessary functionality to VCS attributes.
Comment 25 Martin Entlicher 2003-04-01 14:05:44 UTC
Created attachment 9627 [details]
Here's the binary patch of VCS attributes. Put this into <NB-install>/modules/autoload/patches/org-netbeans-modules-vcscore/ folder.
Comment 26 Martin Entlicher 2003-04-01 14:08:25 UTC
Created attachment 9628 [details]
Here's a test profile (for UNIX), that I used to test the functionality. Put into <NB-user-dir>/system/vcs/config/ folder.
Comment 27 Martin Entlicher 2003-04-01 14:20:09 UTC
I've attached patches of VcsAttributes and a test profile which I've
used to test the functionality. Now it's necessary to add MKDIR_CMD
command to ClearCase profiles. This command should not present any
dialogs to the user.

I don't think we should commit these hacks into the main trunk, so
after verification of the attached patches we'll hopefully get the
permission to commit it directly into release35 branch.
Comment 28 Petr Jiricka 2003-04-01 14:24:18 UTC
Created attachment 9629 [details]
Diff for the WebApps module
Comment 29 pswisnov 2003-04-01 22:14:40 UTC
the clearcase mkdir command value is provided in the profile under the
MKDIR command. I can add it as a variable, but given the time
difference if you want to test this tomorrow you may not want to wait
for me to submit, since I would need Martin to approve the integration
notice anyway.
Comment 30 Martin Entlicher 2003-04-02 15:04:52 UTC
Created attachment 9661 [details]
Patched ClearCase profiles with MKDIR_CMD command. Extract in <NB-user-dir>/system/vcs folder.
Comment 31 Martin Entlicher 2003-04-02 15:09:10 UTC
I've attached patched ClearCase profiles with MKDIR_CMD commands. I
did not test the commands yet.
Petr J. would you please attach the binary patch of the web module so
that we can test all 3 parts before the integration?

You can easily create the patch by:

cd web/core/src

jar cvf Patch31663_web.jar
 org/netbeans/modules/web/context/Bundle.properties
 org/netbeans/modules/web/context/WebContextObject*.class


Comment 32 Petr Jiricka 2003-04-02 15:16:00 UTC
Created attachment 9662 [details]
Binary patch for WebApps
Comment 33 Martin Entlicher 2003-04-03 15:23:59 UTC
Created attachment 9683 [details]
Also the context patch for ClearCase profiles - the MKDIR_CMD command which is already added in the attached ClearCase profiles above.
Comment 34 David Kaspar 2003-04-04 16:37:00 UTC
Verified binary patches in S1S5
Comment 35 Petr Jiricka 2003-04-04 16:48:35 UTC
Created attachment 9714 [details]
New patch for WebApps. Input provided by Ann Suhnachawee.
Comment 36 Martin Entlicher 2003-04-04 16:58:30 UTC
The web module code changes reviewed without objections.
Comment 37 _ ttran 2003-04-04 17:08:00 UTC
approved for 3.5
Comment 38 Martin Entlicher 2003-04-04 17:28:26 UTC
The VCS part of the fix committed into release35 branch:

/cvs/vcscore/src/org/netbeans/modules/vcscore/VcsAttributes.java,v 
<--  VcsAttributes.java
new revision: 1.22.36.4; previous revision: 1.22.36.3

Checking in ClearCase_unix.xml;
/shared/data/helm/cvs/repository/vcsgeneric/profiles/clearcase/src/org/netbeans/modules/vcs/profiles/clearcase/config/ClearCase_unix.xml,v 
<--  ClearCase_unix.xml
new revision: 1.10.2.17; previous revision: 1.10.2.16
done
Checking in ClearCase_win32.xml;
/shared/data/helm/cvs/repository/vcsgeneric/profiles/clearcase/src/org/netbeans/modules/vcs/profiles/clearcase/config/ClearCase_win32.xml,v 
<--  ClearCase_win32.xml
new revision: 1.10.2.17; previous revision: 1.10.2.16
done

Petr J.: please resolve this issue as FIXED after you commit the
appropriate changes in web module. Thanks.
Comment 39 pswisnov 2003-04-04 22:55:52 UTC
I believe there is a specific bug report regarding web module creation
that got bugbridged over. Can I suggest we mark that one fixed by this
patch instead of this, since I think we want to leave open a report
that tracks the general issue going into 4.0.
Comment 40 Petr Jiricka 2003-04-05 16:03:43 UTC
Integrated the WebApps part:

Checking in Bundle.properties;
/cvs/web/core/src/org/netbeans/modules/web/context/Bundle.p
roperties,v  <--  Bundle.properties
new revision: 1.9.26.4; previous revision: 1.9.26.3
done
Checking in WebContextObject.java;
/cvs/web/core/src/org/netbeans/modules/web/context/WebConte
xtObject.java,v  <--  WebContextObject.java
new revision: 1.27.2.5; previous revision: 1.27.2.4
done

Peter, which bug do you have in mind that should be marked 
as fixed ? I am not aware of any.
Comment 41 Martin Entlicher 2003-04-07 10:36:55 UTC
Peter, you're right. This was submitted as a generic problem with
directory creation. However only one specific (the most visible)
problem was fixed. So I've submitted P1 defect #32671 for that
specific problem, that was fixed in S1S 5.0.

I'm changing Target Milestone of this issue to NB 4.0, where this
problem should be fully fixed in a generic way.

Comment 42 Martin Entlicher 2003-06-02 13:41:09 UTC
We need a generic solution of this problem in NB 4.0.
Thus there needs to be a way to determine whether the files and
folders created on the VCS filesystem should be automatically put into
the version control system or not.

I propose to introduce a customizable "Transparent Mode" for that
purpose. As the first step, the transparent mode can be made for
add/remove of files and directories into/from VCS (remove only if
possible by the given version control system).
The property can have values "On/Off".

I thought about 3rd value "Ask", but this is not possible to implement
with current filesystems API. The reason is, that it's risky to show
dialogs in createData(), createFolder(), delete() methods and when a
lot of files/folders will be generated/deleted by some tool, it would
present one dialog per file, which is not acceptable.

If it's necessary to have the option to ask the user or make the
decission by the caller, either the filesystems API would have to be
changed or VCS APIs would have to be called.

Comments?
Comment 43 Martin Entlicher 2003-06-03 09:57:15 UTC
I don't think this problem can be solved well only in VCS modules.
Adding dependency on issue #34113.
Comment 44 Martin Entlicher 2003-08-01 16:07:43 UTC
I've started to study in more detail how Subversion works, and I've
found, that it has one very nice concept, that would help to resolve
not only this, but all changes that happens due to e.g. refactoring
and other automatic tools can be handled that way.

What it is?
In Subversion you can do a lot of things locally, without touching the
server. Similar to CVS, you can add and remove files locally. But in
Subversion you can do a lot more. You can add/remove folders and copy
and move files locally. This is exactly what we need here.
After all local changes are done, the user can review them, make some
adjustment, and after the user is sure that these changes are correct,
commit will send it to the repository.

Unfortunately, other version control systems do not work like this.
But IMHO this is the approach we need to follow. If the version
control does not have appropriate scheduling commands, it's necessary
to remember which changes were done, and offer them to the user at
commit time. Artificial "scheduling" commands can be implemented for
that purpose. This is something, that was already implemented for one
special case in issue #11589.

IMHO this approach can work with all version control systems we
currently support, but ClearCase dynamic views :-((( There will be a
big problem with directories. This problem can be ovecomed only by
proposed solution on "2003-03-31 07:19 PST" AFAIK. If correctly
implemented, it should work in most cases.

Comments? Other ideas? We need to find a solution soon if this is to
be implemented in NetBeans 4.0.
Comment 45 pswisnov 2003-08-01 17:44:19 UTC
Hmm...it seems to me that you are suggesting that we build a new
virtual logged filesystem that can commit a nested transaction log up
into an outer VCS transaction. This seems hard to me.

I think all the bug asked for was an API at the filesystem level that
modules could call if they knew a directory being created was
"important" in the same sense as a data object, or pass on the
decision to the user. Then the onus is on the module implementors to
incrementally port there code to call this API instead of the normal
mkdir API or to add things to their dialog boxes to ask the user if
this new directory should be able to contain versioned elements. The
filesystem type would then have some abstraction overridden by the
vcsgeneric implementation, which would then delegate it to the profile
"MKDIR_VERSIONED" command (for example). On clearcase this would
expand to "cleartool mkdir <dirname>"

If the module doesn't automatically create the thing versioned and it
was supposed to contain versioned elements, then its a bug.

The rational behind having this done at the module level is that often
the module knows pretty well whether a dir will contained versioned
stuff or is just a temporary dir. If there was a filesystem-level
setting we would end up creating versioned dirs for temporary files.
Comment 46 Martin Entlicher 2003-08-01 18:15:19 UTC
Well, even if this issue really reduces to the mkdir problem, then I'm
still not convinced, that the module that creates the directory can
always know whether it's important for the user or not. But if the
module implementator was kind enough to care about this problem, we
can have following solution:

1) A new method to FileObject will be added:
   boolean FileObject.isImportanceAware()
   It would return true if there is some behavior dependent on whether
   the FileObject is important or not (e.g. VCS actions)
2) Instead of simple call to FileObject.createFolder(String name), the
   module writer will check the value of
   FileObject.isImportanceAware() and either presented a dialog, or
   by some other means decided whether the created directory is
   "important" or not.
3) A new method to FileObject will be added:
   FileObject.createFolder(String name, boolean important); The module
   writer will call that method with the appropriate "important" flag.

VcsFileSystem can easily delegate the call of
createFolder(String name, true) to "MKDIR_VERSIONED" command.

Is this roughly what you had on mind? This can work, but all the
responsibility would be on module writers to use this approach. And
since old modules are still using the old APIs, there could be still
cases where the user will end up with local folders.

However I agree, that this is much more simpler to implement (on VCS
side) that the solution with logging scheduled operations.
Comment 47 pswisnov 2003-08-01 19:01:03 UTC
This is more or less what I had in mind. Yes, modules would not always
know whether to create the dir versioned or not, but they are the one
interacting with the user when the directory is being created: its not
practical to have the filesystem layer do this directly (e.g. have it
pop up a question dialog on top of the module UI). This would be chaotic.

We might also be able to be clever about prompting during file
creation in "important" directories to put a check box in the first
pane of the template creation wizard to add to VCS immediately. This
way we don't have to do it in multiple steps.

We might also need an abstraction for "is ready to add files to
directory".
Comment 48 Petr Jiricka 2003-08-04 08:29:35 UTC
Having a piece of UI in every wizard which would ask the user whether
new directories should be added to VCS would solve the problem, I agree. 

On the other hand, it is not necessary to have such a UI in cases that
the directory can be added after they've been created, e.g. in CVS or
ClearCase SNAPSHOT views. Also when the file is not versioned at all,
this UI would not make sense. So the UI would really need to be
included conditionally, only if the directory created is on a
ClearCase DYNAMIC view filesystem. This would create additional
complexity for modules, as well as possibly for Jemmy/Jelly tests.

Given all this, shouldn't we go back and ask ourselves whether we
really need to support ClearCase dynamic views? Could't we just say
that we don't support them very well?

What do other IDEs do about this? Would this scenario work in IDEA,
Eclipse, JBuilder or JDeveloper? Especially Eclipse is interesting,
given that there is a single vendor behind both Eclipse and ClearCase.
 
Comment 49 pswisnov 2003-08-04 16:18:26 UTC
This is a product management question. My intuition is that one has to
support dynamic views well to meet customer expectations about
completeness.

Moreover I think having modules prompt users in this way is a good
thing in and of itself. Module authors know in advance which
directories contain temporaries and which are likely to need to be
versioned. Guiding the user to make good decisions at the time they
are setting up there stuff seems better overall to me from a user
point of view.
Comment 50 Martin Entlicher 2003-08-08 17:28:31 UTC
I've installed Borland JBuilder9 Trial on Solaris and tried to use it
with ClearCase dynamic view.

I was able to place a project (the default Welcome project) into
ClerCase. It imported all files and directories into ClearCase and did
not prompt me for any message (it used "Imported by JBuilder"
comment).

When I created a package, the package was created just locally, it was
not added into ClearCase.
The pop-up menu with ClearCase commands have all commands enabled
(even those which do not apply), even though it runs "cleartool ls"
just *before* it shows the menu! Responsiveness of the menu is bad.

I've created a file in that package and tried to add it into
ClearCase. Unfortunately JBuilder froze me several times (it seems,
that JBuilder executes commands in AWT thread!). But after several
restarts I was able to add the file! (it did add the folder by
"cleartool mkdir -nc test").

Thus JBuilder is using similar approach that I've suggested here on
"2003-03-31 07:19 PST". It moves the files and folders into a
temporary location, creates the necessary folders through ClearCase
and then it moves them back.

I was adding:
./test1/test11
./test1/test11/ATest1Test11Class.java

JBuilder created folder ./test1.preCC:
./test1.preCC/test11
./test1.preCC/test11/ATest1Test11Class.java

This is an intermediate state:
test1@@/main/CHECKEDOUT from /main/0                Rule: CHECKEDOUT
test1.preCC
./test1/test11@@/main/CHECKEDOUT from /main/0       Rule: CHECKEDOUT
./test1/test11/ATest1Test11Class.java
./test1.preCC/test11

And final state:
test1@@/main/1                                     Rule: /main/LATEST
./test1/test11@@/main/2                            Rule: /main/LATEST
./test1/test11/ATest1Test11Class.java@@/main/1     Rule: /main/LATEST


IMHO this is the best solution. IMHO the requirement on other modules
to add dialogs and some special handling around every piece of code,
that creates folders is too intrusive. This in fact says, that
FileObject.createFolder() is deprecated and a different API needs to
be used. I don't think, that this will be ever be accepted by openide
team (but I may be wrong of course).
Comment 51 pswisnov 2003-08-08 17:53:23 UTC
Hmm...well, is the approach of having templates ask in advance whether
to create versioned directories too intrusive to us as netbeans/studio
developers or too intrusive to the user? The question to my mind is
what is the user experience we want to achieve. How many places in 4.0
are directories that hold user artifacts created in the first place now?

One reason we rejected this earlier was out of concern that this would
confuse the state of the editors if files got moved out from under
them, and that 3.5/Nevada did not provide lead time to address these
issues. If we were going to do this I think we would need to talk to
people about the file object implications. If I add a directory
containing some unsaved work I'd be unhappy if the editor lost the work.
Comment 52 Martin Entlicher 2003-08-08 18:59:34 UTC
> Hmm...well, is the approach of having templates ask in advance
> whether to create versioned directories too intrusive to us as
> netbeans/studio developers or too intrusive to the user?

Well, potentially to both ;-) But I'm worried more about
netbeans/studio developers. It's impossible to keep track of this
"ask" implementation long-term unless we deprecate
FileObject.createFolder() and replace it with a different
"version-control-like" API.

Mainly I don't like the fact, that a problem in one module (VCS),
which is relatively independent on others, brings on a necessity of a
bunch of changes in other unrelated modules. It's hard to assure, that
the changes will be applied everywhere and that this approach will be
kept in the future.

> The question to my mind is what is the user experience we want to
> achieve. How many places in 4.0 are directories that hold user
> artifacts created in the first place now?

I have no idea.
I've made a simple grep to NetBeans sources (just standard modules
only). There're many createFolder() calls on DefaultFileSystem, which
does not have to be counted. Then some calls in core, java, jini, mdr,
openide, web - which looks like real creation of folders on other
filesystems. There're not so many, but there are potentially others in
S1S source code and other NetBeans modules.
And there are also 3rd-party modules which are out of our scope.

> One reason we rejected this earlier was out of concern that this
> would confuse the state of the editors if files got moved out from
> under them, and that 3.5/Nevada did not provide lead time to address
> these issues. If we were going to do this I think we would need to
> talk to people about the file object implications. If I add a
> directory containing some unsaved work I'd be unhappy if the editor
> lost the work.

Well, it would not be trivial, but this is possible to safely
implement this with respect to internal processes, that access
FileObjects. Since we have the filesystem, we can control the streams.
We can interrupt the I/O processes and link them again after the move
is done. It would not be easy, but it is possible. I'm more concerned
about external processes (like external compilation), that may be in
the middle of reading of some file.
But in this case, on Windows the move will fail. There will be sharing
violation (I've verified this). This is good, because nothing will get
broken. On UNIX the situation can be worse, however it seems, that at
least Solaris handles this case gracefully:

One terminal:                    Second terminal:
mkdir /tmp/FolderToMove;         cd /tmp/FolderToMove
                                 vi Test.txt
                                 write some text and save
                                 write some more text
mv /tmp/FolderToMove /tmp/FolderToMove.cc
                                 save changes to Test.txt (succeeded!)
mv /tmp/FolderToMove.cc /tmp/FolderToMove
cat /tmp/FolderToMove/Test.txt   cat /tmp/FolderToMove/Test.txt

The changes were correctly written even though the folder was moved in
between. I'm not a UNIX filesystem expert, so I'm not sure whether
this is normal, or it's only a lucky test, that succeeded thanks to
some kind of caching.

This experience suggest, that the "move" approach may work.
And even though this approach is harder to implement, IMHO it's better
to have this localized in one place, so that we do not require changes
in other modules.
Comment 53 Martin Entlicher 2003-08-08 19:16:02 UTC
BTW: it's a pity, that ClearCase does not implement this "move"
approach itself in "cleartool mkdir" in case the directory exists. It
has the proprietary filesystem implementation, thus it can control
everything. IMHO it should be possible to correctly handle the I/O
streams during the move when one implements the filesystem, isn't it?
Comment 54 pswisnov 2003-08-08 19:30:53 UTC
Well, I take it as axiomatic that the user experience should drive the
implementation, and not the other way around. What other tools do is
an interesting data point, but it needs to be qualified: for example,
what if we discovered that many users of this other tool hate the
clearcase integration?

My feeling is that many developers never do anything without starting
with VCS, even if they are prototyping. The user profile of S1S and NB
might differ, but I think this is a pretty commonly held idea about
best practices. Are we thinking of VCS as an add-on because thats how
we implemented it? Would we end up with a better IDE if we built an
assumption into the IDE that people will use VCS in all their
projects, and that not using VCS is the exception?
Comment 55 Martin Entlicher 2003-08-08 19:50:13 UTC
Well, you're absolutely right, that we need to have a good user model.
Since both approaches are hard to implement, we need to be sure which
one of them is the correct one (which will provide the best user
experience). In this context I have more CVS-like (or maybe also
Subversion-like) style of thinking - prepare all changes locally first
and then decide what to actually put into the version control system.
But this may not be how ClearCase users work. We should follow the
users expecations. A problem can be if this requires a change in the
core architecture, but we need to have a good usability, of course.

Can we know what the users really expect? Which approach suits better
to ClearCase-style of work? Do you think, that it's possible to get
some feedback on the JBuilder implementation (e.g. from mailing
lists)?

You're right, that some kind of VCS tracking is necessary for
development of every serious application. This would be a good
argument for including VCS support into the core APIs (openide).
Comment 56 Jiri Kovalsky 2003-08-14 12:42:35 UTC
Excellent guys. I was always against *only* Edit/Compile/Debug cycle.
If you mean your work seriously, you must follow
Edit/Compile/Debug/Versioning approach - exactly as Peter pointed out !
Comment 57 Martin Entlicher 2004-05-03 17:54:18 UTC
Scheduling for future.