General module description:

Module name: Remote File System Bridge.
Module location: fsbridge in ide cluster.
Module depends from masterfs (as friend), dlight.nativeexecution, filesystems.
Modules versioning and dlight.remote.impl depend from fsbridge as friends.

Module API provides:
- Class FileProxy is java.io.File and FileObject independent file abstraction.
- Interface ProvidedExtensions2 is extension of master file system SPI.
- Class Bridge provides conversion between java.io.File, FileProxy and FileObject.
- Class Bridge provides access to remote execution environment.
API package name: org.netbeans.modules.fsbridge.spi

Created attachment 109178 [details]
initial module version

[JG01] "org.netbeans.modules.fsbridge" is not a very descriptive CNB.


[JG02] Would anyway seem cleaner to me to just replace usages of File with URL in the existing org.netbeans.modules.masterfs.providers.*. Would of course be an incompatible change (-> /3) but there are just a few friends to update. NB APIs generally use URL (or URI) when we want to express a (possibly) virtual file path, where FileObject is unsuitable because it can only refer to an existing file.

Created attachment 109632 [details]
second iteration

(In reply to comment #3)
> [JG01] "org.netbeans.modules.fsbridge" is not a very descriptive CNB.
- Module renamed to "org.netbeans.modules.remotefs.bridge"

> [JG02] Would anyway seem cleaner to me to just replace usages of File with URL
> in the existing org.netbeans.modules.masterfs.providers.*. Would of course be
> an incompatible change (-> /3) but there are just a few friends to update. NB
> APIs generally use URL (or URI) when we want to express a (possibly) virtual
> file path, where FileObject is unsuitable because it can only refer to an
> existing file.
- API uses java.net.URL.

(In reply to comment #4)
>> [JG02] [...instead] replace usages of File with URL
>> in the existing org.netbeans.modules.masterfs.providers.*.
>
> - API uses java.net.URL.

Which API? I do not understand this response.

(In reply to comment #5)
> Which API? I do not understand this response.
I partly fixed your suggestion [JG02] (see second attachment).

Now bridge module defines interface:
public interface ProvidedExtensions2 {
ProvidedExtensions.IOHandler getCopyHandler(URL from, URL to);
...
}

This interface implemented by VCS and remote file system. For example:
class FilesystemInterceptor extends ProvidedExtensions implements FileChangeListener, ProvidedExtensions2

No changes in master file system.

Such solution allows to avoid incompatible change.

(In reply to comment #6)
> Such solution allows to avoid incompatible change.

I see. But then why not simply put the new interface in masterfs? Or simpler yet, change ProvidedExtensions directly: introduce methods taking URL rather than File, @Deprecate the methods taking File, and make the URL methods delegate to the File methods for file-protocol URLs (else doing nothing unless overridden).

You would probably not then need a remotefs.bridge module. The patch does not show any example usages of Bridge.getExecutionEnvironment(URL) so it is not clear to me what it does, but perhaps this could simply be in org.netbeans.modules.nativeexecution.api; and FileProxy seems to merely convenience methods.

(In reply to comment #7)
Let me explain proposal that was discussed on initial meeting with Jaroslav Tulach, Tomas Stupka, Vladimir Voskresensky and me.
New API (module) solve problem: "VCS should work for remote file system that used only in C/C++".
We agreed that API (module):
- is a separated module.
- module is a friend of master file system.
- module has two friends: remote file system and versioning.
- no changes in master file system.
- module help versioning migrate from java.io.File to "what ever that support restricted set of java.io.File methods" (See http://wiki.netbeans.org/IoFileInVCS).
This solution allow to divide developed code by teams:
- CND team supports bridge and implements extension of ProvidedExtension+ProvidedExtension2 in the remote file system.
- VCS team implement additional ProvidedExtension2.
This solution allow to restrict access to ProvidedExtension2. Interface is not needed to be implemented in the master file system, parsing API and parsing lucene support.

So if ProvidedExtension2 is located in the master file system it will inherit unneeded  friends.
If ProvidedExtension2 is located in ProvidedExtension it will have influence on the master file system, parsing API and parsing lucene support.
Jaroslav Tulach and Tomas Stupka what do you think about?

About Bridge.getExecutionEnvironment(URL):
The versioning executes commands in some clients (see http://wiki.netbeans.org/FOableVCSSPI195124#Remote_execution).
The bridge should provide possibility to execute commands on remote file system.
So method Bridge.getExecutionEnvironment(URL) allows to access to remote execution.
You are right the method getExecutionEnvironment(URL) an be moved in nativeexecution.api.
But in IMHO it is closer to remote file system.
Vladimir Kvashin and Andrew Krasny what do you think about?
Probably for the versioning method Bridge.getExecutionEnvironment(FileProxy) is more convenient.
In this case the method getExecutionEnvironment(FileProxy) should be in the bridge.

(In reply to comment #8)
> If ProvidedExtension2 is located in ProvidedExtension it will have influence on
> the master file system, parsing API and parsing lucene support.

Well the parsing infrastructure need not attempt to support non-File locations unless and until this becomes a user requirement. JG02 was only a suggestion; it seemed cleaner from an API design perspective to generalize the existing API in masterfs, rather than create a brand new module with almost the same API which is intended to sometimes supersede the old one.

VV1: instead of using class names/reflection, you can use fs.getRoot().getAttribute("...") and make FS impl responsible for that

VV2: Bridge looks very similar to URLMapper. What about FileProxyMapper? I think it is again FS impl who is responsible for real conversion URL <-> FileProxy, so FS impl should provide SPI impl for that

VV3: remotefs.bridge mention "remote", but I don't see real reason for that. I expected current packages from masterfs would be extracted (and renamed) into external module and VCS would depend on this new module (and not master or remote fs). FS impl will also depend on that module and will call defined hooks.

Considering VV2, [JG03] What is the justification for FileProxy - why does the normal FileObject not adequately "[represent a] file on [a] remote or local file system"?

As before, it is difficult to evaluate this proposal when the patch does not include actual code making use of the API and SPI.

(In reply to comment #11)
> Considering VV2, [JG03] What is the justification for FileProxy - why does the
> normal FileObject not adequately "[represent a] file on [a] remote or local
> file system"?
Jarda?
As I understood from explanations during our meetings: VCS can not work in terms of FO, because of deadlocks. Also FOs are about existed files. java.nio.Path was considered in the beginning as replacement of java.io.File in interfaces, but it is very heavy weight, so more simple wrapper around needed java.io.File methods was chosen.

Right, versioning should not make any calls to FileObject to avoid deadlocks, thus it is not adequate to use FileObject when asking versioning to do some operation.

Two comments to the patch:

Y01 I'd like ProvidedExtensions2 to be independent of ProvidedExtensions. Right now it does not inherit from it, good. But ProvidedExtensions2 reference IOHandler. Just define another IOHandler interface in ProvidedExtensions2. Then all references will be gone, I think, and implementors of ProvidedExtensions2 will no longer need dependency on masterfs.

Y02 Rename ProvidedExtensions2 to something meaninful. Maybe File(Proxy)OperationsInterceptor?

[TS1] minor - ProvidedExtensions2.DeleteHandler2 could be called DeleteHandler

[TS2] the same as [VV3] - didn't we agree on having a new fs bridge module? VCS will depend on it instead of depending on masterfs and the bridge will manage the communication between the filesystems (masterfs, and remote) and VCS.

[TS3] based on TS2, and VV3 - there also should be a replacement for o.n.m.masterfs.providers.AnnotationProvider. The same applies for the methods from FileChangeListener, which is implemented by ProvidedExtensions. 

[TS4] regarding the type representing a file: 
- io.File doesn't work in case of the remote filesystem
- we agreed the FileObject is problematic because of the contract between VCS
and FS that no reentrant calls should be made (deadlocks). 
- java.nio.Path was originaly suggested, but is considered to be too heavyweight 
- regarding FileProxy and URL - at some point we also have to enhance the public VCS SPI (issue #195124) by using a different file type then io.File. FileProxy in its current state (private/friend) wouldn't work. VCS could use URL for the communication between the FS bridge and the particular VCS systems and internally FileProxy to do whatever it already needs to do with a io.file (http://wiki.netbeans.org/IoFileInVCS)

[TS5] there are also missing some other methods from masterfs ProvidedExtensions - 
e.g. beforeChange(FileObject fo), or fileDeleted(FileObject fo). How would that work with the remote FS?

Created attachment 112743 [details]
Third iteration

- renamed ProvidedExtensions2
- delegate proxy operations to remote file system
- remove dependency on masterfs
- implement all bridge methods

VV4: we should remove dependency VCS from MasterFS, so FileProxyOperationsInterceptor should not mention itself as add-on for org.netbeans.modules.masterfs.providers.ProvidedExtensions, but rather should be used instead ProvidedExtensions in both VCS and MasterFS, then org.netbeans.modules.masterfs.providers.ProvidedExtensions will be deprecated.

AS1: Sorry for misunderstanding but [VV3] [TS2] [TS5] [TS3] [VV4]
assume changing masterfs and making masterfs dependent from fsbridge.
Jaroslav could you conform it?
My solution was:
- do not touch masterfs
- VCS and RemoteFS implementers of ProvidedExtensions mix FileProxyOperationsInterceptor interface in extension of ProvidedExtensions

[TS6] what about entirely giving up on FileObject in FileProxyOperationsInterceptor? My understanding is that they are in ProvidedExtension only for historical reasons and the first thing we do with them today in VCS is to convert them to io.File anyway. 

[TS7] can't find an equivalent to new File(file, name) - something like Bridge.createFileProxy(fileProxy, name)
In the meantime also found out that .listFiles() is needed too. Already updated http://wiki.netbeans.org/IoFileInVCS#io.File_usages_in_VCS

[TS8] minor - what about renaming createLocalFileProxy(File) to createFileProxy(File) - would be equivalent to other createFileProxy(...) methods

[TS9] looks like Bridge.createFileProxy(final FileObject file) assumes that the given FO is either a remote FO or io.File. In case the FO isn't remote a FileProxy is created based on new File(FO.getPath()). That would be wrong in cases like when the VisibilityQuery asks for the visibility of a file from a jar file.

Y03 I've just realized that the proposed FileProxy exposes non-official API by returning ExecutionEnvironment. This does not make much sense and I will use this opportunity to convince Vladimir and co. to start and finish API review of that module before this one finishes.

Y04 Rename FileProxy to VCSElement (according to ADE terminology, the versioned files are not files, but elements, so it makes sense to talk about elements in context of versioning).

Y04 seems like a bad step for API comprehensibility; "element" is as close to a meaningless term as you can get. It is also inconsistent with the naming convention of java.io.File and org.openide.filesystems.FileObject to which the proposed interface would be an obvious analogue.

(Regarding comment #12, java.nio.file.Path is exactly what we want, but for JDK 6 support I doubt we could legally introduce a copy of the relevant NIO.2 interfaces.)

(In reply to comment #21)
> Y03 I've just realized that the proposed FileProxy exposes non-official API by
> returning ExecutionEnvironment. This does not make much sense and I will use
> this opportunity to convince Vladimir and co. to start and finish API review of
> that module before this one finishes.
It would be really great :-)

(In reply to comment #22)
> Y04 seems like a bad step for API comprehensibility; "element" is as close to a
> meaningless term as you can get. It is also inconsistent with the naming
> convention of java.io.File and org.openide.filesystems.FileObject to which the
> proposed interface would be an obvious analogue.
I agree with Jesse. If Jarda wants to have vcs mentioned, we can think about VCSFile, VCSPath, VCSFilePath, VCSFileProxy, but VCSElement is not the best name

> 
> (Regarding comment #12, java.nio.file.Path is exactly what we want, but for JDK
> 6 support I doubt we could legally introduce a copy of the relevant NIO.2
> interfaces.)
btw, java.nio.file.Path is rather expensive object to create. URL/URI has also performance penalty when re-created intensively for the same paths.

(In reply to comment #24)
> java.nio.file.Path is rather expensive object to create.

Huh? Path is just an interface. The implementation can be as flyweight as you like.

(In reply to comment #22)
> (Regarding comment #12, java.nio.file.Path is exactly what we want, but for JDK
> 6 support I doubt we could legally introduce a copy of the relevant NIO.2
> interfaces.)
A lot of problem with nio.2
- package depends on java 7 language.
- package depends on another java 7 packages.
- package depends on implementation.
So we disided to not take java.nio.file.Path

Right, I do not think it is practical while maintaining Java 6 as a baseline.

> Y04
agree with Jesse and Vladimir. Lets stick with VCSFileProxy. afaik element in the understanding of ADE (or ClearCase) represents more than just a file and is overreaching the scope of the discused case. 

> Right, I do not think it is practical while maintaining Java 6 as a baseline.
we could try to keep the new remotefs able api/spi as friend until jdk7 and see if we are able (or willing) to switch to nio.path then.

(In reply to comment #23)
> (In reply to comment #21)
> > Y03 VCSFileProxy exposes non-official API by
> > returning ExecutionEnvironment. 
> It would be really great :-)

Should stabilization of nativeexecution API be a problem we can use the newly defined ProcessBuilder as introduced in issue 205104.

we have tried today to provide new patch with VCSFileProxy, but finally discovered some points which I'd like to mention, because they can cause rethink of approach :-)
(VV5) SPI currently provided by masterfs (i.e. AnnotationProvider) is implemented by some other than vcs modules. I.e. if we'd like remote files to be badged by parsing infra with error badge => remote fs should introduce the same SPI and parsing should have dependency on it :-(
So, it looks more logical to have AnnotationsProvider extracted in common place and used by both: masterfs and remotefs, while implemented by vcs and parsing api.
(VV6) AnnotationsProvider as is has method to return InterceptorListener which is always returns ProvidedExtensions or null (i.e. parsing api) =>
I'd prefer to get rid of getInterceptor from AnnotationsProvider.
(VV7) InterceptionListener interface is written in terms of FileObjects, but real implentations are always ProvidedExtensions derived classes dealing with java.io.File, so it's under question if we need InterceptionListener at all.

(VV8) java.io.File is used in ProvidedExtensions declared in masterfs to pass needed info, so if ProvidedExtensions is left as is and new bridge module between vcs and masterfs will be created to translate operations from java.io.File into VCSFileProxy => we will have to anyway introduce RemoteFileProxy in remote fs to have tranlation of RemoteFileProxy into VCSFileProxy => the question, why we should introduce one more level of proxing and complexity?
Why it's not better to have FileProxy and FileProxyOperations in common module which is used by all components: masterfs, remotefs and vcs

Re. VV8: common module with FileProxy and Operations is versioning.core. If you want to make it less heavy (in terms) of dependencies, sure let's do it. masterfs cannot depend on versioning.core, as it would be platform->ide cluster dependency which is not allowed. There will have to be masterfs.versioning module bridging the functionality.

(In reply to comment #32)
> Re. VV8: common module with FileProxy and Operations is versioning.core. If you
to be more precise - it will be. te plan is to keep the versioning module with the current public spi
and create versioning.core witch would define:
- the spi between filesystems and vcs
- VCSFileProxy and VCSFileProxyOperations 
- the new planned vcs spi based on VCSFileProxy (friend)

Re. VV8 - my understanding at the moment is that we agreed to place VCSFileProxy into VCS and also that everybody seems to agree with its signature. Filed issue #205612 to cover the changes necessary for a new VCS SPI based on VCSFileProxy.

So, after extracting issue #205612 as separate task, what is expected to be done in the context of this issue?

I.e. we can discuss moving AnnotationProvider into separate place from masterfs to have a way to lookup it in remotefs and ask for remote fs FileObject annotations.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/e48039cb3ba0
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- introduce new module "Remote FS to Versioning Bridge"
- implement closed SPI defined in org.netbeans.modules.versioning.core.filesystems package

API was moved in the bug #205612
So this bug is about implementation of a bridge between remote FS and VCS.

Remove dependency on bug #199812 because fix of bug #205104 is enough for remote execution.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/693ae6c4209a
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- try to fix unit test on Windows

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/a9e56c7a8c2f
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- update versioning core interfaces
- add test for normalization

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/baf24fb46e62
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- extract annotations interface
- introduce interceptor test

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/b70273f9812e
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- fix method canWrite

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/c4dd9a58261a
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- fix rename

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/78166c27534f
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- fix to proxy methods

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/5688c2b39b16
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- adjust interceptor interface with version.core

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/d3d771b41e9c
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- use version.core API

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/3fd8992db9db
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199806 Provide bridge between remote file system and VCS
- fix unit tests