This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 42050

Summary: [2004-05-12] Provide a way to filter out files/folders
Product: platform Reporter: Martin Entlicher <mentlicher>
Component: FilesystemsAssignee: rmatous <rmatous>
Status: RESOLVED FIXED    
Severity: blocker CC: apireviews, dkonecny, jglick, jrojcek, mmirilovic, phrebejk, rmatous, ttran
Priority: P1 Keywords: API, API_REVIEW_FAST, UI
Version: 4.x   
Hardware: All   
OS: All   
URL: http://openide.netbeans.org/proposals/loaders/fsextension/design.html#hiding
Issue Type: ENHANCEMENT Exception Reporter:
Bug Depends on:    
Bug Blocks: 41535, 42008    
Attachments: Acceptable ?

Description Martin Entlicher 2004-04-15 15:47:34 UTC
It's necessary to filter out some unimportant
files (like .#*, *~) or folders (like CVS/, .svn/).

Currently the only component that can hide these
files is the filesystem. However this have big
(and for users unexpected) consequences on some
operations, especially copy and move. See e.g.
issue #42008.

The solution would be to make the filter on the
presentation layer. There can be e.g. some kind of
"VisibilityQuery", that will tell what should be
visible and what not (e.g. boolean
VisibilityQuery.isVisible(java.io.File)).
Comment 1 Martin Entlicher 2004-04-15 15:56:01 UTC
VcsFilesystem also introduce one more ugly thing, that is somewhat
related. It creates (virtual) FileObjects for non-existing files
(files that exist only in VCS repository). If the query could "add"
some files, then virtual file objects would not have to be created at all.
Comment 2 Jesse Glick 2004-04-15 16:11:57 UTC
Nothing to do with build system.
Comment 3 Jesse Glick 2004-04-15 16:12:50 UTC
A nice idea, I think - and fits nicely into plans for Filesystem
Extensions (see URL). Suggest perhaps

  boolean VisibilityQuery.isVisible(FileObject);
  Boolean VisibilityQueryImplementation.isVisible(FileObject);

with null in the latter meaning "unknown". Is there any reason to use
File rather than FileObject?

Not clear who would implement this query. One global impl for now? FS
Ext proposal implies that it would be more complicated.

Also not clear what the effect would be of having a FileObject for a
CVS folder when it is in fact on a CVS filesystem.

In any event, this RFE has nothing to do with the build system.
Logical views can do their own filtering but they have no way of
knowing what to do filter other than hardcoding a list such as

"^(CVS|SCCS|vssver\\.scc|#.*#|%.*%|\\.(cvsignore|svn|DS_Store))$|^\\.[#_]|~$"

(effectively what Ant 1.6 uses).
Comment 4 Jesse Glick 2004-04-15 16:17:47 UTC
IMHO "virtual file objects" should simply be killed right away.
(Certainly the default for new VCS filesystems should not be to create
them, as it does now!) They cause a lot of unnecessary bugs. When you
are looking at a list of files, you should see files that are really
there. If you want to know about VCS operations, go to the Versioning
view.

If we really need to display some file names (with blank "removed"
icons) in the logical view for some reason, it should be a separate
API serving information from the VCS to the folder node (or we could
wait for Looks to let the VCS module insert such nodes itself).
Comment 5 Martin Entlicher 2004-04-15 16:38:53 UTC
No special reason for java.io.File. FileObject will be better, because
FileObject is produced by the filesystem.

Currently every VCS filesystem would implement the query, perhaps also
LocalFilesystem if the "Ignored Files" property should be present
somewhere in 4.0.

--

Currently "Versioning View" contains the whole filesystem, not just
the projects contents. This can be very handy, but OTOH it's not very
flexible. If we decide not to show virtual files in logical/physical
views, we need to assure that users will not have problems with that.
Comment 6 rmatous 2004-04-27 16:31:34 UTC
Jesse wrote:
[Also not clear what the effect would be of having a FileObject for a
CVS folder when it is in fact on a CVS filesystem.]

If you won't have such FileObject you can't manipulate with such
folder and its children, you can't implement any raw file view. 

The second one problem are "virtual file objects" that prevent you
from copying any folder containing such FileObjects because if you ask
"virtual file objects" for input stream you get IOException. All views
must represent such virtual FileObjects there is no way how to get rid
of them (you must implement view directly on top of java.io.File -
which is what we don't want - at least because of external changes ). 
I completely agree that virtual FileObjects could be only part of
versioning explorer.


Frankly I think that FileSystem implentations based on top of
java.io.File shouldn't neither hide files nor provide any virtuals
(e.g. the same for .nbattrs). I also don't like idea of having methods
like FileObject.isVisible.

View implementation should be responsible for final decision what will
be presented. Package view won't probably present cvs folders. On the
other hand all files view (or midnight commander view or whatever
else) might present cvs folders.

But view implementation won't probably know what to hide for cvs, rcs
and any arbitrary supported vcs. Should every view provide its own API
and SPI (some query) that will be implemented by VCS FS impl. ? Or
should be just one such API general for all views (and answer would be
considered just as a hint) ? Should be such API part of  FS Extension
API  ?  What with this issue for promo-D  ?


Comment 7 Jesse Glick 2004-04-27 23:20:30 UTC
"Frankly I think that FileSystem implentations based on top of
java.io.File shouldn't neither hide files nor provide any virtuals
(e.g. the same for .nbattrs). I also don't like idea of having methods
like FileObject.isVisible." - agreed. These APIs were a nice idea but
they have caused more problems than they are worth.

"Should every view provide its own API and SPI (some query) that will
be implemented by VCS FS impl.? Or should be just one such API general
for all views (and answer would be considered just as a hint)?" -
probably the latter, IMHO.

"Should be such API part of FS Extension API?" - could be. The FS EXT
API is whatever we make of it. If we have direct references to e.g.
Look than of course it needs to be a separate API; for VisibilityQuery
(and similar file-related things like SharabilityQuery and
CollocationQuery) they could be part of the regular Filesystems API or
not. If we do want to make a whole new API to contain such things, I
think it would be great, but we need to make the decision pretty soon.

"What with this issue for promo-D?" - I don't know, were you planning
on taking any vacation before feature freeze? :-) Provisionally
marking as a planned item for promo-D but it's Trung's call whether we
can afford this.
Comment 8 _ ttran 2004-04-27 23:50:06 UTC
forget about FS Ext if we want to talk about Promotion D

> Logical views can do their own filtering but they have no way of
> knowing what to do filter other than hardcoding a list such as
> 
>
"^(CVS|SCCS|vssver\\.scc|#.*#|%.*%|\\.(cvsignore|svn|DS_Store))$|^\\.[#_]|~$"
> 
> (effectively what Ant 1.6 uses).

I would suggest buildsys (java package views) to do this right away. 
Web app project would be able do the same.  That's very good for ppl
who for some reason prefer external VCS client (GUI or not) to our VCS
integration and thus don't use IDE VCS modules




Comment 9 Jesse Glick 2004-04-28 00:14:04 UTC
Re. not using the IDE's VCS modules - not such a great idea, I'm
afraid, because even if you never use the VCS integration features (*)
the new CollocationQueryImplementation relies on there being VCS
mounts, and without that your projects will get filled with
private-storage absolute paths when you should be using shared-storage
relative paths.

(*) I'm making an honest effort to use the CVS integration in addition
to command-line CVS (and Emacs' PCL-CVS). It really doesn't work. The
persistent CVS metadata cache is so badly broken if you use an
external CVS tool that not only are file status badges almost always
wrong (or missing entirely), but the CVS context menus usually either
are broken ("Refresh" not in the CVS submenu but a nonfunctional
"Refresh" in the main context menu); do nothing when you select an
action; or (most frightening) attempt to run the command on a
completely different file (or folder!) than you actually have
selected, which could lead to tragedy if you don't have the "Expert"
mode on so you can cancel the action before it is run. I don't know
how many of these problems are a result of using external CVS commands
(since I do so every time), but I have to assume that QA never tries this.
Comment 10 Jaroslav Tulach 2004-04-28 13:46:09 UTC
I somehow got cced on this issue, I do not orient well and I do not
want to contribute to other misorientation of other people. So: 1.
should not this issue have a web page listing usecases/non-usecases
that would be uptodate (the fsext page obviously is not)? 2. Is not it
better (before the topic settles down a bit) to talk on a mailing list?

I think that #1 is crucial, without it a conclusion is hard to reach.
Comment 11 rmatous 2004-04-28 15:09:47 UTC
I think that Petr Hrebejk is missing on CC list because AFAIK he works
on project views. I think that his opinions attached to this issue 
could be good basis for formulating use cases. 
Comment 12 rmatous 2004-04-28 15:11:16 UTC
Petr imagine that vcs support won't hide  vcs specific folders and
files like CVS folder anymore. What should come after ? 
Comment 13 Jesse Glick 2004-04-28 19:08:16 UTC
In the absence of a shared query, we could put identical filtering
into both FolderChildren (openide/loaders) and PackageView
(java/project) as a temporary hack.
Comment 14 Martin Entlicher 2004-04-30 17:18:03 UTC
> The persistent CVS metadata cache is so badly broken if you use an
> external CVS tool that not only are file status badges almost always
> wrong (or missing entirely), but the CVS context menus usually
> either are broken ("Refresh" not in the CVS submenu but a
> nonfunctional "Refresh" in the main context menu)

This can happen when the folders/files are lost from VCS cache and
folders are marked as [Local]. A reproducible case was fixed into NB
3.6. If this happens, it's best to run Rec. Refresh on the FS root (if
 it's not nb_all in which case you can get OOME :-(.

> do nothing when you select an action;

I haven't seen that yet. Do you have repro case?

> or (most frightening) attempt to run the command on a
> completely different file (or folder!) than you actually have
> selected, which could lead to tragedy if you don't have the "Expert"
> mode on so you can cancel the action before it is run.

I've never seen that in 3.6 FCS. It was in some old 3.6 dev version
due to a bug with selected nodes in some component. Though I've
encountered this behavior several times in current dev. IMHO it's not
a problem of VCS modules, but of components which badly set the set of
selected nodes. It looks not to be deterministic to me. I would submit
a bug for it, but it's not clear to me yet whose fault it is.

> I don't know how many of these problems are a result of using
> external CVS commands (since I do so every time), but I have to
> assume that QA never tries this.

Just the very first one IMO. We have an enhancement for VCS cache
auto-correction using CVS/Entries.
Comment 15 Martin Entlicher 2004-04-30 17:28:51 UTC
That was offtopic here, sorry.

So it's not planned to implement someting like VisibilityQuery into
promotion D, right? It should be considered IMO.

The hardcoded list is O.K. for me, but PVCS profile filters also
"vcs.cfg". However that file does not seem to be used much in recent
versions of PVCS.
In addition to that CVS profile filters .nbcicat.bat file on Windows,
which it needs to retrieve commit template. This is a temporary file
which lives there only while commit is running, so it should not cause
problems.

Concerning virtual files, we can disable them and have them only in
Versioning view. However, the impact on users workflow and usability
needs to be analyzed first. Users will loose the file after "CVS
Remove" and will have to switch the view to be able to commit it (or
do the commit on the parent folder).
Comment 16 Jesse Glick 2004-04-30 17:57:35 UTC
"Concerning virtual files, we can disable them and have them only in
Versioning view. However [...] Users will loose the file after "CVS
Remove" and will have to switch the view to be able to commit it (or
do the commit on the parent folder)." - naturally. But if you're
deleting a file, in most cases you're making some other changes in
other files anyway (e.g. to not refer to it any longer), so you're
going to have to commit from some ancestor folder anyway in order for
your commit to make any sense. Probably committing from the Versioning
view is better anyway since you can see at a glance all added,
removed, and modified files (I hope).

Whatever the UI decision, IMHO keeping deleted files as FileObject's
is just not an option architecturally. It's very buggy. If we *have*
to display them in the logical view for UI reasons, that should be
done explicitly by the logical view, appending them to the real files
in the folder for purposes of o.o.n.Children.
Comment 17 rmatous 2004-05-05 17:17:13 UTC
Created attachment 14733 [details]
Acceptable ?
Comment 18 Jesse Glick 2004-05-05 18:55:42 UTC
Prefer isVisible (positive aspect) to isHidden (negative aspect).

Consider using Boolean in VQI (null = no information); or at least
document in VQ that the default is visible unless at least one VQI
says hidden.

What are the expected impls of VQI? One global VQI in vcscore
delegating to the VCS of the FileObject if there is one? Also a global
VQI independent of vcscore for non-VCS-controlled files?

Should go thru apireviews as fast track if you intend to do it.
Comment 19 Jaroslav Tulach 2004-05-06 09:13:28 UTC
I do not like solutions without usecases. May I (once again) ask for
list of all usecases non-usecases this is trying to solve?
Comment 20 rmatous 2004-05-06 11:39:40 UTC
Use case: 
- presentation layer needs to  hide some unimportant java.io.File-s
and hasn't enough information for decision what is unimportant and
expects other module to provide such sort of information. There is
important to mention that this information doesn't need to be obeyed
at all. 


Additional comment:
There is expected that deciscion what is unimportant should be
provided by modules that implement FileSystem based on top of
java.io.File that provides FileObjects reachable outside this module
and is expected that   these FileObjects will be visualized also
outside of this module. 


Comment 21 Jaroslav Tulach 2004-05-06 12:45:11 UTC
If this is the only use case I suggest:

public boolean FileObject isHidden ();
protected boolean AbstractFileSystem.hidden (String path);

this satisfies the usecase, makes sure that the filesystem is the one
that decides about the hidden state and allows us to use the method in
opeide-loaders.jar immediatelly.
Comment 22 _ ttran 2004-05-06 12:56:20 UTC
This is needed for D.  Correct?  If so please add core-promod and set
TM  to promo-d
Comment 23 _ ttran 2004-05-06 12:58:02 UTC
Yarda: you ignored Jesse's previous comment

> Prefer isVisible (positive aspect) to isHidden (negative aspect).

Comment 24 rmatous 2004-05-06 13:24:03 UTC
Naturally I considered this solution but I don't like it:
- definitely every filesystem impl. isn't supposed to provide decision
about java.io.Files
- there isn't expected that every filesystem impl. must provide
FileObjects reachable outside module

There isn't necessary to make sure that the filesystem is the one that
decides about the hidden state. I think that mentioning it clearly in
documentation is enough.

I also considered to put VisibilityQuery in org.openide.filesystems
package. But then I found that there are similar queries that are more
 or less related to filesystems and maybe all (or some) of them would
deserve to be in one package (maybe the current one isn't the best). 

Comment 25 rmatous 2004-05-06 13:29:32 UTC
I have no strong opinion about negative or positive aspect. If there
is someone who has stronger preference than me then please let me know
and I change it appropriately.
Comment 26 Jaroslav Tulach 2004-05-06 13:44:02 UTC
I do not mind whether we use isHidden or isVisible. I am just saying
that usage of project/queries is overkill if we have just the usecase
given by Radek.
Comment 27 David Konecny 2004-05-06 14:00:54 UTC
What is overkill? Query design pattern (how I call it) it very
lightweight and is more flexible that a new method in FS - somebody
else can provide different impl, hide default ones, etc. Also
evolution story of Query design pattern is clearer and should be
simpler because of strict split of API/SPI.

Btw. there is already couple of other more or less FS oriented queries
in projects/queries module and if there would be more time I would
propose to create new module openide/fsqueries (or fsextensions but
that already has some other meaning) where all these FS related
queries should be moved. But we do not have that time. From time point
of view this query perfectly fits into this module.
Comment 28 David Konecny 2004-05-06 14:02:46 UTC
Huh, I wanted to say "from this point of view" and not from "time point".
Comment 29 rmatous 2004-05-06 14:20:14 UTC
Frankly I would be surprised if there was added method e.g.
java.io.File.isVisible. But I would be completely OK if I had
something like Explorer.isVisible (File), FileView.isVisible (File) or
Project.isVisible (File). The difference is that "visibility" doesn't
seem to be naturall  property of java.io.File.
Comment 30 rmatous 2004-05-06 14:41:10 UTC
I'm completely OK with tiny recommendations but if you have strong
negative opinion and you completely disagree with proposed API (even
after last explaining comments) then please don't let me wait the
whole week if I could ask.
Comment 31 Jesse Glick 2004-05-06 14:46:00 UTC
I'm OK with the current API though I would repeat the request for
positive wording in all API methods to avoid confusing people with
double negatives. Also I would like to know concretely who will
initially implement VQI, i.e. whether it will work on nonversioned
directories, versioning directories, or both, and how this will happen.
Comment 32 Jaroslav Tulach 2004-05-06 15:36:03 UTC
1. The issue 41074 suggests to move the filesystem related queries
into fs package. 

2. The usecase provided by Radek says "There is expected that
deciscion what is unimportant should be provided by modules that
implement FileSystem". That speaks for having this as a method of FO
and FS

3. Queries seem to allow other modules than providers of objects to
answer the queries. E.g they are more powerful than the only usecase
Radek is speaking about. That is what I meant by overkill.


I am fine with whatever solution is choosen, but as I was asked to
provide an opinion, I am saying that the queries solution offers more
than what was requested by the described usecase(s). That is imho
signal that either the usecases or the solution are not correct. 

I'd like to emphatize once more the issue 41074: e.g. move both
queries into filesystems package to signal that the threading is
different for sharability and visibility than for filebuilt query.
Comment 33 Martin Entlicher 2004-05-06 15:49:57 UTC
> Also I would like to know concretely who will initially implement VQI, 
> i.e. whether it will work on nonversioned directories, versioning
> directories, or both, and how this will happen.

vcscore module will definitely implement VQI. There are more files
then just CVS and *~ that are ignored, individual VCS profiles can
define additional files to ignore.

I also suggest that it would be good to have some global ignore
property so that the user can e.g. hide all CVS folders regardless of
what kind of filesystem they happen to use. (Or perhaps that global
property can be managed by masterfs).
Comment 34 Jesse Glick 2004-05-06 16:20:42 UTC
"I also suggest that it would be good to have some global ignore
property so that the user can e.g. hide all CVS folders regardless of
what kind of filesystem they happen to use. (Or perhaps that global
property can be managed by masterfs)." - this makes sense to me. Even
if you are not mounting a CVS filesystem, e.g. in case you wish to use
an external CVS tool, you would very likely still want to hide CVS
folders. This could be done with either the query style or the
FileObject.isHidden (prefer isVisible) style, I think. (In the latter
case MasterFS would supply a default in case the delegate FO did not
mark the file as hidden.)
Comment 35 Jesse Glick 2004-05-06 16:25:49 UTC
Re. "move both queries into filesystems package" - I am fine with that
(though it would be a slight regression in API clarity that
o.o.filesystems has no distinct API and SPI packages). Trung I think
you had some problem with doing such a change before - do you still
care? If not, I would suggest that Radek move both SharabilityQuery
and CollocationQuery (and their SPI interfaces) into o.o.filesystems.
If we don't do that now, we could still move these into another
package/module in the future (but not o.o.filesystems) by deprecating
the original classes and interfaces, moving them into the new module,
introducing new copies with the desired package name, and delegating
bidirectionally. Obviously it would be much simpler if we got the
location right the first time around.
Comment 36 rmatous 2004-05-06 17:25:25 UTC
I agree with almost everything in last comments. Just a few notes:

- I can't help but I don't like FileObject.isVisible. I preffer
Whatever.isVisible (FileObject) style
- I agree that project/queries isn't the best package but on the other
hand  o.o.filesystems package isn't also good place for me. Why not to
really introduce standalone module openide/fsextension  (or fsutils or
whatever name - I really don't intend to replace DataSystem into
promo-D ) ?
- if package with queries should be moved then I would preffer to do
it ASAP (definitely before promo-D)

Comment 37 Jesse Glick 2004-05-10 16:49:12 UTC
*** Issue 43010 has been marked as a duplicate of this issue. ***
Comment 38 rmatous 2004-05-11 14:24:18 UTC
Will be integrated into project/queries to be able use and implement
VQI according to SPI. As soon as there is agreement where to move
mentioned queries will be moved (see #41074).

Initially implementation VQI:
- VCS Support 
- global impl. (accroding to Martin Entlicher comment) - probably
implemented in core and presented to user in Tools/Options (IDESettings)
Comment 39 rmatous 2004-05-11 18:04:26 UTC
I realized that there is missing some sort of notification about
changes cause:
- VQI may be disabled if module that provides it is disabled
- global VQI based on regular expression provided by user via
IDESetting  must fire events from its nature 

I suggest to change static method isVisible to nonstatic and implement
VQ as a singleton and add methods addChangeListener,
removeChangeLister (both method also into VQI)  
Comment 40 rmatous 2004-05-12 16:20:51 UTC
Integrated. For next discussion about moving queries use #41074.