Bug 199136

Created attachment 108694 [details]
nps snapshot

it significantly affects usability of full remote

As far as the "full remote" fs goes, the primary fix is on your side - to improve behavior of yourFileObject.getMimeType(). That would be acceptably low risk fix for 7.0.1, imho.

Otherwise I agree, it is desirable to move the file type recognition outside of AWT: ergonomics#8b479b68575f This kind of fix is however not fully functionally compatible (I had to fix Files and Favorites views) to be considered for safe backport.

Moreover it seems to me that this fix is not enough anyway. The performance of remotefs still remains poor.

Integrated into 'main-golden', will be available in build *201106051000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/88d091fc8b45
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: help for #199136 - 27s - FolderChildren.createNodes(...) should run on background
(transplanted from 460357e755a9a25ec78d01ccdee17d4919e2b4c7)

If I understand correctly, this is going to be an effectively incompatible change for anyone using FilterNode to produce a refined view of a folder tree. Which is a lot of people. Right?

If so, I think this fix has to be modified; perhaps only activate it for files which "look" remote (see bug #159628).

I do not think I have any similar reports of EQ freezes from the Hudson remote filesystem (hudson/src/org/netbeans/modules/hudson/impl/RemoteFileSystem.java), though bug #198868 in this area seems to have regressed recently.

I've marked our FS as slow and remote
http://hg.netbeans.org/releases?cmd=changeset;node=460357e755a9

Causing AE in LibrariesNode, see issue #199308
Incompatible change.

I've reported bug  199391 to track the FolderChildren rewrite. That one is not going to happen for 7.0.x. After discussing this issue with Vladimir, I am assigning it to remotefs to speedup its getMimeType implementation.

Here is a patch that I used to demonstrate that fixing the "getMimeType()" will improve overall performance of the explorer:

diff -r b911094529a9 dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java
--- a/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java      Mon Jun 13 13:59:29 2011 +0200
+++ b/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java      Tue Jun 14 13:28:40 2011 +0200
@@ -114,6 +114,11 @@
             flags &= ~mask;
         }
     }
+                                                                                                                            
+    @Override                                                                                                               
+    public String getMIMEType() {                                                                                           
+        return "text/plain";                                                                                                
+    }                                                                                                                       
                                                                                                                             
     /*package*/ boolean isPendingRemoteDelivery() {                                                                         
         return getFlag(BEING_UPLOADED);   

Of course you need to delegate to FileUtil.getMimeType and prevent the system to open the streams (which is the cause of the slowness).

Jarda, thanks for the patch. Unfortunately it is not applicable, because in that case binary files and header files without extensions will not work at all.

I understand that such approach can improve something, but it's incorrect.
i.e. we don't ask editor infrastructure to return empty text for documents instead of reading content from files. We just read content out of edt.
the same should be done for mime recognition

(In reply to comment #10)
> the patch. Unfortunately it is not applicable

Of course it is not applicable! That is what I wrote:

(In reply to comment #9)
> Here is a patch that I used to demonstrate that fixing the "getMimeType()" will
> improve overall performance of the explorer...
> ...
> Of course you need to delegate to FileUtil.getMimeType and prevent the system
> to open the streams (which is the cause of the slowness).

the patch just demonstrates all the scrolling problems with remotefs are caused by slow getMimeType.

Method FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
do not call FileObject.getMIMEType().
It does not allow to override getMIMEType()

Stack:
org.netbeans.modules.remote.impl.fs.RemotePlainFile.getInputStream(RemotePlainFile.java:149)
org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166)
org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166)
org.openide.filesystems.MIMESupport$CachedFileObject.getInputStream(MIMESupport.java:407)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern.match(MIMEResolverImpl.java:801)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.accept(MIMEResolverImpl.java:1109)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.access$2900(MIMEResolverImpl.java:727)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.resolve(MIMEResolverImpl.java:687)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.access$800(MIMEResolverImpl.java:658)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Impl.findMIMEType(MIMEResolverImpl.java:317)
org.openide.filesystems.MIMESupport$CachedFileObject.resolveMIME(MIMESupport.java:376)
org.openide.filesystems.MIMESupport$CachedFileObject.getMIMEType(MIMESupport.java:317)
org.openide.filesystems.MIMESupport.findMIMEType(MIMESupport.java:138)
org.openide.filesystems.FileUtil.getMIMEType(FileUtil.java:1483)
org.openide.loaders.ExtensionList.isRegistered(ExtensionList.java:172)
org.openide.loaders.UniFileLoader.findPrimaryFile(UniFileLoader.java:94)
org.openide.loaders.MultiFileLoader.findPrimaryFileImpl(MultiFileLoader.java:365)
org.openide.loaders.MultiFileLoader.handleFindDataObject(MultiFileLoader.java:99)
org.openide.loaders.DataObjectPool.handleFindDataObject(DataObjectPool.java:161)
org.openide.loaders.DataLoader.findDataObject(DataLoader.java:407)
org.openide.loaders.DataLoader.findDataObject(DataLoader.java:380)
org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:535)
org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:493)
org.openide.loaders.DataObject.find(DataObject.java:523)
org.openide.loaders.FolderChildren.createNode(FolderChildren.java:232)
org.openide.loaders.FolderChildren$DelayedNode.run(FolderChildren.java:424)
org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424)
org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968)

I see one way to fix method:
FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
by dirty hack:

private boolean isMimeResolving() {
    for(StackTraceElement element : Thread.currentThread().getStackTrace()) {
        if ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName())) {
            return true;
        }
    }
    return false;
}

Jaroslav, what do you think about?

first implementation of magic cache:
http://hg.netbeans.org/cnd-main/rev/51accf793c88

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/51accf793c88
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow

invalidate magic cache:
http://hg.netbeans.org/cnd-main/rev/33b83e7b506e

read 80 bytes of file:
http://hg.netbeans.org/cnd-main/rev/f3cdd9df1914

vv159170: Please review fixes.
QA: Please verify fix.

(In reply to comment #15)
> I see one way to fix method:
> FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
> by dirty hack:
> 
> private boolean isMimeResolving() {
>     for(StackTraceElement element : Thread.currentThread().getStackTrace()) {
>         if
> ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName()))
> {
>             return true;
>         }
>     }
>     return false;
> }
> 
> Jaroslav, what do you think about?

I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it then.

Possibly consider enhancing the MIMEResolvers API to directly prevent any content sniffing. Would be an additional boolean attribute to some methods, I guess.

(In reply to comment #21)
> I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on
> in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it
> then.
Probably you do not behold that fix overrides getMimeType() and uses ThreadLocal variable. Problem in stack #14. Remote file object is not present at stack. So dirty hack is applicable only for this stack. (See dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemotePlainFile.java)

My question is:
Is it possible use another fix for solving problem in stack #14?

(In reply to comment #22)
> ... 
> Problem in stack #14. Remote file object is not present
> at stack.
> ...
I would restate it as follows (Alexander, please correct me if I;m wrong):
some clients use FileUtil.getMIMEType directly, not FileObject.getMIMEType - in this case thread local does not work.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/33b83e7b506e
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
- add invalidation of magic cache

verified in release701_dev build #5647.
Please review the fix and push it into 7.0.1 branch.

proposed fix speeds up responsiveness, but introduces regressions in file mime-types recognition. 
Can we increase buffer size to have the same mime-recognition result as in local file system case?
For complete fix of the bug the current change made for 7.1 should be ported as well (which redirects icon calculation in bg thread)

(In reply to comment #26)
> Can we increase buffer size to have the same mime-recognition result as in
> local file system case?
Of cause we can increase. Question is in a cost of increasing buffer.
Increasing magic buffer from 80 bytes to 4000 bytes:
- time of executing "od" is increased in 3 times (for example /usr/bin: 3 seconds -> 10 seconds).
- disk space (and number of transferred bytes) is increased in 35 times (for example /usr/bin: 0.2 Mb -> 7 Mb).
- java memory requires another strategy (now it stored on hard ref -> should be stored on soft ref).
Are you agree to have such fee for questionable benefit of recognizing rare headers without extension and without magic in first line?

How about making the buffer size customizable through cli option or in Tools->Options?

http://hg.netbeans.org/releases/rev/fb8ef63a94d8
- increased size of magic 80->4000
- added exception for fixCookieSet

(In reply to comment #22)
> Probably you do not behold that fix overrides getMimeType() and uses
> ThreadLocal variable. Problem in stack #14. Remote file object is not present
> at stack. So dirty hack is applicable only for this stack. (See

I have not noticed before, but I see it now.

> My question is:
> Is it possible use another fix for solving problem in stack #14?

I reported bug 199642 to remind me about this problem. For the time being, inspecting the stack is probably the only way.

http://hg.netbeans.org/releases/rev/3045fefce4b3
- fix get mime type of link remote file object

Some changes was done.
vv159170: Please review fixes again.
QA: Please verify fix again.

as parity tests local vs. remote mime resolving gives 100% same results fix should not introduce regressions.
I agree that it would improve user experience in 7.0.1

Two additional minor fixes
http://hg.netbeans.org/releases/rev/27e0404ee748
http://hg.netbeans.org/releases/rev/f5efb6b423fa

Additional fix of java.lang.IllegalThreadStateException: Should never be called from AWT thread:
http://hg.netbeans.org/releases?cmd=changeset;node=d0d6372977e6

verified in dev build

I reviewed all mentioned fixes, they looks correct and safe enough.
I also wrote a test and ran it on my Linux machine against /home, /usr, /bin, /lib, test detected no differences in mime resolving between local and remote file systems.

Integrated in release701:
51accf793c88 transplanted to 11f14f6f4336
33b83e7b506e transplanted to 1677fa4ef1b0
f3cdd9df1914 transplanted to c1ad674f8d11
fb8ef63a94d8 transplanted to 64b081e52285
3045fefce4b3 transplanted to eca0801a645e
27e0404ee748 transplanted to 4066a35fdd39
f5efb6b423fa transplanted to e84fa045b99b
d0d6372977e6 transplanted to 0a7d3f68a430

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/24ec45e9f0f0
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
- increased size of magic 80->4000
- added exception for fixCookieSet
(transplanted from 64b081e52285f572082f18d03ec184a6d3ac5029)

Integrated into 'releases'
Changeset: http://hg.netbeans.org/releases/rev/11f14f6f4336
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow

verified in NetBeans IDE 7.0.1 (Build 201107102202)