Bug 199136 - RemoteFS.getMimeType is slow
RemoteFS.getMimeType is slow
Status: VERIFIED FIXED
Product: cnd
Classification: Unclassified
Component: Remote
7.0
All All
: P2 (vote)
: 7.0.1
Assigned To: Alexander Simon
issues@cnd
EXCEPTIONS_REPORT perf-profileme
: PERFORMANCE
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-06-03 11:12 UTC by Vladimir Voskresensky
Modified: 2011-07-12 11:49 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
nps snapshot (67.97 KB, application/nps)
2011-06-03 11:12 UTC, Vladimir Voskresensky
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vladimir Voskresensky 2011-06-03 11:12:01 UTC
This issue was reported manually by vv159170.
It already has 1 duplicates 


Build: NetBeans IDE 7.0.1 Dev (Build 20110603-4e44b7e2d1d1)
VM: Java HotSpot(TM) 64-Bit Server VM, 19.1-b02, Java(TM) SE Runtime Environment, 1.6.0_24-b07
OS: Linux

User Comments:
vv159170: adding remote folder (from Services) to Favorites

vv159170: scrolling Favorites when remote folder is exapnded



Maximum slowness yet reported was 27840 ms, average is 17734
Comment 1 Vladimir Voskresensky 2011-06-03 11:12:08 UTC
Created attachment 108694 [details]
nps snapshot
Comment 2 Vladimir Voskresensky 2011-06-03 14:21:03 UTC
it significantly affects usability of full remote
Comment 3 Jaroslav Tulach 2011-06-04 06:39:56 UTC
As far as the "full remote" fs goes, the primary fix is on your side - to improve behavior of yourFileObject.getMimeType(). That would be acceptably low risk fix for 7.0.1, imho.

Otherwise I agree, it is desirable to move the file type recognition outside of AWT: ergonomics#8b479b68575f This kind of fix is however not fully functionally compatible (I had to fix Files and Favorites views) to be considered for safe backport.

Moreover it seems to me that this fix is not enough anyway. The performance of remotefs still remains poor.
Comment 4 Quality Engineering 2011-06-05 14:04:58 UTC
Integrated into 'main-golden', will be available in build *201106051000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/88d091fc8b45
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: help for #199136 - 27s - FolderChildren.createNodes(...) should run on background
(transplanted from 460357e755a9a25ec78d01ccdee17d4919e2b4c7)
Comment 5 Jesse Glick 2011-06-06 14:44:29 UTC
If I understand correctly, this is going to be an effectively incompatible change for anyone using FilterNode to produce a refined view of a folder tree. Which is a lot of people. Right?

If so, I think this fix has to be modified; perhaps only activate it for files which "look" remote (see bug #159628).

I do not think I have any similar reports of EQ freezes from the Hudson remote filesystem (hudson/src/org/netbeans/modules/hudson/impl/RemoteFileSystem.java), though bug #198868 in this area seems to have regressed recently.
Comment 6 Vladimir Voskresensky 2011-06-06 15:51:41 UTC
I've marked our FS as slow and remote
http://hg.netbeans.org/releases?cmd=changeset;node=460357e755a9
Comment 7 Tomas Zezula 2011-06-10 10:33:50 UTC
Causing AE in LibrariesNode, see issue #199308
Incompatible change.
Comment 8 Jaroslav Tulach 2011-06-14 07:23:46 UTC
I've reported bug  199391 to track the FolderChildren rewrite. That one is not going to happen for 7.0.x. After discussing this issue with Vladimir, I am assigning it to remotefs to speedup its getMimeType implementation.
Comment 9 Jaroslav Tulach 2011-06-14 11:29:27 UTC
Here is a patch that I used to demonstrate that fixing the "getMimeType()" will improve overall performance of the explorer:

diff -r b911094529a9 dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java
--- a/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java      Mon Jun 13 13:59:29 2011 +0200
+++ b/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java      Tue Jun 14 13:28:40 2011 +0200
@@ -114,6 +114,11 @@
             flags &= ~mask;
         }
     }
+                                                                                                                            
+    @Override                                                                                                               
+    public String getMIMEType() {                                                                                           
+        return "text/plain";                                                                                                
+    }                                                                                                                       
                                                                                                                             
     /*package*/ boolean isPendingRemoteDelivery() {                                                                         
         return getFlag(BEING_UPLOADED);   

Of course you need to delegate to FileUtil.getMimeType and prevent the system to open the streams (which is the cause of the slowness).
Comment 10 Vladimir Voskresensky 2011-06-15 09:04:13 UTC
Jarda, thanks for the patch. Unfortunately it is not applicable, because in that case binary files and header files without extensions will not work at all.
Comment 11 Vladimir Voskresensky 2011-06-15 09:06:59 UTC
I understand that such approach can improve something, but it's incorrect.
i.e. we don't ask editor infrastructure to return empty text for documents instead of reading content from files. We just read content out of edt.
the same should be done for mime recognition
Comment 12 Jaroslav Tulach 2011-06-16 06:25:17 UTC
(In reply to comment #10)
> the patch. Unfortunately it is not applicable

Of course it is not applicable! That is what I wrote:

(In reply to comment #9)
> Here is a patch that I used to demonstrate that fixing the "getMimeType()" will
> improve overall performance of the explorer...
> ...
> Of course you need to delegate to FileUtil.getMimeType and prevent the system
> to open the streams (which is the cause of the slowness).

the patch just demonstrates all the scrolling problems with remotefs are caused by slow getMimeType.
Comment 13 Alexander Simon 2011-06-17 16:25:06 UTC
Method FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
do not call FileObject.getMIMEType().
It does not allow to override getMIMEType()
Comment 14 Alexander Simon 2011-06-17 16:26:12 UTC
Stack:
org.netbeans.modules.remote.impl.fs.RemotePlainFile.getInputStream(RemotePlainFile.java:149)
org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166)
org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166)
org.openide.filesystems.MIMESupport$CachedFileObject.getInputStream(MIMESupport.java:407)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern.match(MIMEResolverImpl.java:801)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.accept(MIMEResolverImpl.java:1109)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.access$2900(MIMEResolverImpl.java:727)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.resolve(MIMEResolverImpl.java:687)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.access$800(MIMEResolverImpl.java:658)
org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Impl.findMIMEType(MIMEResolverImpl.java:317)
org.openide.filesystems.MIMESupport$CachedFileObject.resolveMIME(MIMESupport.java:376)
org.openide.filesystems.MIMESupport$CachedFileObject.getMIMEType(MIMESupport.java:317)
org.openide.filesystems.MIMESupport.findMIMEType(MIMESupport.java:138)
org.openide.filesystems.FileUtil.getMIMEType(FileUtil.java:1483)
org.openide.loaders.ExtensionList.isRegistered(ExtensionList.java:172)
org.openide.loaders.UniFileLoader.findPrimaryFile(UniFileLoader.java:94)
org.openide.loaders.MultiFileLoader.findPrimaryFileImpl(MultiFileLoader.java:365)
org.openide.loaders.MultiFileLoader.handleFindDataObject(MultiFileLoader.java:99)
org.openide.loaders.DataObjectPool.handleFindDataObject(DataObjectPool.java:161)
org.openide.loaders.DataLoader.findDataObject(DataLoader.java:407)
org.openide.loaders.DataLoader.findDataObject(DataLoader.java:380)
org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:535)
org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:493)
org.openide.loaders.DataObject.find(DataObject.java:523)
org.openide.loaders.FolderChildren.createNode(FolderChildren.java:232)
org.openide.loaders.FolderChildren$DelayedNode.run(FolderChildren.java:424)
org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424)
org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968)
Comment 15 Alexander Simon 2011-06-17 17:34:33 UTC
I see one way to fix method:
FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
by dirty hack:

private boolean isMimeResolving() {
    for(StackTraceElement element : Thread.currentThread().getStackTrace()) {
        if ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName())) {
            return true;
        }
    }
    return false;
}

Jaroslav, what do you think about?
Comment 16 Alexander Simon 2011-06-17 17:55:57 UTC
first implementation of magic cache:
http://hg.netbeans.org/cnd-main/rev/51accf793c88
Comment 17 Quality Engineering 2011-06-18 15:10:14 UTC
Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/51accf793c88
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
Comment 18 Alexander Simon 2011-06-20 09:19:45 UTC
invalidate magic cache:
http://hg.netbeans.org/cnd-main/rev/33b83e7b506e
Comment 19 Alexander Simon 2011-06-20 09:38:26 UTC
read 80 bytes of file:
http://hg.netbeans.org/cnd-main/rev/f3cdd9df1914
Comment 20 Alexander Simon 2011-06-20 10:02:27 UTC
vv159170: Please review fixes.
QA: Please verify fix.
Comment 21 Jaroslav Tulach 2011-06-20 14:08:40 UTC
(In reply to comment #15)
> I see one way to fix method:
> FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes)
> by dirty hack:
> 
> private boolean isMimeResolving() {
>     for(StackTraceElement element : Thread.currentThread().getStackTrace()) {
>         if
> ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName()))
> {
>             return true;
>         }
>     }
>     return false;
> }
> 
> Jaroslav, what do you think about?

I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it then.

Possibly consider enhancing the MIMEResolvers API to directly prevent any content sniffing. Would be an additional boolean attribute to some methods, I guess.
Comment 22 Alexander Simon 2011-06-20 16:09:36 UTC
(In reply to comment #21)
> I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on
> in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it
> then.
Probably you do not behold that fix overrides getMimeType() and uses ThreadLocal variable. Problem in stack #14. Remote file object is not present at stack. So dirty hack is applicable only for this stack. (See dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemotePlainFile.java)

My question is:
Is it possible use another fix for solving problem in stack #14?
Comment 23 Vladimir Kvashin 2011-06-21 09:39:20 UTC
(In reply to comment #22)
> ... 
> Problem in stack #14. Remote file object is not present
> at stack.
> ...
I would restate it as follows (Alexander, please correct me if I;m wrong):
some clients use FileUtil.getMIMEType directly, not FileObject.getMIMEType - in this case thread local does not work.
Comment 24 Quality Engineering 2011-06-21 16:06:14 UTC
Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/33b83e7b506e
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
- add invalidation of magic cache
Comment 25 Alexander Pepin 2011-06-22 10:29:52 UTC
verified in release701_dev build #5647.
Please review the fix and push it into 7.0.1 branch.
Comment 26 Vladimir Voskresensky 2011-06-22 13:52:37 UTC
proposed fix speeds up responsiveness, but introduces regressions in file mime-types recognition. 
Can we increase buffer size to have the same mime-recognition result as in local file system case?
For complete fix of the bug the current change made for 7.1 should be ported as well (which redirects icon calculation in bg thread)
Comment 27 Alexander Simon 2011-06-22 14:39:46 UTC
(In reply to comment #26)
> Can we increase buffer size to have the same mime-recognition result as in
> local file system case?
Of cause we can increase. Question is in a cost of increasing buffer.
Increasing magic buffer from 80 bytes to 4000 bytes:
- time of executing "od" is increased in 3 times (for example /usr/bin: 3 seconds -> 10 seconds).
- disk space (and number of transferred bytes) is increased in 35 times (for example /usr/bin: 0.2 Mb -> 7 Mb).
- java memory requires another strategy (now it stored on hard ref -> should be stored on soft ref).
Are you agree to have such fee for questionable benefit of recognizing rare headers without extension and without magic in first line?
Comment 28 Leonid Lenyashin 2011-06-22 14:45:29 UTC
How about making the buffer size customizable through cli option or in Tools->Options?
Comment 29 Alexander Simon 2011-06-22 16:28:27 UTC
http://hg.netbeans.org/releases/rev/fb8ef63a94d8
- increased size of magic 80->4000
- added exception for fixCookieSet
Comment 30 Jaroslav Tulach 2011-06-23 06:52:50 UTC
(In reply to comment #22)
> Probably you do not behold that fix overrides getMimeType() and uses
> ThreadLocal variable. Problem in stack #14. Remote file object is not present
> at stack. So dirty hack is applicable only for this stack. (See

I have not noticed before, but I see it now.

> My question is:
> Is it possible use another fix for solving problem in stack #14?

I reported bug 199642 to remind me about this problem. For the time being, inspecting the stack is probably the only way.
Comment 31 Alexander Simon 2011-06-23 08:55:04 UTC
http://hg.netbeans.org/releases/rev/3045fefce4b3
- fix get mime type of link remote file object
Comment 32 Alexander Simon 2011-06-23 09:01:15 UTC
Some changes was done.
vv159170: Please review fixes again.
QA: Please verify fix again.
Comment 33 Vladimir Voskresensky 2011-06-23 16:06:44 UTC
as parity tests local vs. remote mime resolving gives 100% same results fix should not introduce regressions.
I agree that it would improve user experience in 7.0.1
Comment 34 Vladimir Kvashin 2011-06-30 11:47:40 UTC
Two additional minor fixes
http://hg.netbeans.org/releases/rev/27e0404ee748
http://hg.netbeans.org/releases/rev/f5efb6b423fa
Comment 35 Alexander Simon 2011-06-30 11:54:02 UTC
Additional fix of java.lang.IllegalThreadStateException: Should never be called from AWT thread:
http://hg.netbeans.org/releases?cmd=changeset;node=d0d6372977e6
Comment 36 dnikitin 2011-06-30 12:26:55 UTC
verified in dev build
Comment 37 Vladimir Kvashin 2011-06-30 13:28:52 UTC
I reviewed all mentioned fixes, they looks correct and safe enough.
I also wrote a test and ran it on my Linux machine against /home, /usr, /bin, /lib, test detected no differences in mime resolving between local and remote file systems.
Comment 38 Alexander Simon 2011-06-30 14:03:34 UTC
Integrated in release701:
51accf793c88 transplanted to 11f14f6f4336
33b83e7b506e transplanted to 1677fa4ef1b0
f3cdd9df1914 transplanted to c1ad674f8d11
fb8ef63a94d8 transplanted to 64b081e52285
3045fefce4b3 transplanted to eca0801a645e
27e0404ee748 transplanted to 4066a35fdd39
f5efb6b423fa transplanted to e84fa045b99b
d0d6372977e6 transplanted to 0a7d3f68a430
Comment 39 Quality Engineering 2011-07-01 14:03:39 UTC
Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/24ec45e9f0f0
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
- increased size of magic 80->4000
- added exception for fixCookieSet
(transplanted from 64b081e52285f572082f18d03ec184a6d3ac5029)
Comment 40 Quality Engineering 2011-07-06 00:01:11 UTC
Integrated into 'releases'
Changeset: http://hg.netbeans.org/releases/rev/11f14f6f4336
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #199136 RemoteFS.getMimeType is slow
Comment 41 dnikitin 2011-07-12 11:49:37 UTC
verified in NetBeans IDE 7.0.1 (Build 201107102202)


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo