This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
This issue was reported manually by vv159170. It already has 1 duplicates Build: NetBeans IDE 7.0.1 Dev (Build 20110603-4e44b7e2d1d1) VM: Java HotSpot(TM) 64-Bit Server VM, 19.1-b02, Java(TM) SE Runtime Environment, 1.6.0_24-b07 OS: Linux User Comments: vv159170: adding remote folder (from Services) to Favorites vv159170: scrolling Favorites when remote folder is exapnded Maximum slowness yet reported was 27840 ms, average is 17734
Created attachment 108694 [details] nps snapshot
it significantly affects usability of full remote
As far as the "full remote" fs goes, the primary fix is on your side - to improve behavior of yourFileObject.getMimeType(). That would be acceptably low risk fix for 7.0.1, imho. Otherwise I agree, it is desirable to move the file type recognition outside of AWT: ergonomics#8b479b68575f This kind of fix is however not fully functionally compatible (I had to fix Files and Favorites views) to be considered for safe backport. Moreover it seems to me that this fix is not enough anyway. The performance of remotefs still remains poor.
Integrated into 'main-golden', will be available in build *201106051000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/88d091fc8b45 User: Vladimir Voskresensky <vv159170@netbeans.org> Log: help for #199136 - 27s - FolderChildren.createNodes(...) should run on background (transplanted from 460357e755a9a25ec78d01ccdee17d4919e2b4c7)
If I understand correctly, this is going to be an effectively incompatible change for anyone using FilterNode to produce a refined view of a folder tree. Which is a lot of people. Right? If so, I think this fix has to be modified; perhaps only activate it for files which "look" remote (see bug #159628). I do not think I have any similar reports of EQ freezes from the Hudson remote filesystem (hudson/src/org/netbeans/modules/hudson/impl/RemoteFileSystem.java), though bug #198868 in this area seems to have regressed recently.
I've marked our FS as slow and remote http://hg.netbeans.org/releases?cmd=changeset;node=460357e755a9
Causing AE in LibrariesNode, see issue #199308 Incompatible change.
I've reported bug 199391 to track the FolderChildren rewrite. That one is not going to happen for 7.0.x. After discussing this issue with Vladimir, I am assigning it to remotefs to speedup its getMimeType implementation.
Here is a patch that I used to demonstrate that fixing the "getMimeType()" will improve overall performance of the explorer: diff -r b911094529a9 dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java --- a/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java Mon Jun 13 13:59:29 2011 +0200 +++ b/dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemoteFileObjectBase.java Tue Jun 14 13:28:40 2011 +0200 @@ -114,6 +114,11 @@ flags &= ~mask; } } + + @Override + public String getMIMEType() { + return "text/plain"; + } /*package*/ boolean isPendingRemoteDelivery() { return getFlag(BEING_UPLOADED); Of course you need to delegate to FileUtil.getMimeType and prevent the system to open the streams (which is the cause of the slowness).
Jarda, thanks for the patch. Unfortunately it is not applicable, because in that case binary files and header files without extensions will not work at all.
I understand that such approach can improve something, but it's incorrect. i.e. we don't ask editor infrastructure to return empty text for documents instead of reading content from files. We just read content out of edt. the same should be done for mime recognition
(In reply to comment #10) > the patch. Unfortunately it is not applicable Of course it is not applicable! That is what I wrote: (In reply to comment #9) > Here is a patch that I used to demonstrate that fixing the "getMimeType()" will > improve overall performance of the explorer... > ... > Of course you need to delegate to FileUtil.getMimeType and prevent the system > to open the streams (which is the cause of the slowness). the patch just demonstrates all the scrolling problems with remotefs are caused by slow getMimeType.
Method FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes) do not call FileObject.getMIMEType(). It does not allow to override getMIMEType()
Stack: org.netbeans.modules.remote.impl.fs.RemotePlainFile.getInputStream(RemotePlainFile.java:149) org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166) org.netbeans.modules.remote.impl.fs.RemoteLinkBase.getInputStream(RemoteLinkBase.java:166) org.openide.filesystems.MIMESupport$CachedFileObject.getInputStream(MIMESupport.java:407) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern.match(MIMEResolverImpl.java:801) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.accept(MIMEResolverImpl.java:1109) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type.access$2900(MIMEResolverImpl.java:727) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.resolve(MIMEResolverImpl.java:687) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$FileElement.access$800(MIMEResolverImpl.java:658) org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Impl.findMIMEType(MIMEResolverImpl.java:317) org.openide.filesystems.MIMESupport$CachedFileObject.resolveMIME(MIMESupport.java:376) org.openide.filesystems.MIMESupport$CachedFileObject.getMIMEType(MIMESupport.java:317) org.openide.filesystems.MIMESupport.findMIMEType(MIMESupport.java:138) org.openide.filesystems.FileUtil.getMIMEType(FileUtil.java:1483) org.openide.loaders.ExtensionList.isRegistered(ExtensionList.java:172) org.openide.loaders.UniFileLoader.findPrimaryFile(UniFileLoader.java:94) org.openide.loaders.MultiFileLoader.findPrimaryFileImpl(MultiFileLoader.java:365) org.openide.loaders.MultiFileLoader.handleFindDataObject(MultiFileLoader.java:99) org.openide.loaders.DataObjectPool.handleFindDataObject(DataObjectPool.java:161) org.openide.loaders.DataLoader.findDataObject(DataLoader.java:407) org.openide.loaders.DataLoader.findDataObject(DataLoader.java:380) org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:535) org.openide.loaders.DataLoaderPool.findDataObject(DataLoaderPool.java:493) org.openide.loaders.DataObject.find(DataObject.java:523) org.openide.loaders.FolderChildren.createNode(FolderChildren.java:232) org.openide.loaders.FolderChildren$DelayedNode.run(FolderChildren.java:424) org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424) org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968)
I see one way to fix method: FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes) by dirty hack: private boolean isMimeResolving() { for(StackTraceElement element : Thread.currentThread().getStackTrace()) { if ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName())) { return true; } } return false; } Jaroslav, what do you think about?
first implementation of magic cache: http://hg.netbeans.org/cnd-main/rev/51accf793c88
Integrated into 'main-golden' Changeset: http://hg.netbeans.org/main-golden/rev/51accf793c88 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing Bug #199136 RemoteFS.getMimeType is slow
invalidate magic cache: http://hg.netbeans.org/cnd-main/rev/33b83e7b506e
read 80 bytes of file: http://hg.netbeans.org/cnd-main/rev/f3cdd9df1914
vv159170: Please review fixes. QA: Please verify fix.
(In reply to comment #15) > I see one way to fix method: > FileUtil.getMIMEType(FileObject fo, String... withinMIMETypes) > by dirty hack: > > private boolean isMimeResolving() { > for(StackTraceElement element : Thread.currentThread().getStackTrace()) { > if > ("org.netbeans.modules.openide.filesystems.declmime.MIMEResolverImpl$Type$FilePattern".equals(element.getClassName())) > { > return true; > } > } > return false; > } > > Jaroslav, what do you think about? I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it then. Possibly consider enhancing the MIMEResolvers API to directly prevent any content sniffing. Would be an additional boolean attribute to some methods, I guess.
(In reply to comment #21) > I'd say it is dirty hack. Using ThreadLocal is safer and faster. Just set it on > in RemoteFileObjectBase.getMimeType() call to the MIMEResolvers and unset it > then. Probably you do not behold that fix overrides getMimeType() and uses ThreadLocal variable. Problem in stack #14. Remote file object is not present at stack. So dirty hack is applicable only for this stack. (See dlight.remote.impl/src/org/netbeans/modules/remote/impl/fs/RemotePlainFile.java) My question is: Is it possible use another fix for solving problem in stack #14?
(In reply to comment #22) > ... > Problem in stack #14. Remote file object is not present > at stack. > ... I would restate it as follows (Alexander, please correct me if I;m wrong): some clients use FileUtil.getMIMEType directly, not FileObject.getMIMEType - in this case thread local does not work.
Integrated into 'main-golden' Changeset: http://hg.netbeans.org/main-golden/rev/33b83e7b506e User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing Bug #199136 RemoteFS.getMimeType is slow - add invalidation of magic cache
verified in release701_dev build #5647. Please review the fix and push it into 7.0.1 branch.
proposed fix speeds up responsiveness, but introduces regressions in file mime-types recognition. Can we increase buffer size to have the same mime-recognition result as in local file system case? For complete fix of the bug the current change made for 7.1 should be ported as well (which redirects icon calculation in bg thread)
(In reply to comment #26) > Can we increase buffer size to have the same mime-recognition result as in > local file system case? Of cause we can increase. Question is in a cost of increasing buffer. Increasing magic buffer from 80 bytes to 4000 bytes: - time of executing "od" is increased in 3 times (for example /usr/bin: 3 seconds -> 10 seconds). - disk space (and number of transferred bytes) is increased in 35 times (for example /usr/bin: 0.2 Mb -> 7 Mb). - java memory requires another strategy (now it stored on hard ref -> should be stored on soft ref). Are you agree to have such fee for questionable benefit of recognizing rare headers without extension and without magic in first line?
How about making the buffer size customizable through cli option or in Tools->Options?
http://hg.netbeans.org/releases/rev/fb8ef63a94d8 - increased size of magic 80->4000 - added exception for fixCookieSet
(In reply to comment #22) > Probably you do not behold that fix overrides getMimeType() and uses > ThreadLocal variable. Problem in stack #14. Remote file object is not present > at stack. So dirty hack is applicable only for this stack. (See I have not noticed before, but I see it now. > My question is: > Is it possible use another fix for solving problem in stack #14? I reported bug 199642 to remind me about this problem. For the time being, inspecting the stack is probably the only way.
http://hg.netbeans.org/releases/rev/3045fefce4b3 - fix get mime type of link remote file object
Some changes was done. vv159170: Please review fixes again. QA: Please verify fix again.
as parity tests local vs. remote mime resolving gives 100% same results fix should not introduce regressions. I agree that it would improve user experience in 7.0.1
Two additional minor fixes http://hg.netbeans.org/releases/rev/27e0404ee748 http://hg.netbeans.org/releases/rev/f5efb6b423fa
Additional fix of java.lang.IllegalThreadStateException: Should never be called from AWT thread: http://hg.netbeans.org/releases?cmd=changeset;node=d0d6372977e6
verified in dev build
I reviewed all mentioned fixes, they looks correct and safe enough. I also wrote a test and ran it on my Linux machine against /home, /usr, /bin, /lib, test detected no differences in mime resolving between local and remote file systems.
Integrated in release701: 51accf793c88 transplanted to 11f14f6f4336 33b83e7b506e transplanted to 1677fa4ef1b0 f3cdd9df1914 transplanted to c1ad674f8d11 fb8ef63a94d8 transplanted to 64b081e52285 3045fefce4b3 transplanted to eca0801a645e 27e0404ee748 transplanted to 4066a35fdd39 f5efb6b423fa transplanted to e84fa045b99b d0d6372977e6 transplanted to 0a7d3f68a430
Integrated into 'main-golden' Changeset: http://hg.netbeans.org/main-golden/rev/24ec45e9f0f0 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing Bug #199136 RemoteFS.getMimeType is slow - increased size of magic 80->4000 - added exception for fixCookieSet (transplanted from 64b081e52285f572082f18d03ec184a6d3ac5029)
Integrated into 'releases' Changeset: http://hg.netbeans.org/releases/rev/11f14f6f4336 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing Bug #199136 RemoteFS.getMimeType is slow
verified in NetBeans IDE 7.0.1 (Build 201107102202)