Bug 268773 - Building pkg-config on enum takes very long time
Building pkg-config on enum takes very long time
Status: VERIFIED FIXED
Product: cnd
Classification: Unclassified
Component: Project
8.2
All All
: P3 (vote)
: 8.2
Assigned To: Alexander Simon
issues@cnd
82patch1-verified
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-11-01 17:24 UTC by Vladimir Kvashin
Modified: 2016-12-13 10:16 UTC (History)
0 users

See Also:
Issue Type: DEFECT
:


Attachments
proposed patch (5.00 KB, patch)
2016-11-02 17:28 UTC, Alexander Simon
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vladimir Kvashin 2016-11-01 17:24:27 UTC
Investigation shows that /home/kvashin/... is accessed on remote many times.
Investigation shows that it is accessed by line converter when processing errors in the build output.
Comment 1 Vladimir Kvashin 2016-11-01 19:59:16 UTC
More details. I set up a remote host (enum) using a local user. I used SFTP transport. My pkg-config sources were on my local disk. I created a project from existing sources with default options. There was long delays when building the project.

Investigation showed that 
1) There were lots of access to /home/vkvashin while sources were copied to /home/remote_test_vk/.netbeans/remote/.... and there were no  /home/vkvashin directory at all.
2) This was because of an error parser that tried to resolve short remote file names against incorrect /home/vkvashin directory (note that remote user was remote_test_vk, not vkvashin)
Comment 2 Vladimir Kvashin 2016-11-01 21:29:28 UTC
Alexander just pushed a (probably partial) fix a96955bc1ff5 into enum/release82
Comment 3 Alexander Simon 2016-11-02 11:44:21 UTC
Root causes of slowness are:
- a lot of dead process on remote machine like to:
20505 1 0 Jun 28 ? 0:00 /var/tmp/dlight_tester/40ff781/0801276374/pty --dir
- a lot of garbage in /tmp like to:
tmp.ZGaGks
It are artifacts of killed (on unexpected finishing remote tests/terminals/projects)

Of cause accessing to home adds slowness too.
Comment 4 Alexander Simon 2016-11-02 13:15:28 UTC
Also slowness is result of mounting dead (or very slow) file systems.
Comment 5 Vladimir Kvashin 2016-11-02 14:02:59 UTC
I noticed the following. I used cnd built on change set

changeset:   312687:5e87ea9e5c55
branch:      release82
tag:         tip
user:        Alexander Simon <alexvsimon@netbeans.org>
date:        Tue Nov 01 18:30:56 2016 +0300
summary:     fixing Bug #268766 NullPointerException at org.netbeans.modules.cnd.discovery.buildsupport.ToolsWrapperUtility.fixScript

I had local pkg-config project and built it on enum with SFTP sync.
The following stack accesses /home/vkvashin/...

"pool-3-thread-1"
	at org.netbeans.modules.remote.impl.fs.server.FSSResponse.createTimeoutException(FSSResponse.java:179)
	at org.netbeans.modules.remote.impl.fs.server.FSSResponse.getNextPackage(FSSResponse.java:163)
	at org.netbeans.modules.remote.impl.fs.server.FSSTransport.stat_or_lstat(FSSTransport.java:195)
	at org.netbeans.modules.remote.impl.fs.server.FSSTransport.lstat(FSSTransport.java:165)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystemTransport.lstat(RemoteFileSystemTransport.java:193)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getSpecialDirChildEntry(RemoteDirectory.java:681)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.readEntries(RemoteDirectory.java:656)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getDirectoryStorageImpl(RemoteDirectory.java:1120)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getDirectoryStorage(RemoteDirectory.java:533)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getFileObject(RemoteDirectory.java:410)
	at org.netbeans.modules.remote.impl.fs.RemoteFileObject.getFileObject(RemoteFileObject.java:421)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getFileObject(RemoteDirectory.java:402)
	at org.netbeans.modules.remote.impl.fs.RemoteFileObject.getFileObject(RemoteFileObject.java:421)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:503)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getFileObject(RemoteDirectory.java:400)
	at org.netbeans.modules.remote.impl.fs.RemoteFileObject.getFileObject(RemoteFileObject.java:421)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:503)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getFileObject(RemoteDirectory.java:400)
	at org.netbeans.modules.remote.impl.fs.RemoteFileObject.getFileObject(RemoteFileObject.java:421)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:503)
	at org.netbeans.modules.remote.impl.fs.RemoteDirectory.getFileObject(RemoteDirectory.java:400)
	at org.netbeans.modules.remote.impl.fs.RemoteFileObject.getFileObject(RemoteFileObject.java:421)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:503)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:496)
	at org.netbeans.modules.remote.impl.fs.RemoteFileSystem.findResource(RemoteFileSystem.java:110)
	at org.netbeans.modules.remote.spi.FileSystemProvider.getFileObject(FileSystemProvider.java:239)
	at org.netbeans.modules.cnd.toolchain.execution.ErrorParser.resolveFile(ErrorParser.java:125)
	at org.netbeans.modules.cnd.toolchain.execution.GCCErrorParser.handleLine(GCCErrorParser.java:261)
	at org.netbeans.modules.cnd.toolchain.execution.GCCErrorParser.handleLine(GCCErrorParser.java:182)
	at org.netbeans.modules.cnd.spi.toolchain.CompilerLineConvertor.handleLine(CompilerLineConvertor.java:139)
	at org.netbeans.modules.cnd.spi.toolchain.CompilerLineConvertor.convert(CompilerLineConvertor.java:112)
	at org.netbeans.modules.cnd.makeproject.api.DefaultProjectActionHandler$ProcessChangeListener.convert(DefaultProjectActionHandler.java:544)
	at org.netbeans.modules.cnd.makeproject.api.DefaultProjectActionHandler$ProcessChangeListener$$Lambda$89.1899107601.convert
	at org.netbeans.api.extexecution.print.LineProcessors$PrintingLineProcessor.processLine(LineProcessors.java:148)
	at org.netbeans.api.extexecution.base.input.InputProcessors$Bridge.processInput(InputProcessors.java:181)
	at org.netbeans.modules.extexecution.base.input.DefaultInputReader.readInput(DefaultInputReader.java:104)
	at org.netbeans.api.extexecution.base.input.InputReaderTask.run(InputReaderTask.java:190)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Comment 6 Alexander Simon 2016-11-02 17:28:29 UTC
Created attachment 162732 [details]
proposed patch

Vladimir, please review the patch
Comment 7 Alexander Simon 2016-11-03 15:13:47 UTC
fixed in enum, branch release82, change set bfd94271d20c
Comment 8 Quality Engineering 2016-11-10 03:22:14 UTC
Integrated into 'main-silver', will be available in build *201611100001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/f50b9f5ff70c
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing Bug #268773 Building pkg-config on enum takes very long time
(transplanted from bfd94271d20cb33fa45c50ef19cb6163b28a005c)
Comment 9 soldatov 2016-12-13 10:16:41 UTC
Verified in NetBeans IDE 8.2 (Build 201612122312)

Main problem: Oracle's homes extremely slow! Really "extremely".


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo