This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
When I parse a C++ project (either with "Reparse project" from the code menu or automatically) I often see (by looking at the progress bar) in the middle of the parsing that parsing slows down. If I look in more detail I see that parsing then has dropped back from using ~3 cores to just a single core (I have a quad-core). In the progress bar, it then is typically stuck for a while on a single file. I have a fairly big project: parsing reports > 3000 files analyzed once its finished. I give the JVM plenty of memory (>3G) so that doesn't seem to be the problem. I've even switched between Oracle 1.7, Oracle 1.8 ea and IBM but I usually see the same behavior so it really seems to be a bottleneck in the parsing code itself and not in the JVM runtime, GC ... I cannot provide my source code since it's a proprietary product but if I can assist in another way in diagnosing, let me know.
Very interesting report! It might be related to the moment then parsing is done and we render the objects in just one thread (like linking after compilation). However I'm not sure. What can help I think is to dump threads before the drop and after. http://wiki.netbeans.org/GenerateThreadDump
Please, provide thread dump. Btw, if you are using 7.2 then we changed number of parser threads. In your configuration it will be 2 threads. Btw, giving more than 3Gb for Oracle's jvm is a way to double used memory :-) try -J-Xmx2800m, in this case pointers will be still 32bits long and you can save memory almost twice
fyi, to change number of threads you can use i.e. -J-Dcnd.modelimpl.parser.threads=3
(In reply to comment #2) > Please, provide thread dump. Btw, if you are using 7.2 then we changed number > of parser threads. In your configuration it will be 2 threads. > Btw, giving more than 3Gb for Oracle's jvm is a way to double used memory :-) > try -J-Xmx2800m, in this case pointers will be still 32bits long and you can > save memory almost twice Volodya, User indicated that it started with 3 threads and in the middle of parsing dropped to one. It can not be related to the defaults. And according to report he uses 7.3... although I'm not sure it is correct.
(In reply to comment #4) > (In reply to comment #2) > > Please, provide thread dump. Btw, if you are using 7.2 then we changed number > > of parser threads. In your configuration it will be 2 threads. > > Btw, giving more than 3Gb for Oracle's jvm is a way to double used memory :-) > > try -J-Xmx2800m, in this case pointers will be still 32bits long and you can > > save memory almost twice > > Volodya, > > User indicated that it started with 3 threads and in the middle of parsing > dropped to one. It can not be related to the defaults. > And according to report he uses 7.3... although I'm not sure it is correct. Yes, I'm using 7.3 ea: Product Version: NetBeans IDE Dev (Build 201209100001)
Created attachment 124361 [details] Stack dump while seeing 3 cores used
Created attachment 124362 [details] Stack dump while seeing 1 core used So this is somewhere around the halfway point (according to the progress bar of the parser).
(In reply to comment #2) > Please, provide thread dump. Btw, if you are using 7.2 then we changed number > of parser threads. In your configuration it will be 2 threads. > Btw, giving more than 3Gb for Oracle's jvm is a way to double used memory :-) > try -J-Xmx2800m, in this case pointers will be still 32bits long and you can > save memory almost twice Good tip. I guess compressed references threshold?
(In reply to comment #3) > fyi, to change number of threads you can use i.e. > -J-Dcnd.modelimpl.parser.threads=3 Another good tip! I'll try it with 4 (have 4 cores and while I'm parsing, I can't really do anything particularly useful anyway).
thanks for the dumps. I see what's going on with "Reparse". Please, find in Help->About where is "Cache dir", close IDE, remove cache dir, reopen IDE => IDE will parse project. Do you have the similar issue, if you observe initial parse?
(In reply to comment #10) > thanks for the dumps. I see what's going on with "Reparse". > > Please, find in Help->About where is "Cache dir", close IDE, remove cache dir, > reopen IDE => IDE will parse project. > Do you have the similar issue, if you observe initial parse? - I've switched to the latest build: 201209160001 - Erased the cache dir (netbeans not running) - Started netbeans - It immediately started to parse my C++ project. It did so using 4 cores all the way through. - Then I immediately did a reparse. Parsing ran using 4 cores until 73% of the project. Then fell back to 1 core for half a minute of so. Then it used 4 cores again till the end.
(In reply to comment #11) > - Then I immediately did a reparse. Parsing ran using 4 cores until 73% of the > project. Then fell back to 1 core for half a minute of so. Then it used 4 > cores again till the end. We still need a thread dump on 73% parsing. Now I can only guess that: 1. your project has a one complicated file that includes a lot of headers. Parsing time of the file is long. 2. while IDE parses the file, other threads wait finishing parsing of the file because they need a shared resources such as common preprocessed headers.
(In reply to comment #12) > (In reply to comment #11) > > - Then I immediately did a reparse. Parsing ran using 4 cores until 73% of the > > project. Then fell back to 1 core for half a minute of so. Then it used 4 > > cores again till the end. > We still need a thread dump on 73% parsing. > Now I can only guess that: > 1. your project has a one complicated file that includes a lot of headers. > Parsing time of the file is long. > 2. while IDE parses the file, other threads wait finishing parsing of the file > because they need a shared resources such as common preprocessed headers. I think that "Stack dump while seeing 1 core used" is already what you need. The "3 cores used" is a dump at full speed, "1 cores used" is a dump when I see the load drop back to one processor. Do you want another dump? With regards to 1.: might be. Can I look at some logging info to see how much time is spent on every file?
Alexander, as I previously wrote I see what's going on with reparse: reparse calls DeepReparsingUtils for each file which have to be reparsed from own "Code Model Client Request :csmproject action Reparse Project" task. During that time ppHandler is requested for each file. Then file is put in parser queue. => 4 parser threads are doing parse faster than "reparse project" task prepares ppHandler => drop to 1 core
Just wanted to confirm that the issue still seems present in 201211200002. I'm now using a system with 32 hyperthreads. Explicitly triggering reparsing flies like the wind (using 2000% CPU resources) and then drops back to one thread and then goes full speed again.
I just noticed another thing. I've changed the include paths of the C++ project I'm working on. That triggers a re-parse. This parse is only using 150% CPUs while it could go to 3000% if it wanted to (I've configured the number of parser threads to 30).
(In reply to comment #16) > I just noticed another thing. I've changed the include paths of the C++ > project I'm working on. That triggers a re-parse. This parse is only using > 150% CPUs while it could go to 3000% if it wanted to (I've configured the > number of parser threads to 30). FYI: parsing scalability is not good enough. A code model performance measurement shows following: 1 thread: 15K lines/sec 2 threads: 23K lines/sec (x1.5) or 51% of using additional CPU. 4 threads: 31K lines/sec (x2.0) or 34% of using additional CPUs. Environment: 8 cores computer. Adding more threads does not significant improve parsing speed. Only a heat release is increased. It seems code model parsing cannot achieve x3.5 on any number of threads (CPUs). So advice: do not use more then 4 parsing threads. Code model has a restricted scalability now.
(In reply to comment #17) > (In reply to comment #16) > > I just noticed another thing. I've changed the include paths of the C++ > > project I'm working on. That triggers a re-parse. This parse is only using > > 150% CPUs while it could go to 3000% if it wanted to (I've configured the > > number of parser threads to 30). > FYI: parsing scalability is not good enough. > A code model performance measurement shows following: > 1 thread: 15K lines/sec > 2 threads: 23K lines/sec (x1.5) or 51% of using additional CPU. > 4 threads: 31K lines/sec (x2.0) or 34% of using additional CPUs. > Environment: 8 cores computer. > > Adding more threads does not significant improve parsing speed. Only a heat > release is increased. > It seems code model parsing cannot achieve x3.5 on any number of threads > (CPUs). > So advice: do not use more then 4 parsing threads. Code model has a restricted > scalability now. Interesting. However, I think that something else is going on besides multi-processor scalability issues. When I for example disable and then re-enable content assist on the project, then I see the processor load go up to 2000%-3000%. The parser then goes through the project at very high speed. When modifying the include paths I see the processor load go only to 150%. It looks like in the latter case the parser is provided with a thread pool of 2 threads while in the former, the thread pool has 30 threads. BTW, I've seen the same behavior when I do a modification to a core include file with many dependents: this triggers a lot of parsing but it seems to use only 1 or 2 processors.
Just wanted to confirm that the issue is still there in the 7.3 release. Other than that, working with a large C++ project is going more and more smoothly. Congrats on the good work!