This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 197297 - NetBeans is unusable with the Mozilla source code
Summary: NetBeans is unusable with the Mozilla source code
Status: RESOLVED FIXED
Alias: None
Product: cnd
Classification: Unclassified
Component: Code Model (show other bugs)
Version: 7.0.1
Hardware: All All
: P2 normal (vote)
Assignee: Vladimir Voskresensky
URL:
Keywords:
: 199697 202247 (view as bug list)
Depends on: 207544
Blocks: 123872
  Show dependency tree
 
Reported: 2011-03-31 20:33 UTC by jwatt
Modified: 2012-10-05 09:43 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jwatt 2011-03-31 20:33:21 UTC
This is just a flyby bug report created during my search for an IDE that will handle the Mozilla source code nicely. Eclipse handles Mozilla's source pretty well, but it's a poor user interface. NetBeans looks a lot nicer, but I'm sorry to find that it really can't handle Mozilla. The Mozilla source is pretty easy to build nowadays, especially on linux and mac, but if anyone working on NetBeans is interested in working out NetBeans problems on a big project like Mozilla then I'd be happy to help in getting Mozilla set up and in discussing the issues.
Comment 1 soldatov 2011-03-31 22:06:08 UTC
Can you add some additional comments? For me NetBeans has not problems with Firefox. I described my investigation on my site, but on russian language only.

Google translate:
http://translate.google.com/translate?js=n&prev=_t&hl=ru&ie=UTF-8&layout=2&eotf=1&sl=ru&tl=en&u=http%3A%2F%2Fnetbeans.mojgorod.ru%2Fnetbeans_firefox.html
Comment 2 jwatt 2011-04-27 13:10:21 UTC
Hi soldatov. Sorry, I got really busy just after filing this bug and everything else got dropped. I'm still pretty busy, but I'll run through things again in the next couple of weeks and get back here with more details.

I'll probably create separate bugs for some separate issues and mark them as dependencies of this bug so their relationship can be tracked.
Comment 3 jwatt 2011-04-27 21:55:23 UTC
The instructions given in the article linked in comment 1 won't work with current Mozilla code.

Alexander, I'll go through the process of building Mozilla and attempting to get NetBeans working with it again, and I'll take careful notes for this bug, but it would probably be a good idea to do all that on the same platform as you if you're going to be working on this bug. I can build on Mac 10.6, Ubuntu, or Windows 7. What would your preference be? (It's a bit of a pain to build on Windows, so I'd rather try on one of the other two if possible, but I'll do it on Windows in you prefer.)
Comment 4 Alexander Simon 2011-04-28 12:38:54 UTC
I built mozilla browser 4 on Ubuntu.
You are right the mozilla browser is a challenging project for NB 7.0.
Comment 5 Leonid Lenyashin 2011-06-16 10:15:47 UTC
Alexander, what is so special about Mozilla? How many of our big customers may face the same issue? How come Eclipse is doing better in this case? Isn't it a P2?
Comment 6 Alexander Simon 2011-06-16 17:52:13 UTC
jwatt,

Could you clarify:
- How did you create NB project?
- What was the biggest problem in NetBeans?
- Are you going to file separate bugs? (or you gave up NetBeans)
- What does Eclipse handle better then NetBeans?

Your answers (experience) help us to improve next NetBeans release. 

Thanks,
Alexander
Comment 7 jwatt 2011-06-21 12:56:30 UTC
Hi. No, no, I've not given up on NetBeans. I've just not yet made the time to reconstruct the steps I took last time and write things up. Most likely I'm not going to manage to do that this evening, but I'll block out tomorrow evening for that and get back to you then.
Comment 8 jwatt 2011-06-23 22:57:10 UTC
I've created a wiki page here:

https://developer.mozilla.org/En/Developer_Guide/Using_NetBeans_with_Mozilla

Hopefully that's enough to start with for any of you guys that are interested to reproduce the problems, and hopefully figure out how to improve things.
Comment 9 soldatov 2011-06-24 07:08:16 UTC
Can you specify Firefox build and your hardware? Can I use Firefox 4.0.1?
Comment 10 jwatt 2011-06-24 07:55:22 UTC
I'm building tip (HEAD) from mozilla-central - so the code that will become Firefox 7. (That's the code I'm interested in seeing working, since that's what developers are working with nowadays.) I get the "NetBeans is ususably unresponsive" problems on both a MacBook Pro with OS X 10.6 and Thinkpad with Ubuntu 11.04 installed. Both have 8 GB RAM and fast processors, so I'm pretty sure if you follow the instructions as given on the wiki page you'll be able to reproduce the problems on any Mac or Ubuntu machines. Solaris maybe not, but I'd suspect so there to.
Comment 11 jwatt 2011-06-24 07:59:47 UTC
I really, really wouldn't recommend building Firefox 4.0.1 for this investigation, but if that's what you really want to do for some reason, the best way to do it depends on where you're at. If you already have a checkout of Mozilla central, then you can pull the changesets from the release 2.0 (Firefox 4.0) repository into your mozilla-central repo:

  hg pull http://hg.mozilla.org/releases/mozilla-2.0/

Or if you don't have mozilla-central, then you can just clone that entire release repo:

  hg clone http://hg.mozilla.org/releases/mozilla-2.0/

You can then get a working copy of the 4.0.1 source by checking out the 4.0.1 release tag using:

  hg co -r FIREFOX_4_0_1_RELEASE

Another way to get the 4.0.1 source is by downloading the .tar.bz source snapshot from the applicable releases 'source' directory from ftp.mozilla.org as indicated in the wiki page.

I'd warn that the build instructions for 4.0 are probably a bit different to mozilla-central though, so expect to have to change things a bit. (Also even if 4.0.1 happens to work okay in NetBeans roughly following the wiki instructions (actually that's doubtful), that's not really all that helpful since it's now a dead branch.)
Comment 12 Vladimir Voskresensky 2011-06-24 08:29:22 UTC
I think memory is problem of slow parsing. I had to switch to 32bit JVM and give it -J-Jxm2G to be able to be parsed withing 200 sec.
Comment 13 Vladimir Voskresensky 2011-06-24 08:29:48 UTC
What I did originally:
preparing firefox for Netbeans on Ubuntu. I use the latest 7.0.1 Netbeans bits.

#sudo apt-get build-dep firefox
#sudo apt-get install mercurial libasound2-dev libcurl4-openssl-dev libnotify-dev libxt-dev libiw-dev mesa-common-dev autoconf2.13 yasm

#hg clone http://hg.mozilla.org/mozilla-central/

#cd mozilla-central

created file .mozconfig in mozilla-central folder with the content:
-----------

. $topsrcdir/browser/config/mozconfig

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-debug

#I don't have the newest yasm, so disabled some modules as
ac_add_options --disable-webm
ac_add_options --disable-libjpeg-turbo

#use parallel build
mk_add_options MOZ_MAKE_FLAGS="-j 8"
------------

Have *not* issued build from command line, but instead prepared project first as:
started Netbeans with -J-Xmx2G

   1. Click the New Project button, select "C/C++ Project with Existing Sources", then click "Next".
   2. Specify mozilla-central as the root source directory, select the configuration mode "Custom", then click "Next".
   3. Specify "Using an existing makefile" and point to mozilla-central/client.mk. Unselect the "Clean and Build after Finish" option, and click "Next".
   4. Leave everything as is ("Build Command" to be "make -f client.mk", the "Clean Command" to be "make -f client.mk clean"), then click "Next".
   5. Leave everything as is and click Next
   6. Leave "Automatic Configuration" and click "Next".
   7. Change "Project Location" path to be out of sources, i.e. /home/user/NetBeansProjects
   8. press Finish

When project is opened in Project tab => right click on project and turn off 'Code Assistance -> C/C++ Code Assistance' to save time on parsing not yet configured project.

Now build firefox directly from IDE:
btw, to speed up I specified parallel build using "-j" in .mozconfig and special compiler flags are not needed if building directly from IDE using Clean & Build actions.
So, invoked Clean & Build from project context menu and after finished build (it took about 25 min on my laptop ).
After that progress did not stopped and I saw a lot of exceptions in console[1], but when it stopped => reviewing properties of source files I can see that IDE configured all options used during build.
After that I closed IDE and restarted it to eliminate possible memory leaks occurred during build process and following configuration of code assistance.
I have 64 bits jvm => started with -J-Xmx4G
When project is opened waited more till CPU gone to 0% (exceptions like [1] again in console)
Then turned on Code Assistance from context menu and it parsed about 200 secs on my laptop (with cpu time as 20 min). IDE ate 2.7Gb
Running IDE for parse with 2G caused stop of parsing at about 80% and almost not going further.
Comment 14 jwatt 2011-06-24 09:43:17 UTC
(In reply to comment #13)
> created file .mozconfig in mozilla-central folder with the content:

Not using the debug options is kinda cheating, since that will reduce the amount of memory NetBeans will have to use. ;)

> started Netbeans with -J-Xmx2G

I just tried this, and it made a big, big difference, even with my debug Firefox build. NetBeans flew through the first 70% of parsing probably roughly the same sort of speed that Eclipse does, although it did seem to slow down after that. In the end it took about 8 minutes to finish, which is a huge improvement. Scrolling performance is now also no longer a problem.

That said, I'm still having responsiveness problems with some things.

* The "Go to File" dialog takes about 2-3 seconds searching for "*SVGFrame" for example (Eclipse's "Open Resource" dialog is instantaneous), and typing seems to freeze as you're typing in.

* The "Go to Symbol" dialog seems clunky, since as I try to type in "BuildDisplayList" the typing freezes repeatedly. The search results appear no slower than in Eclipse's "C/C++ Search" though, so that's good (well, still room for improvement in both IDEs there ;) ).

I'm out of time to experiment with this further today, but I'll report back on any further responsiveness problems.

I did notice another problem with correctness:

* When I open file nsFrame.cpp, select the symbol "GetStyleDisplay" and from the context menu select "Go To Declaration" or "Go To Implementation", NetBeans seems to completely make up a result because I end up in nsIFrame.h line 142, which is utterly wrong.

> btw, to speed up I specified parallel build using "-j" in .mozconfig and
> special compiler flags are not needed if building directly from IDE using
> Clean & Build actions.
> After that progress did not stopped and I saw a lot of exceptions in
> console[1], but when it stopped => reviewing properties of source files I can
> see that IDE configured all options used during build.

I think you just got lucky with the files you checked the properties on. If you use -j, then the build output from the various make processes gets jumbled and build output parsers will not be able to make complete sense of all of the combined build output. Even if (a big if) all the individual lines from the various make processes remain intact in the combined build output (i.e. without lines from one make process interrupting in the middle of lines from other processes), and even if you use -w to get the "Entering/Leaving" lines, the interleaving of "Entering/Leaving" lines from various make processes will mean the closest "Entering" line proceeding a gcc line building a source file will not always be the correct one. In this case the build output parser will not necessarily be able to know which source file the gcc line is actually building, or from which CWD, preventing it from always correctly resolving relative include paths specified on the command line, for example. This problem is further compounded by the fact that Mozilla has plenty of source files with the same name in various parts of the source tree, and some source files (particularly under the 'nss' directory) are not specified using absolute paths.

In short I think avoiding -j is a good idea while trying to work though the problems NetBeans has with Mozilla, since it avoids one source of bogus problems.

> Then turned on Code Assistance from context menu and it parsed about 200 secs
> on my laptop (with cpu time as 20 min). IDE ate 2.7Gb
> Running IDE for parse with 2G caused stop of parsing at about 80% and almost
> not going further.

Hmm, although things slowed down for me at about 70-80%, they didn't stop.

[Side rant - why does Java even require memory to be pre-allocated? I don't see why it can't do what all modern software does and dynamically allocate more!]
Comment 15 soldatov 2011-06-24 09:57:51 UTC
(In reply to comment #14)
> [Side rant - why does Java even require memory to be pre-allocated? I don't see
> why it can't do what all modern software does and dynamically allocate more!]
I think aggressive garbage collector is a Achilles' heel of Java
Comment 16 Vladimir Voskresensky 2011-06-24 15:55:48 UTC
(In reply to comment #14)
> 
> Not using the debug options is kinda cheating, since that will reduce the
> amount of memory NetBeans will have to use. ;)
[skipped]
> 
> I think you just got lucky with the files you checked the properties on. If you
> use -j, then the build output from the various make processes gets jumbled and
> build output parsers will not be able to make complete sense of all of the
> combined build output. Even if (a big if) all the individual lines from the
> various make processes remain intact in the combined build output (i.e. without
> lines from one make process interrupting in the middle of lines from other
> processes), and even if you use -w to get the "Entering/Leaving" lines, the
> interleaving of "Entering/Leaving" lines from various make processes will mean
> the closest "Entering" line proceeding a gcc line building a source file will
> not always be the correct one. In this case the build output parser will not
> necessarily be able to know which source file the gcc line is actually
> building, or from which CWD, preventing it from always correctly resolving
> relative include paths specified on the command line, for example. This problem
> is further compounded by the fact that Mozilla has plenty of source files with
> the same name in various parts of the source tree, and some source files
> (particularly under the 'nss' directory) are not specified using absolute
> paths.
> 
> In short I think avoiding -j is a good idea while trying to work though the
> problems NetBeans has with Mozilla, since it avoids one source of bogus
> problems.
Thanks for your explanations, potential issues and good understanding of IDE internals :-)
Fortunately I'm still correct - starting from upcoming 7.0.1 you don't have to specify any special option, because Netbeans has differentiating feature which you can see in Tools->Options->C++->Project Options
It is "Use smart build analyzer to configure code assistance" (in 7.0 it is off by default)
It does all the magic when Clean&Build is called from ide and handles all potential issues you've mentioned above :-)
Comment 17 Vladimir Voskresensky 2011-06-24 15:58:58 UTC
(In reply to comment #14)
> 
> > started Netbeans with -J-Xmx2G
> 
> I just tried this, and it made a big, big difference, even with my debug
> Firefox build. NetBeans flew through the first 70% of parsing probably roughly
> the same sort of speed that Eclipse does, although it did seem to slow down
> after that. In the end it took about 8 minutes to finish, which is a huge
> improvement. 
What about bitness of your java?
If it is 64bit JVM, can you, please, try -J-Xmx4G?

Anyway I recommend to use 32bit JVM which uses 2 time less memory (due to pointers size) without any real performance penalty on NB
Comment 18 jwatt 2011-06-24 18:17:33 UTC
(In reply to comment #16)
> Fortunately I'm still correct - starting from upcoming 7.0.1 you don't have to
> specify any special option, because Netbeans has differentiating feature which
> you can see in Tools->Options->C++->Project Options
> It is "Use smart build analyzer to configure code assistance" (in 7.0 it is off
> by default)
> It does all the magic when Clean&Build is called from ide and handles all
> potential issues you've mentioned above :-)

Intriguing. I'd be really interested to know more about how it manages to handle the problems I mentioned. Maybe that would be better talked about in private email off this bug though. :)

(In reply to comment #17)
> What about bitness of your java?

32-bit.
Comment 19 jwatt 2012-01-20 00:16:39 UTC
(In reply to comment #16)
> Thanks for your explanations, potential issues and good understanding of IDE
> internals :-)
> Fortunately I'm still correct - starting from upcoming 7.0.1 you don't have to
> specify any special option, because Netbeans has differentiating feature which
> you can see in Tools->Options->C++->Project Options
> It is "Use smart build analyzer to configure code assistance" (in 7.0 it is off
> by default)
> It does all the magic when Clean&Build is called from ide and handles all
> potential issues you've mentioned above :-)

Is the smart build analyzer still part of NetBeans 7.1, and still turned on by default? I can't find any option to turn it on/off in the IDE (Tools > Options doesn't exist any more, and the Configure Code Assistance Wizard doesn't mention it in its "Select analyzer" dropdown), and if I build like normal a lot of the options the compiler is invoked with are missing.
Comment 20 soldatov 2012-01-20 07:13:37 UTC
(In reply to comment #19)
> Tools > Options doesn't exist any more
On MacOSX Tools|Options == NetBeans|Preferencies... for a lot of times
Comment 21 jwatt 2012-01-20 09:29:43 UTC
Oops. Thank you!
Comment 23 Quality Engineering 2012-03-17 10:39:16 UTC
Integrated into 'main-golden', will be available in build *201203170400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/ad794a7c7a0e
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: fixing #197297 - NetBeans is unusable with the Mozilla source code
- reuse field when only one macro is defined/undefined in snapshot, no need for collection
Comment 24 Quality Engineering 2012-03-20 12:52:58 UTC
Integrated into 'main-golden', will be available in build *201203200400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/67181412c942
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: fixing #197297 - NetBeans is unusable with the Mozilla source code
- generate empty macro replacement tokens by request only
Comment 25 Vladimir Voskresensky 2012-04-03 18:41:55 UTC
*** Bug 199697 has been marked as a duplicate of this bug. ***
Comment 26 Vladimir Voskresensky 2012-04-03 18:42:03 UTC
*** Bug 202247 has been marked as a duplicate of this bug. ***
Comment 28 jwatt 2012-07-31 18:48:18 UTC
Hi Vladimir. Sounds great! I don't know anything about non-official builds of NetBeans, but if you can let me know when and where I can find builds with these fixes I'll happily test them out on the Mozilla source.
Comment 29 Vladimir Voskresensky 2012-07-31 20:32:39 UTC
try bits from
http://bits.netbeans.org/dev/nightly/latest

btw if you  use oracle's jdk 6/7 then 2Gb heap is enough even on 64bit version
Comment 30 Vladimir Voskresensky 2012-08-15 02:25:55 UTC
The recent work reduced memory consumption to 750-900Mb, so now mozilla works fine with -J-Xmx1G heap and no any tricks with 32/64 modes are needed.
Comment 31 Vladimir Voskresensky 2012-08-15 04:19:14 UTC
of course -J-Xmx2G can speed up intensive use of Find Usages, because caches will live longer comparing to -J-Xmx1G
Comment 32 bgirard 2012-10-05 04:35:30 UTC
Thanks for your work Vladimir. I've just tried the nightly netbeans build and so far it's by far the best I've seen something handle the mozilla code base.

Is there a way to use a header for macro definitions? We use -include <obj>/dist/include/mozilla-config.h which contains ~100 #define I don't want to maintain manually.
Comment 33 Vladimir Voskresensky 2012-10-05 09:29:10 UTC
(In reply to comment #32)
> Thanks for your work Vladimir. I've just tried the nightly netbeans build and
> so far it's by far the best I've seen something handle the mozilla code base.
This is great. Btw, what is your Xmx limit you use now? Our measurements shows that after the last changes -J-Xmx1G should be enough, but the confirmation from real user is very welcomed.

> 
> Is there a way to use a header for macro definitions? We use -include
> <obj>/dist/include/mozilla-config.h which contains ~100 #define I don't want to
> maintain manually.
If you clean&build your project just at least once from IDE => It will auto detect everyting and "-include" as well. No need to do anything manually. 
See nbproject/private/Default.properties file to realize the trick.

Btw, after that you'll have for source files Compile File (F9) in editor calling exactly what is supposed to be called for this compilation unit if you'd like just to check compile errors.
Comment 34 Vladimir Voskresensky 2012-10-05 09:43:08 UTC
Just to record here the simplest way to put Mozilla into IDE:
It could be like:
1) adjust .mozconfig content after checkout
2) no need to build from command line after 1)
3) run NB and use wizard File->New Project->C/C++->Project from existing sources.
on the first step of wizard I specified top level source dir and switch to Custom mode
on the second I specified client.mk as make file
on the third I change clean command to be
     rm -rf obj-debug
then next, next, next, finish and firefox is built and parsed.