This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 198273 - Slow getRequiredElements prevent update after update of autoupdate
Summary: Slow getRequiredElements prevent update after update of autoupdate
Status: VERIFIED FIXED
Alias: None
Product: platform
Classification: Unclassified
Component: Autoupdate (show other bugs)
Version: 7.0
Hardware: PC Windows XP
: P1 normal (vote)
Assignee: Jaroslav Tulach
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-03 09:22 UTC by Peter Nabbefeld
Modified: 2011-06-14 04:58 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
Here's now a profiler snapshot (28.25 KB, application/octet-stream)
2011-05-16 08:50 UTC, Peter Nabbefeld
Details
New profiler snapshot (100.05 KB, application/octet-stream)
2011-05-20 06:21 UTC, Peter Nabbefeld
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Nabbefeld 2011-05-03 09:22:10 UTC
I've tried to use NB7.0 (as daily build is already corrupted) und updated the autoupdate modules - now NB does no autoupdate anymore. If I cancel the plugin manager after several minutes, I can get NB to work again, after some more minutes it blocks even other programms. Probably, after some more time it would give me an OOME (as the behaviour is typical for this), have no time for waiting.

Cannot manage any plugins because of this, as autoupdate does not even show the list of installed plugins.
Comment 1 Tomas Danek 2011-05-03 11:08:43 UTC
how did you do update of autoupdate? in stable release 7.0 should be right now no updates available, and as i checked in my 7.0 there aren't any? 
Did you tweak update center URLs?
Comment 2 Peter Nabbefeld 2011-05-03 11:19:19 UTC
(In reply to comment #1)
> how did you do update of autoupdate? in stable release 7.0 should be right now
> no updates available, and as i checked in my 7.0 there aren't any? 
> Did you tweak update center URLs?

Yes, I'm always adding the URL for latest community updates, as I'm missing some modules otherwise (e.g. "Open in System", which is important for me). But Autoupdate services 1.26/Autoupdate UI 1.23 seems to be buggy. AU is a significant feature of NB, so it should be tested seriously.
Comment 3 Tomas Danek 2011-05-03 11:31:16 UTC
which URL do you add? 
updating of standard set of netbeans modules from any other that official update center is not supported scenario.
These are modules you should have in your 7.0, and those should work properly.
org.netbeans.modules.autoupdate.ui [1.22.1 201104080000]
org.netbeans.modules.autoupdate.services [1.25.1 201104080000]
Comment 4 Peter Nabbefeld 2011-05-03 11:39:07 UTC
(In reply to comment #3)
> which URL do you add? 
> updating of standard set of netbeans modules from any other that official
> update center is not supported scenario.

The relevant one:
http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz
Comment 5 Tomas Danek 2011-05-03 13:00:53 UTC
> The relevant one:
> http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz

these are bits for daily build, i suggest you not to mix stable modules, with dev ones. You can reinstall 7.0, install plugins you need, but do not update NetBeans itself with modules from this UC.
Comment 6 Peter Nabbefeld 2011-05-03 13:04:30 UTC
(In reply to comment #5)
> > The relevant one:
> > http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz
> 
> these are bits for daily build, i suggest you not to mix stable modules, with
> dev ones. You can reinstall 7.0, install plugins you need, but do not update
> NetBeans itself with modules from this UC.

What are Updates good for, if I may not use them??? Especially, I'd expect AU to work without problems, as AU is very basic for the IDE. AU modules should never corrupt any build, wether stable or daily.
Comment 7 dlipin 2011-05-03 13:13:27 UTC
Peter,

If your NB failed with OOM, there should be heap dump created at ${userdir}/var/log/heapdump.hprof. Please upload it somewhere (depositfiles, rapidshare, etc) and post the link here.

If you NB got stuck, please do some some subsequent thread dumps, and attach those to the issue. 
http://wiki.netbeans.org/GenerateThreadDump

> as autoupdate does not even show the list of installed plugins.
Do you run on JDK7, btw?
Comment 8 Peter Nabbefeld 2011-05-03 13:27:33 UTC
(In reply to comment #7)
> ...
> Do you run on JDK7, btw?
Yes.
Comment 9 Tomas Danek 2011-05-03 16:04:19 UTC
> What are Updates good for, if I may not use them??? Especially, I'd expect AU
> to work without problems, as AU is very basic for the IDE. AU modules should
> never corrupt any build, wether stable or daily.
You are not doing update in terms what update is, you are putting dev modules into stable release (which is not supported scenario) using autoupdate mechanism. Official updates will appear on NetBeans Distribution UC, stable release (e.g.) should not be updated from URL.
Comment 10 Peter Nabbefeld 2011-05-06 08:43:37 UTC
I've also got the following problem with current AU:
1. Added a module created by my own -> NB tears in 64 outdated modules
   (obviously related to outdated dependencies).
2. Update takes about 3,5 hours.
3. After installing all those modules, I had to update them again.
4. Took again about 3,5 hours.
All this on a modern PC:
WinXP SP3
AMD Athlon 64 X2 Dual Core 6000+, 3GHz 3GB
That is far too long! Time seems to increase exponential.

What might cause those problems?
1. Don't know, how old this note is, probably might be helpful:
   Note: the former Autoupdate module allows declaration of former
   <code>AutoupdateType</code> on XML layer, these declaration are
   read as new one UpdateProvider by reason of backward compatability.
   (from UpdateProvider javadoc)
2. LocalNBMsProvider seems to be broken (as it tears in outdated deps).
3. Didn't have time enough to investigate, but it seems dependency
   checking is done online, even using multiple requests, producing
   heavy load on the update servers (which I guess You won't like),
   and needing establishing and quitting connections all the time.
   Probably I'll have some time to look at it this week-end.
However, it seems there've been made serious changes to AU, and
time needed for some updates grows from one release to the next,
so PLEASE CHECK!
Comment 11 Jaroslav Tulach 2011-05-09 08:55:50 UTC
Guys, please learn how to profile your IDE! Without it you risk, I close the problem as worksforme. http://wiki.netbeans.org/FitnessViaPartnership
Comment 12 Jaroslav Tulach 2011-05-09 08:58:42 UTC
I've seen some problem myself and speed it up in ergonomics#3352065e797f

Marking fixed. As soon as you deliver profiler snapshots, feel free to reopen.
Comment 13 Peter Nabbefeld 2011-05-09 09:03:04 UTC
(In reply to comment #11)
> Guys, please learn how to profile your IDE! Without it you risk, I close the
> problem as worksforme. http://wiki.netbeans.org/FitnessViaPartnership

Hm, is it still possible to catch useful profiling data *after* collecting the dependencies? If there're 64 updates collected after 3,5 hours, You'll probably understand I'll not restart it. Dependency checking seems (currently) to need an exponentialoly increasing amount of time.
Comment 14 Peter Nabbefeld 2011-05-09 09:03:52 UTC
(In reply to comment #12)
> I've seen some problem myself and speed it up in ergonomics#3352065e797f
> 
> Marking fixed. As soon as you deliver profiler snapshots, feel free to reopen.

Just for interest: Could You provide a link to the changeset, please?
Comment 15 Quality Engineering 2011-05-11 19:41:33 UTC
Integrated into 'main-golden', will be available in build *201105111436* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/3352065e797f
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #198273: Don't try to load other DTDs (like something from W3C) than those applicable to autoupdate
Comment 16 Peter Nabbefeld 2011-05-16 08:50:39 UTC
Created attachment 108310 [details]
Here's now a profiler snapshot
Comment 17 Peter Nabbefeld 2011-05-16 08:51:31 UTC
Added profiler snapshot. Hopefully You'll now be able to fix it.
Comment 18 Peter Nabbefeld 2011-05-16 08:59:35 UTC
Just noted sth.:
After closing the plugins dialog, NB used still about 99% cpu time! Probably there's an infinite loop starting after some error occurs (e.g. because of an invalid/outdated UC link)?
Comment 19 Jaroslav Tulach 2011-05-16 13:43:59 UTC
5 calls to org.netbeans.modules.autoupdate.services.Utilities.handleBackwardCompatability()
result in 60 calls to org.netbeans.modules.autoupdate.services.Utilities.getModuleInstance() 
which triggers 801 calls to 
org.openide.util.Lookup.lookupAll()

With a bit of refactoring, one call should be enough.
Comment 20 Peter Nabbefeld 2011-05-16 14:06:38 UTC
NB tells me, it wants to update 66 modules, so I should have enough data for testing.
Comment 21 Jaroslav Tulach 2011-05-17 11:48:27 UTC
ergonomics#d7e522762f6f
Comment 22 Quality Engineering 2011-05-18 08:50:37 UTC
Integrated into 'main-golden', will be available in build *201105180400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/d7e522762f6f
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #198273: Cache cnb -> ModuleInfo mapping
Comment 23 Peter Nabbefeld 2011-05-19 08:17:39 UTC
Spec version seem not to be upgraded, so AU upgrade does not work!
Comment 24 Jaroslav Tulach 2011-05-19 15:28:21 UTC
True, I have not bump up spec version of AutoUpdate. We are not used to do that during development cycle.
Comment 25 Peter Nabbefeld 2011-05-20 05:52:03 UTC
I do understand, that spec upgrade is not needed in every case.

IMHO, it should be done in the following cases:
- Important user impact (I've tried AU services by fetching whole NB sources, upgrading AU service spec version myself to 1.27.2, und moving the nbm file into upgrade folder: Upgrade took only about ten minutes compoared to 3,5 hours).
- Incompatible API changes.
- Incompatible dependencies changes.
Comment 26 Peter Nabbefeld 2011-05-20 06:21:54 UTC
Created attachment 108406 [details]
New profiler snapshot

Most of the time seems now to be spent for waiting - probably this could be improved?
Comment 27 Jaroslav Tulach 2011-06-07 08:07:10 UTC
Verify the fix, please.
Comment 28 Peter Nabbefeld 2011-06-07 08:21:27 UTC
(In reply to comment #27)
> Verify the fix, please.

See comment #25: "Upgrade took only about ten minutes compared to 3,5
hours" - that's been about 4% of the time it took before!
So, yes, the supplied fix saves a lot amount of time.

Please, see also comment #26: Much time seems to be wasted for waiting - could You please take a look at it? If possible, this should be fixed.

Setting state to "verified", because most important performance blocker has been fixed, while it would be great to get also the waiting issue (#26) optimized.
Comment 29 Jaroslav Tulach 2011-06-13 11:54:18 UTC
changeset:   200796:6baa6101c289
branch:      release701
tag:         tip
parent:      200795:714b80c116e5
parent:      198519:d7e522762f6f
user:        Jaroslav Tulach <jtulach@netbeans.org>
date:        Mon Jun 13 13:52:52 2011 +0200
summary:     Merge of #198273 to release701
Comment 30 Quality Engineering 2011-06-14 04:58:56 UTC
Integrated into 'releases'
Changeset: http://hg.netbeans.org/releases/rev/d7e522762f6f
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #198273: Cache cnb -> ModuleInfo mapping