This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 197467 - 20s-60s - xml.retriever.catalog.impl.CatalogModelImpl.getModelSource()
Summary: 20s-60s - xml.retriever.catalog.impl.CatalogModelImpl.getModelSource()
Status: RESOLVED FIXED
Alias: None
Product: xml
Classification: Unclassified
Component: Retriever (show other bugs)
Version: 7.0
Hardware: All All
: P2 normal with 2 votes (vote)
Assignee: Svata Dedic
URL:
Keywords: PERFORMANCE
: 200634 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-04-06 01:36 UTC by Exceptions Reporter
Modified: 2011-11-01 21:31 UTC (History)
12 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter: 178077


Attachments
nps snapshot (18.00 KB, application/nps)
2011-04-06 01:36 UTC, Exceptions Reporter
Details
nps snapshot (73.80 KB, application/nps)
2011-05-19 14:17 UTC, Peter Nabbefeld
Details
Stylesheet (2.05 KB, text/xml)
2011-06-10 11:39 UTC, Peter Nabbefeld
Details
Simple Test case - Try to auto-complete the <Test> tag (599 bytes, text/xml)
2011-08-08 13:50 UTC, Peter Nabbefeld
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Exceptions Reporter 2011-04-06 01:36:08 UTC
Build: NetBeans IDE 7.0 RC1 (Build 201103280000)
VM: Java HotSpot(TM) 64-Bit Server VM, 19.1-b02, Java(TM) SE Runtime Environment, 1.6.0_24-b07
OS: Linux
Maximum slowness yet reported was 28660 ms, average is 24612
Comment 1 Exceptions Reporter 2011-04-06 01:36:12 UTC
Created attachment 107534 [details]
nps snapshot
Comment 2 Peter Nabbefeld 2011-05-19 14:17:35 UTC
Created attachment 108394 [details]
nps snapshot
Comment 3 Jaroslav Tulach 2011-05-24 11:20:32 UTC
xml.retriever.catalog downloads huge data over net, I am sure this will be P2 soon.
Comment 4 Peter Nabbefeld 2011-05-24 11:37:21 UTC
(In reply to comment #3)
> xml.retriever.catalog downloads huge data over net, I am sure this will be P2
> soon.

Yeah, that explains why I sometimes see "Downloading" instead of "Please wait". BTW, in one case, it took me even 139234ms.

Seems, the models for schema are rebuilt every time code completion becomes active ...

Probably, one issue with my stylesheets is usage of "plain" tags (i.e. tags without related namespace) - sorry, I do need them. Maybe, the resolution of namespaces is tagged as "incomplete" and thus rebuilt every time code completion is invoked?
Comment 5 Peter Nabbefeld 2011-06-07 09:12:56 UTC
I've read bug should have been upgraded to P2 automatically by exception reporter after reaching 20 reportsr. This seems to be broken, as there're already 37 duplicate reports, thus upgrading priority manually.
Comment 6 Petr Jiricka 2011-06-10 11:27:46 UTC
Can someone please attach a sample XML document which reproduces this issue? Thanks.
Comment 7 Peter Nabbefeld 2011-06-10 11:39:08 UTC
Created attachment 108840 [details]
Stylesheet

Just to make sure the problem exists with this file, I've initiated code completion (ctrl+space) directly behind the <tr> tag in line 12.
Comment 8 Petr Jiricka 2011-06-10 12:28:49 UTC
Ah, this is for XML schema files - I though this issue was about schema-driven XML documents. Thanks, now I can reproduce using this example. 

When I try for regular schema-driven XML file (I am using this schema as an example: http://examples.oreilly.com/9780596002527/examples/simple-1-ns-lib.xsd), then it works correctly - after the first usage, the schema is downloaded and catalog.xml file in the project is created, which points to a local copy of the schema under nbproject/private/retriever. Though I am not sure how this is supposed to work if the file is outside of a project.

When debugging the example attached by Peter Nabbefeld, I see that the IDE attempts to download this URL: http://www.w3.org/1999/XSL/Transform. Sounds like the IDE should have a local cached copy of the XSD corresponding to this, i.e. http://www.w3.org/2007/schema-for-xslt20.xsd.

BTW, did this ever work before? When trying in NB 6.9, I am also getting a "Please wait..." message everytime, and then it does not even produce any results and says "No suggestions".

Also, here is the stack trace under NB 7.0:

org.netbeans.modules.xml.retriever.impl.URLResourceRetriever.getInputStreamOfURL(URLResourceRetriever.java:114)
org.netbeans.modules.xml.retriever.impl.URLResourceRetriever.retrieveDocument(URLResourceRetriever.java:102)
org.netbeans.modules.xml.retriever.catalog.impl.CatalogModelImpl.getModelSource(CatalogModelImpl.java:255)
org.netbeans.modules.xml.retriever.catalog.impl.CatalogModelImpl.doGetModelSource(CatalogModelImpl.java:236)
org.netbeans.modules.xml.retriever.catalog.impl.CatalogModelImpl.getModelSource(CatalogModelImpl.java:171)
org.netbeans.modules.xml.schema.completion.util.DefaultModelProvider.getCompletionModel(DefaultModelProvider.java:118)
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:790)
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:751)
org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:107)
org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:84)
org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:223)
org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424)
org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968)
Comment 9 Petr Jiricka 2011-06-10 12:31:32 UTC
> Ah, this is for XML schema files

Sorry, I meant XSL files.
Comment 10 Peter Nabbefeld 2011-06-10 12:33:19 UTC
AFAIR, the latest working XSL file support was with 6.7, but it had some other
problems which are probably already fixed due to changes in XML model support.
Comment 11 Marek Fukala 2011-06-23 08:56:35 UTC
fixed in web-main#3a94f1d7edd0, releases/release70#1839920b1091

if there's neither schemaLocation nor noNsSchemaLocation attribute in the root tag and there's a declaration of http://www.w3.org/1999/XSL/Transform, then if the version attribute of the same tag is 1.0 some pseudo DTD grammar for xslt1.0 is used and if the version is 2.0 then the xslt2.0 schema is used as the completion model.
Comment 12 Marek Fukala 2011-06-23 13:06:10 UTC
Martine or Petrove, can you please review the fix, thanks!
Comment 13 Petr Pisl 2011-06-24 09:26:54 UTC
Basically I agree with the fix. 

Just small notes. 
- the texts like "http://www.w3c.org..." could be constants
- in the condition 

if(attr.getValue().trim().equals("http://www.w3.org/1999/XSL/Transform"))

I'm not sure whether the attr.getValue() coudln't be null and it would be better to have it like:

if("http://www.w3.org/1999/XSL/Transform".equals(attr.getValue().trim())

or

if(CONSTANT_NAME.equals(attr.getValue().trim())
Comment 14 Peter Nabbefeld 2011-06-24 09:37:40 UTC
(In reply to comment #13)
> I'm not sure whether the attr.getValue() coudln't be null and it would be
> better to have it like:
> 
> if("http://www.w3.org/1999/XSL/Transform".equals(attr.getValue().trim())
> 
> or
> 
> if(CONSTANT_NAME.equals(attr.getValue().trim())

The latter wouldn't work either on null values, as You cannot trim them.

BTW, has spec versions been upgraded, so modules will be updated via UC?
Comment 15 Petr Jiricka 2011-06-24 09:49:14 UTC
> BTW, has spec versions been upgraded, so modules will be updated via UC?

All spec versions were updated between 7.0 and the 7.0.1 codebase, so when 7.0.1 is released, you will be able to get 7.0.1 (including this fix) as an update.
Comment 16 Peter Nabbefeld 2011-06-24 09:54:41 UTC
(In reply to comment #15)
> > BTW, has spec versions been upgraded, so modules will be updated via UC?
> 
> All spec versions were updated between 7.0 and the 7.0.1 codebase, so when
> 7.0.1 is released, you will be able to get 7.0.1 (including this fix) as an
> update.

Should have asked more precise: When will I be able to get this update for my dev build? While it's not always as stable as a release, it has the latest features and allows me to file bugs already when a feature is introduced - before a release, and before too many users want to depend on a feature which probably doesn't work stable.
Comment 17 Marek Fukala 2011-06-24 10:04:11 UTC
(In reply to comment #13)
> Basically I agree with the fix. 
> 
> Just small notes. 
> - the texts like "http://www.w3c.org..." could be constants
If you are aware of such constant in the org.w3c domain please let me know. I'm not a fun of making constants for one usage, moreover it's far more readable the way it is. It may be slightly faster to have it as final constant (although I believe the JVM optimalization will eliminate the difference) but with respect how often the code is invokek it's absolutely a marginal issue.

> 
> if(attr.getValue().trim().equals("http://www.w3.org/1999/XSL/Transform"))
> 
> I'm not sure whether the attr.getValue() coudln't be null and it would be
> better to have it like:
I believe it will never be null. If the value is empty then it return empty string. If the value is not there then it's invalid xml and the code won't run.

Thank you for the review.

> 
> if("http://www.w3.org/1999/XSL/Transform".equals(attr.getValue().trim())
> 
> or
> 
> if(CONSTANT_NAME.equals(attr.getValue().trim())
Comment 18 Marek Fukala 2011-06-24 10:08:19 UTC
(In reply to comment #16)
> (In reply to comment #15)
> > > BTW, has spec versions been upgraded, so modules will be updated via UC?
> > 
> > All spec versions were updated between 7.0 and the 7.0.1 codebase, so when
> > 7.0.1 is released, you will be able to get 7.0.1 (including this fix) as an
> > update.
> 
> Should have asked more precise: When will I be able to get this update for my
> dev build? While it's not always as stable as a release, it has the latest
> features and allows me to file bugs already when a feature is introduced -
> before a release, and before too many users want to depend on a feature which
> probably doesn't work stable.

We would have to update the spec versions after each bugfix to fulfill such requirement, wouldn't we? I believe it's more practical to download daily builds (for example weekly) if you decide to work on dev that relying on the updates.
Comment 19 Peter Nabbefeld 2011-06-24 10:20:19 UTC
(In reply to comment #18)
> 
> We would have to update the spec versions after each bugfix to fulfill such
> requirement, wouldn't we? I believe it's more practical to download daily
> builds (for example weekly) if you decide to work on dev that relying on the
> updates.
>
I've already noticed, UC support is not what it should look like and enhancements are low priority. A good solution would be to have a patch version or to consider date for updating.

Because of this, it would be nice to provide links to the corresponding changeset of the bugfix, so I could at least update modules manually via module manager.
Comment 20 Marek Fukala 2011-06-24 10:33:46 UTC
I've added the changeset here already when closing the issue: web-main#3a94f1d7edd0

just add it to the mercurial http repo url:

http://hg.netbeans.org/web-main/rev/3a94f1d7edd0
Comment 21 Peter Nabbefeld 2011-06-24 10:36:38 UTC
(In reply to comment #20)
> I've added the changeset here already when closing the issue:
> web-main#3a94f1d7edd0
> 
> just add it to the mercurial http repo url:
> 
> http://hg.netbeans.org/web-main/rev/3a94f1d7edd0

Thank You! I'm not using mercurial for anything other than for NetBeans, so I've got some problems with such "shortlinks" - probably a functionality for supporting links could be added to bugzilla? Just an idea.
Comment 22 Quality Engineering 2011-06-24 14:26:34 UTC
Integrated into 'main-golden', will be available in build *201106240600* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/3a94f1d7edd0
User: Marek Fukala <mfukala@netbeans.org>
Log: #197467 - 20s-60s - xml.retriever.catalog.impl.CatalogModelImpl.getModelSource()
Comment 23 Marian Mirilovic 2011-06-30 07:33:53 UTC
I do not see an integration into release701 ....
Comment 24 Petr Jiricka 2011-06-30 09:16:14 UTC
Here it is: http://hg.netbeans.org/releases/rev/1839920b1091
Comment 25 Marian Mirilovic 2011-06-30 09:19:13 UTC
(In reply to comment #24)
> Here it is: http://hg.netbeans.org/releases/rev/1839920b1091

Ok, thanks.
Comment 26 Quality Engineering 2011-07-05 23:59:15 UTC
Integrated into 'releases'
Changeset: http://hg.netbeans.org/releases/rev/1839920b1091
User: Marek Fukala <mfukala@netbeans.org>
Log: #197467 - 20s-60s - xml.retriever.catalog.impl.CatalogModelImpl.getModelSource()
Comment 27 Peter Nabbefeld 2011-08-08 09:42:33 UTC
This has been working for some days - but since some time I'm getting the problems again :-)

Please look at latest stack trace, especially, if this time it's caused at some other location - maybe, exception reporter just doesn't assign the reports correctly (and should open a new issue ...)
Comment 28 Marek Fukala 2011-08-08 09:59:01 UTC
(In reply to comment #27)
> This has been working for some days - but since some time I'm getting the
> problems again :-)
> 
> Please look at latest stack trace, especially, if this time it's caused at some
> other location - maybe, exception reporter just doesn't assign the reports
> correctly (and should open a new issue ...)

What is the report/issue?
Comment 29 Peter Nabbefeld 2011-08-08 12:23:02 UTC
(In reply to comment #28)
> 
> What is the report/issue?

http://statistics.netbeans.org/analytics/exception.do?id=522788

The problem - again - is auto completion (adding the corresponding closing tag).
Comment 30 Marek Fukala 2011-08-08 12:46:24 UTC
I believe no one changed the code recently so it must have either regressed somewhere else you it is a different case - can you attach the file which causes the issue please?
Comment 31 Martin Fousek 2011-08-08 13:48:09 UTC
Hmm, I'm afraid that this isn't only one regression which happened. I was fixing another one in bug #190570.

BTW, so I also did some small changes in RetrieverImpl.java but I don't think that it could impact this issue. It was just improving CC for Maven based projects on place where otherwise NPE was thrown.
Comment 32 Peter Nabbefeld 2011-08-08 13:50:55 UTC
Created attachment 109859 [details]
Simple Test case - Try to auto-complete the <Test> tag
Comment 33 Svata Dedic 2011-09-20 08:51:43 UTC
Rather ridiculous situation: the IDE is dowloading content of the referenced schemas/DTD again and again (because of code completion) - and they should be cached locally. On the other hand, bug #116072 was filed some time ago NOT to auto-create catalog.xml files in the project. I guess that automatic modification of these would be almost as bad as creation (without prompt or explicit configuration).

I could change the code so that the file is actually cached and even recorded in the 'private' catalog (inside nbproject dir), but the 'master' project-wide catalog is not updated for code completion initiated fetches. It's somewhat magical and would deserve an option for catalog management in the future.

I wonder what functionality (if any) would be broken by this change.
Comment 34 Peter Nabbefeld 2011-09-22 06:37:31 UTC
(In reply to comment #33)
> Rather ridiculous situation: the IDE is dowloading content of the referenced
> schemas/DTD again and again (because of code completion) - and they should be
> cached locally. On the other hand, bug #116072 was filed some time ago NOT to
> auto-create catalog.xml files in the project. I guess that automatic
> modification of these would be almost as bad as creation (without prompt or
> explicit configuration).
> 
> I could change the code so that the file is actually cached and even recorded
> in the 'private' catalog (inside nbproject dir), but the 'master' project-wide
> catalog is not updated for code completion initiated fetches. It's somewhat
> magical and would deserve an option for catalog management in the future.
> 
> I wonder what functionality (if any) would be broken by this change.

I've read about bug #116072, it will probably also be valid for nbproject/private. You don't e.g. add language description files (such as *.nbs) for each Java dialect to the project, the compiler is used, instead.

IMO, the schemas belong to XML processing, not to the projects, in general. You could use the system filesystem to store catalogs and schemas. If a private schema is needed for some project, there should be an option to add it to the project, like for privately used libraries.

In either case, You should probably invent Jesse to comment, as he filed bug #116072.
Comment 35 Svata Dedic 2011-09-30 21:04:31 UTC
More or less fixed. I made the IDE cache resources in nbproject/private + write to the private catalog.xml, but the main catalog is not updated automatically from the completion queries. If no project context is available, user cache dir is used.

Not ideal, but seems to work satisfactorily. The whole XML retrieval/caching should be reviewed during the next release + some refresh/update management should be added.
Comment 36 Quality Engineering 2011-10-01 14:03:32 UTC
Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/a81e7a596f23
User: Svata Dedic <sdedic@netbeans.org>
Log: #197467: mostly fixed retrieval of xml schemas. Referrents of shipped schemas are stored in user cachedir
Comment 37 Jesse Glick 2011-11-01 21:31:06 UTC
*** Bug 200634 has been marked as a duplicate of this bug. ***