When you load an Ant-based project, project.xml must be validated against schema. This requires a bunch of stuff in Xerces. Not news; true since 6.5. (As of #142680, a given project.xml is not revalidated at every startup so long as it is not modified.)

But now it is getting validated again! It was not validating a week ago, now it validates for some reason. Have we changed the format of the project files? If so, we need to fix the test to contain newer version...

No idea what changed, but I think it is more likely a problem with the test than the code.

There are no changes in the j2se project schema.
Jardo, which project type are you using?

Integrated into 'main-golden', will be available in build *201102050000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/4ed84992d14f
User: Jesse Glick <jglick@netbeans.org>
Log: Reducing log verbosity, making level=FINE more useful for diagnosis of e.g. #195029.

@Tomáš: Good to know there were no XSL and schema changes. That increases the likehood this is really problem in the code. I don't exactly what version the test uses, but to find out:

$ cd ide.kit/test/whitelist
$ cat readme.txt

and typing 

$ ant 

seems to start download of the ZIP file with the project. The URL is

http://hg.netbeans.org/binaries/FDCBE13B6CA7B154F6EE29BEF3B9404F7A7B1A57-lime6.zip

@Jesse's changes: I don't understand why you removed REGRESSION, PERFORMANCE keywords (the behavior is changed compared to 6.9 and changed to be worse) and changed priority to P3 (in addition to the above, no changes can be propagated from ergonomics build). So this is clearly not a "test p3 & ignore" problem.

There were no XSD changes by t XSL was updated. This may be the reason.
Anyway I will try.

I am trying to investigate too. I can start the test in 

$ cd ide.kit/test/whitelist
$ cat readme.txt
$ ant -Ddebug.port=4000

When I place breakpoint to AntBasedProjectFactorySingleton:370 (around the known hashes check), it is not invoked at all during first run (good, there is no project).

The breakpoint is hit for the second time. CRC value is computed as 0xa891a89f, but knownHashes is empty. Thus the code starts to validate([1]!). The computed CRC is then added to the knownHashes, but possibly before the upgrade of the files(?). Then there were next two hash values added into the knownHashes preferences node asynchronously. Possibly, these may not be written to the knownHashes set at all in the test(?), if it finishes sooner.

That is all I know by now.

[1] slightly surprising to me as this means the code always validates the XML files until it accumulates the list of known hashes having some well known precomputed in newly installed build would be more effective

I chatted briefly with Tomáš. The CRC check is done in loadProject. The upgrade of XML files code is run in ProjectHook (e.g. later). The upgrade is handled by some helper classes in project.ant - these classes shall register the new CRC check. Passing to Jesse to deal with that.

In addition Tomáš mentioned that the proper CRC values are in some properties file. The need for usages of Preferences for storing CRC values is unclear. Why it is not enough to check the value in nbproject/genfiles.properties and skip the validation if the value is the same[1]?


[1] This would eliminate the need to validate anything on the "first" start when the preferences node is not filled yet.

(In reply to comment #6)
> I don't understand why you removed REGRESSION, PERFORMANCE keywords

There is no demonstrated change in behavior of the IDE, only of a test which may or may not correspond to a real use case. When I manually checked the behavior of the XML validation cache in a dev build, it worked fine.

> no changes can be propagated from ergonomics build

Stop running this test in the ergonomics build then. I certainly made no changes in the ergonomics repo so it makes no sense for this test to be blocking propagation of unrelated changes from that repo.

(In reply to comment #8)
> having some well known [hashes]
> precomputed in newly installed build would be more effective

Not possible since each project will typically differ at least in <name>.

(In reply to comment #9)
> The upgrade of XML files code is run in ProjectHook
> [...] these classes shall register the new CRC check.

Not sure what this means. Are you referring to org.netbeans.modules.java.j2seproject.UpdateProjectImpl? That would only be active once in the lifetime of a project, so whatever it does is not a performance issue, though the test might be inadvertently triggering this by using a very old sample project.

Perhaps you meant that AntProjectHelper.saveXml could also cache the CRC of what it saves, since it has already validated that XML so it could make it slightly faster to load next time - a useful optimization, though a small benefit compared to the existing cache since it only takes effect in the case that project.xml is modified, which is much less often than it is read.

> Is it not enough to check the value in nbproject/genfiles.properties and skip
> the validation if the value is the same?

(Assuming the project uses GeneratedFilesHelper.) This may be possible; would have the advantage of keeping the information more localized (avoiding NbPreferences), and the minor disadvantage of forcing a revalidation after making and then reverting a change to project.xml. Would also need to consider whether permitting XML validation to be bypassed by content in the project source tree itself would introduce any security risk, i.e. whether any code calling getPrimaryConfigurationData(true) could misbehave badly if given invalid data. Also need to check if this even works (without a newly introduced field specially for this purpose) if there are modifications to build.xml or build-impl.xml or if an older version of the stylesheet is present at runtime; in either case the new .data.crc32 will not be written even after a project.xml modification.

I think we can agree on following:

(In reply to comment #10)
> Perhaps you meant that AntProjectHelper.saveXml could also cache the CRC of
> what it saves, since it has already validated that XML so it could make it
> slightly faster to load next time - a useful optimization, 

OK. Based on this I added the XML validation classes to whitelist for now: ergonomics#66c7074f7c13 
Revert this commit when integrating your fix.


> > Is it not enough to check the value in nbproject/genfiles.properties and skip
> > the validation if the value is the same?
> 
> (Assuming the project uses GeneratedFilesHelper.) This may be possible; 

This would be nice in general (if the additional concerns can be solved). The test is not however going to be influenced by this additional improvement.

Integrated into 'main-golden', will be available in build *201102090000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/66c7074f7c13
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #195029: Temporarily allowing load of XML validation classes on startup. Caused by cache miss in the AntBasedProjectFactory.loadProject code due to upgrade of the project files to newer version not updating the cache.

I see plenty of whitelist violations; reverting #66c7074f7c13 does not seem to add any, certainly nothing related to Xerces. Anyway I should be able to fix based on unit tests.

core-main #7fa214936dcf

Integrated into 'main-golden', will be available in build *201204270400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/7fa214936dcf
User: Jesse Glick <jglick@netbeans.org>
Log: #195029: cache miss in knownValidProjectXmlCRC32s during test.
Save the checksum of project.xml being written (after it has been validated),
so it would not need to be redundantly validated when loaded after restart.