Created attachment 143746 [details]
stacktrace

I am not able to reproduce it but it seems that the CND ClassPath root (file:/C:/Users/jprox/Documents/NetBeansProjects/cppapplication_1/) is not normalized.

Tomas, why do you think that's because of normalization?

And is normalization included in ClassPathImplementation contract? I see nothing about normalization in ClassPathImplementation javadocs.

I don't see CND fault here and, which is more important, don't have any idea how to fix this.

Unfortunately I am not able to reproduce it.
It should be somehow reproduceable on systems with non case sensitive FS (windows).
The problem is that the classpath root: cppapplication_1 differs in case with normalized file CppApplication_1/nbproject/.
Is the classpath created from FileObjects, URLs or Files?

Created attachment 145201 [details]
stacktrace

I was installing the HelloQtWorld_1 sample. This error appeared immediately after the new project wizard was closed

This bug already has 10 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=203128

Created attachment 146630 [details]
stacktrace

Open C++ project from NB 8.0...

This bug already has 20 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=203128

*** Bug 242986 has been marked as a duplicate of this bug. ***

In fact nothing to do with parsing.api.
It's either cnd issue of FS issue depending which case is correct.
java.lang.IllegalArgumentException: file:/C:/Users/Jimbo/Documents/NetBeansProjects/HelloQtWorld_1/nbproject/ not in file:/C:/Users/Jimbo/Documents/NetBeansProjects/helloqtworld_1/

Means that either cad created a ClassPath root with non normalised URL - in case when helloqtworld_1 is correct to MasterFS has wrong FileObject names in case HelloQtWorld_1 is correct.

As all the reports comes from cad project seems that the first is the case:
The cnd created a ClassPath root with non normalised URL.
Where is the code creating the cnd classapth?

(In reply to Tomas Zezula from comment #12)
> As all the reports comes from cad project seems that the first is the case:
> The cnd created a ClassPath root with non normalised URL.
> Where is the code creating the cnd classapth?
See classes:
org.netbeans.modules.cnd.makeproject.MakeProject.MutableCP
org.netbeans.modules.cnd.makeproject.MakeSources

Tomas,
Again, where is a contract on normalized resources in ClassPathImplementation?
I do not see.

The ClassPath can be either created from FileObjects which are already normalised or from URLs which come either from FileObjects or already normalised Files.
The Files should be normalised immediately after an user input as the normalisation is very expensive, so all the IDE parts expect (assert) that the files are normalised rather then doing normalisation again and again.

(In reply to Tomas Zezula from comment #15)
> The ClassPath can be either created from FileObjects which are already
> normalised or from URLs which come either from FileObjects or already
> normalised Files.
> The Files should be normalised immediately after an user input as the
> normalisation is very expensive, so all the IDE parts expect (assert) that
> the files are normalised rather then doing normalisation again and again.
Please add it in the API doc.
I do not see how CND can add not normalized FO. See MakeSources.initSources():

        FileObjectBasedSources sources = new FileObjectBasedSources();
        Set<FileObject> added = new HashSet<>();
        sourceRootList.add(project.getProjectDirectory().getPath()); // add remote project itself to the tail
        for (String name : sourceRootList) {
            String path = CndPathUtilities.toAbsolutePath(project.getProjectDirectory(), name);
            path = RemoteFileUtil.normalizeAbsolutePath(path, fsEnv);
            String displayName = (fsEnv.isLocal() ? "" : fsEnv.getDisplayName() + ":") + path; //NOI18N
            FileObject fo = RemoteFileUtil.getFileObject(path, fsEnv);
            if (!added.contains(fo)) {
                if (fo == null) {
                    new NullPointerException("Null file object for " + fsEnv + ':' + path).printStackTrace(); //NOI18N
                } else {
                    sources.addGroup(project, GENERIC, fo, displayName);
                    added.add(fo);
                }
            }
        }
        return sources;

I looked at MakeProject.MutableCP and it seems pretty safe to me.
The URL are always taken from FileObjects which should be already normalised.
Is someone in your team able to reproduce this issue?
I have no Windows and it happens only on OS with case insensitive FS.
I will try tomorrow ask someone from QA to try to reproduce it.

(In reply to Tomas Zezula from comment #17)
> I looked at MakeProject.MutableCP and it seems pretty safe to me.
> The URL are always taken from FileObjects which should be already normalised.
> Is someone in your team able to reproduce this issue?
> I have no Windows and it happens only on OS with case insensitive FS.
> I will try tomorrow ask someone from QA to try to reproduce it.

I tried to reproduce the bug without success.

(In reply to Alexander Simon from comment #18)
> (In reply to Tomas Zezula from comment #17)
> > I looked at MakeProject.MutableCP and it seems pretty safe to me.
> > The URL are always taken from FileObjects which should be already normalised.
> > Is someone in your team able to reproduce this issue?
> > I have no Windows and it happens only on OS with case insensitive FS.
> > I will try tomorrow ask someone from QA to try to reproduce it.
> 
> I tried to reproduce the bug without success.
I found one of possible scenarios:
1. Create new project C/C++ application. IDE creates project in the folder CppApplication_1.
2. Close project in IDE.
3. In terminal rename folder CppApplication_1->test and test->cppapplication_1.
4. Open project in the folder cppapplication_1.
See exception.
It seems bug in cached normalized paths in the FileUtils.

Hi Alex,
I've tried the rename and I got a bit different but related exception:

java.lang.AssertionError: Need to normalize /Users/tom/NetBeansProjects/CppApplication_1(class java.io.File) was /Users/tom/NetBeansProjects/cppapplication_1(class java.io.File)
	at org.openide.filesystems.FileUtil.assertNormalized(FileUtil.java:165)
	at org.openide.filesystems.FileUtil.assertNormalized(FileUtil.java:149)
	at org.openide.filesystems.FileChangeImpl.addFileChangeListenerImpl(FileChangeImpl.java:220)
	at org.openide.filesystems.FileUtil.addFileChangeListener(FileUtil.java:306)
	at org.netbeans.api.java.classpath.ClassPath$RootsListener.safeAddListener(ClassPath.java:1290)
	at org.netbeans.api.java.classpath.ClassPath$RootsListener.addRoots(ClassPath.java:1223)
	at org.netbeans.api.java.classpath.ClassPath.createRoots(ClassPath.java:331)
	at org.netbeans.api.java.classpath.ClassPath.getRoots(ClassPath.java:269)
	at org.netbeans.api.java.classpath.ClassPath.findOwnerRoot(ClassPath.java:532)
	at org.netbeans.api.java.classpath.ClassPath.contains(ClassPath.java:550)
	at org.netbeans.modules.cnd.makeproject.MakeProjectClassPathProvider.findClassPath(MakeProjectClassPathProvider.java:102)
	at org.netbeans.api.java.classpath.ClassPath.getClassPath(ClassPath.java:636)
	at org.netbeans.modules.parsing.impl.indexing.errors.Utilities.getSourceClassPathFor(Utilities.java:78)
	at org.netbeans.modules.parsing.impl.indexing.errors.ErrorAnnotator$1.run(ErrorAnnotator.java:315)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.jav

What I've found by debugging so far:
1st) The root passed to parsing.api has wrong case
(file:/Users/tom/NetBeansProjects/CppApplication_3/)

2nd) The root (PathResource) in the MutableCP has also wrong case
(file:/Users/tom/NetBeansProjects/CppApplication_3/)

3rd) The problem is in the MakeSources which returns the old root FileObject (file:/Users/tom/NetBeansProjects/CppApplication_3/). Unfortunately FileObject.isValid() returns true even the FO is illegal (non normalised).

I am now looking to MakeSources why it does not refresh the SourceGroup.

Created attachment 148548 [details]
Problematic code

Showing MakeSources creating a SourceGropup from non normalised FileObject.
See watches. The first one showing the File obtained from FileObject the second one showing the normalised File from the same FileObject.

I did more experiments:
I've changed the MakeSources to listen on delegate (FileObjectBasedSources) in this case root change should cause refresh of MakeSources.
However changing the name case does not cause any FS event.
More surprisingly the org.netbeans.modules.cnd.remote.fs.CndFileSystemProviderImpl returns the FileObject even for non existent FileObject and claims that it's valid.
The FileObject is MasterFS FolderFO.
I am not sure if there is some caching in CndFileSystemProviderImpl or it's bug in MasterFS.

I talked with Jarda and we both agree that ideally the MasterFS should change the name after the rename.
However the test case is quite synthetic so I am not sure how it can happen during normal New C/C++ project.

From parsing api point of view the cnd project provides invalid classpath.
Please evaluate the CndFileSystemProviderImpl and if no caching please reassign to master fs.

One more thing:
I've verified the MasterFS events during the external rename of project folder from CppApplication_8 to cppapplication_8 (change of case). The rename did not cause any FileObject events on the project folder. However the native listener delivered an event to MasterFS as seen from LOG:
FINE [org.netbeans.modules.masterfs.watcher.macosx.OSXNotifier]: Event on /Users/tom/NetBeansProjects/
On Mac the FSEventStream delivers change in enclosing folder.

So the issue seems to belong to MasterFS.

(In reply to Tomas Zezula from comment #23)
...
> I am not sure if there is some caching in CndFileSystemProviderImpl or it's
> bug in MasterFS.

CndFileSystemProviderImpl does not use any cache for local files.

It just delegates to FileSystemProvider (o.n.m.remote.spi.FileSystemProvider in module dlight.remote), which in turn - in case of local file system - calls LocalFileSystemProvider (in the same module).

The latter delegates everything to FileUtil.

>The latter delegates everything to FileUtil.
In this case please reassign to MasterFS as it cannot be solved on the CND level as there is no rename event.

I can reproduce the issue on Windows, the exception is thrown almost every time I create a new "C/C++ Application" project (right after starting NetBeans).

The problem is that the lowercase file name is cached in o.n.m.masterfs.filebasedfs.children.ChildrenSupport (field notExistingChildren) when new project wizard checks whether the project folder exists (at this time it doesn't exists yet, but the lowercase name is stored and used later when the ChildrenSupport instance is refreshed).

The reason it is implemented this way in masterfs is (I suppose) performance. Fixing this could make the platform slower (I haven't found a solution that is reliable and cheap yet), and generally such a significant change deep in filesystems is quite risky, so I would prefer not to fix this now (8.0.1) unless some safer solution is proposed.

A simple workaround is not to lowercase the path in CndFileUtils.getFlags() at all, or use FileUtil.normalizeFile instead (which will also make it slower, but the impact should be smaller).

(I'll attach the patches shortly.)

Created attachment 148592 [details]
Naive fix in masterfs

Naive fix for masterfs which checks whether the files are normalized in ChildrenSupport.

Test org.netbeans.modules.masterfs.filebasedfs.fileobjects.StatFilesTest fails.

(Attaching solely for demonstration of the problematic part and evaluation.)

Created attachment 148593 [details]
Draft of workaround in CndFileUtils

Please check this patch and consider removing or replacing the "path.toLowerCase()" invocation in cnd.utils. This is the place where the incorrect (not normalized) path is generated.
Thank you.

Reassigning to cnd/Other for now.
Please assign the issue back to me after evaluation of the workaround.

> Created attachment 148592 [details]
> Naive fix in masterfs
> (Attaching solely for demonstration of the problematic part and evaluation.)
The patch would not work correctly on Unix system, getCanonicalFile() resolves symlinks there. (FileUtil.normalizePath() cannot be used because incorrect normalized path for the lowercase path can be cached in FileUtil.normalizedRef.)

Alexander,
could you found real steps to reproduce the issue.

Steps in the comment #19 looks artificial.
Most user comments in exception reports claim that exception was appeared while creating simple application.

BTW Alexander, you can find lots of such exceptions in IDE test log on our Windows machine.

(In reply to Jaroslav Havlin from comment #29)
> Created attachment 148593 [details]
> Draft of workaround in CndFileUtils
> 
> Please check this patch and consider removing or replacing the
> "path.toLowerCase()" invocation in cnd.utils. This is the place where the
> incorrect (not normalized) path is generated.
> Thank you.
Unfortunately this patch cannot be applied because it results in significant performance degradation of C/C++ parser.

Probably method 
FileUtil.createFolder(FileObject folder, String name)
should clear normalizedRef cache for created folder?

Created attachment 148643 [details]
Test case

> Probably method FileUtil.createFolder(FileObject folder, String name)
> should clear normalizedRef cache for created folder?
Indeed, to fix the letter-case related bugs in filesystem, we'll have to improve caching of normalized files. It will be necessary, but not sufficient, the main problem is in ChildrenSupport.

I cannot reproduce the bug in the IDE any more (probably some external conditions changed), but I managed to create a test case.

http://hg.netbeans.org/core-main/rev/1a5f26e11ceb
Fixed NamingFactory and ChildrenSupport.
Please verify (if you can verify it as soon as possible, we might try to integrate the fix into 8.0.1).
Thank you.

> Indeed, to fix the letter-case related bugs in filesystem, we'll have to 
> improve caching of normalized files.
I'm sorry, I was wrong. The cache is cleared in constructor of FileEvent, so it works correctly already.

Integrated into 'main-silver', will be available in build *201408130001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/1a5f26e11ceb
User: Jaroslav Havlin <jhavlin@netbeans.org>
Log: #240156: Do not update cached names when refreshing not existing children

ChildrenSupport.notExistingChildren can contain files with incorrect (not canonical) names.
If they are detected as valid (if they really exists, but with different canonical name), the
cached name is updated to the incorrect case. We need a way to tell the NamingFactory that
the passed file name is not guaranteed to be canonical and that the cached name should not
be changed.

Should this bug be included in the patch (801patch1-candidate)?
If so, please verify it carefully.

(In reply to Jaroslav Havlin from comment #39)
> Should this bug be included in the patch (801patch1-candidate)?
> If so, please verify it carefully.

I agree, Jirka please verify.

I can still reproduce this exception, but the steps to reproduce are little more complicated. 

Steps to reproduce:
1) create new C++ application CppApplication_1 
2) delete the application with the whole folder
3) create new C++ application cppApplication_1 (note the first letter is not capital)

-> exception is thrown

Should I file new issue for it? Since it is not P2 anymore


Creating new C++ application with unique name works fine. Renaming while changing only the case of the letters works as well. So this issue maybe considered as verified


Product Version: NetBeans IDE Dev (Build 20140905-2854a56cc00e)
Updates: Updates available
Java: 1.8.0_20; Java HotSpot(TM) 64-Bit Server VM 25.20-b23
Runtime: Java(TM) SE Runtime Environment 1.8.0_20-b26
System: Windows 7 version 6.1 running on amd64; Cp1250; en_US (nb)
User directory: C:\Users\jprox\AppData\Roaming\NetBeans\dev
Cache directory: C:\Users\jprox\AppData\Local\NetBeans\Cache\dev

(In reply to Jiri Prox from comment #41)
> I can still reproduce this exception, but the steps to reproduce are little
> more complicated. 
> [...]
> Should I file new issue for it? Since it is not P2 anymore
Yes, please.

> Creating new C++ application with unique name works fine. Renaming while
> changing only the case of the letters works as well. So this issue maybe
> considered as verified.
OK, I'm changing status to verified.

Thank you very much.

Integrated into release801 branch:
http://hg.netbeans.org/releases/rev/97140ec5c5b1 (fix)
http://hg.netbeans.org/releases/rev/fbadb208d2a9 (version and description)

*** Bug 247420 has been marked as a duplicate of this bug. ***