Created attachment 106809 [details]
stacktrace

Integrated into 'main-golden', will be available in build *201103100400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/4e166562bc36
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: fixing #196425 -  AssertionError: no FileObject
-- extra checks

reproduced on NetBeans IDE Dev (Build 110310-5288f673a0aa) (#5288f673a0aa)
Exception with extra check:

java.lang.AssertionError: no FileObject in [org.netbeans.modules.masterfs.filebasedfs.FileBasedFileSystem@ad5b0c] for /var/tmp/as204739-cnd-test-downloads/pkg-config-0.25/glib-1.2.10/gmodule/gmoduleconf.h FileUtil.toFileObject = MasterFileObject[/var/tmp/as204739-cnd-test-downloads/pkg-config-0.25/glib-1.2.10/gmodule/gmoduleconf.h@820353:e80a28,valid=true]
	at org.netbeans.modules.cnd.apt.support.ResolvedPath.<init>(ResolvedPath.java:74)
	at org.netbeans.modules.cnd.apt.utils.APTIncludeUtils.resolveFilePath(APTIncludeUtils.java:79)
	at org.netbeans.modules.cnd.apt.impl.support.APTIncludeResolverImpl.resolveFilePath(APTIncludeResolverImpl.java:123)
	at org.netbeans.modules.cnd.apt.impl.support.APTIncludeResolverImpl.resolveInclude(APTIncludeResolverImpl.java:90)
	at org.netbeans.modules.cnd.apt.support.APTAbstractWalker.onInclude(APTAbstractWalker.java:100)
	at org.netbeans.modules.cnd.apt.support.APTWalker.onAPT(APTWalker.java:223)
	at org.netbeans.modules.cnd.apt.support.APTWalker.toNextNode(APTWalker.java:329)
	at org.netbeans.modules.cnd.apt.support.APTWalker.nextTokenImpl(APTWalker.java:301)
	at org.netbeans.modules.cnd.apt.support.APTWalker.access$200(APTWalker.java:61)
	at org.netbeans.modules.cnd.apt.support.APTWalker$WalkerTokenStream.nextToken(APTWalker.java:103)
	at org.netbeans.modules.cnd.apt.support.APTWalker$WalkerTokenStream.nextToken(APTWalker.java:95)
	at org.netbeans.modules.cnd.antlr.TokenStreamSelector.nextToken(TokenStreamSelector.java:36)
	at org.netbeans.modules.cnd.apt.support.APTExpandedStream.nextToken(APTExpandedStream.java:111)
	at org.netbeans.modules.cnd.apt.support.APTExpandedStream.nextToken(APTExpandedStream.java:73)
	at org.netbeans.modules.cnd.apt.utils.APTCommentsFilter.nextToken(APTCommentsFilter.java:69)
	at org.netbeans.modules.cnd.apt.utils.APTCommentsFilter.nextToken(APTCommentsFilter.java:57)
	at org.netbeans.modules.cnd.apt.impl.support.lang.APTBaseLanguageFilter$FilterStream.nextToken(APTBaseLanguageFilter.java:143)
	at org.netbeans.modules.cnd.antlr.TokenBuffer.<init>(TokenBuffer.java:58)
	at org.netbeans.modules.cnd.antlr.TokenBuffer.<init>(TokenBuffer.java:48)
	at org.netbeans.modules.cnd.antlr.LLkParser.<init>(LLkParser.java:27)
	at org.netbeans.modules.cnd.antlr.LLkParserNoEx.<init>(LLkParserNoEx.java:38)
	at org.netbeans.modules.cnd.modelimpl.parser.generated.CPPParser.<init>(CPPParser.java:451)
	at org.netbeans.modules.cnd.modelimpl.parser.generated.CPPParser.<init>(CPPParser.java:457)
	at org.netbeans.modules.cnd.modelimpl.parser.CPPParserEx.<init>(CPPParserEx.java:111)
	at org.netbeans.modules.cnd.modelimpl.parser.CPPParserEx.getInstance(CPPParserEx.java:126)
	at org.netbeans.modules.cnd.modelimpl.parser.ParserProviderImpl$Antlr2CppParser.init(ParserProviderImpl.java:110)
	at org.netbeans.modules.cnd.modelimpl.csm.core.FileImpl.doParse(FileImpl.java:1164)
	at org.netbeans.modules.cnd.modelimpl.csm.core.FileImpl._parse(FileImpl.java:821)
	at org.netbeans.modules.cnd.modelimpl.csm.core.FileImpl.ensureParsed(FileImpl.java:513)
	at org.netbeans.modules.cnd.modelimpl.csm.core.ParserThread._run(ParserThread.java:130)
	at org.netbeans.modules.cnd.modelimpl.csm.core.ParserThread.run(ParserThread.java:72)
	at org.netbeans.modules.cnd.modelimpl.csm.core.ParserThreadManager$Wrapper.run(ParserThreadManager.java:91)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968)

FS gurus, can you evaluate what's wrong? 
How could it be that findRosource does not find what is found by FileUtils.toFileObject?
path is normalized in both cases.

The following assertion fails
assert fileSystem.findResource(path.toString()) != null : "no FileObject in " + fileSystem + " for " + path + " FileUtil.toFileObject = " + FileUtil.toFileObject(new File(FileUtil.normalizePath(path.toString())));

With the message:
no FileObject in
[org.netbeans.modules.masterfs.filebasedfs.FileBasedFileSystem@ad5b0c] for
/var/tmp/as204739-cnd-test-downloads/pkg-config-0.25/glib-1.2.10/gmodule/gmoduleconf.h
FileUtil.toFileObject =
MasterFileObject[/var/tmp/as204739-cnd-test-downloads/pkg-config-0.25/glib-1.2.10/gmodule/gmoduleconf.h@820353:e80a28,valid=true]

If you are mixing external changes with FileObject access, then I guess this is normal. I wrote a test (ergonomics#c9988b90979b) to verify that findResource does not check the actual state of disk (to eliminate I/O), while toFileObject does. If you are in an environment that does bunch of external changes consider using "refresh" or prefer toFileObject.

If the state on disk is not influenced by external changes, then I no idea what can be wrong. What now? Close as invalid, or do you want to take the bug back?

(In reply to comment #5)
> If you are mixing external changes with FileObject access, then I guess this is
> normal. I wrote a test (ergonomics#c9988b90979b) to verify that findResource
> does not check the actual state of disk (to eliminate I/O), while toFileObject
> does. 
Is it possible to understand somehow from API or javadoc?
Imho, it's very bad, because FileObject was good as abstraction which allowed as work with remote sources by just providing own impl of remote file sytem...

> If you are in an environment that does bunch of external changes consider
> using "refresh" or prefer toFileObject.
> 
> If the state on disk is not influenced by external changes, then I no idea what
> can be wrong. What now? Close as invalid, or do you want to take the bug back?
I will take it back. We can not release 7.0 with this bug.
Assert was added to find root cause of NPE after using return value of ResolvedPath.getFileObject (see issue #196269)

The other problem we need to deal with is complaint that FileSystems are slow. Most of the slowness used to be caused by Master FileSystem trying to reflect reality as close as possible. Thus even a simple create FileObject operation required three or four disk touches. That was not useful either.

FileSystems still remain good abstraction, you just have to be aware that if you mix internal (via FileSystem API) and external (either java.io.File; but we check those a bit; but mostly external process changes), the state can get out of sync. Imho there is nothing surprising on that. NFS also need not propagate changes done by other machine for a while.

In case you believe there is a difference between the cache and reality, it is easy to ask masterfs to synchronize by calling FileUtil.refreshAll(), FileObject.refresh() or FileUtil.toFileObject(...) or toFile(...). Just remember that it will increse the amount of I/O at that moment significantly. 

I can put this text into a javadoc, just tell me where you'll find it appropriate...

(In reply to comment #7)
> FileSystems still remain good abstraction, you just have to be aware that if
> you mix internal (via FileSystem API) and external (either java.io.File; but we
> check those a bit; but mostly external process changes), the state can get out
> of sync. Imho there is nothing surprising on that. NFS also need not propagate
> changes done by other machine for a while.
It could be related somehow to ext changes, but...
We moved completely component where assert is triggered to FileObjects from java.io.Files
and as I can see now the problem was:
- we made a conclusion about "existence" of FileObject based on non-null FileUtil.toFileObject return value 
- and right after that using fs.findResource gives "null".
It's a problem, right?

I think for master FS impl *after* call to FileUtil.toFileObject in case of non-null return value and no any 'delete'-like external changes fs.findResouce must also return non-null. What do you think?

use FileUtil.toFileObject instead of findResource for masterfs
http://hg.netbeans.org/cnd-main/rev/ae2c49704e2a

(In reply to comment #8)
> I think for master FS impl *after* call to FileUtil.toFileObject in case of
> non-null return value and no any 'delete'-like external changes fs.findResouce
> must also return non-null. What do you think?

This is what my test is testing and that is OK, imho:

assert fs.findResource(res) == null;
assert FileUtil.toFileObject(res) != null
assert fs.findResource(res) != null;

If the last call to fs.findResource(res) returns null, then yes, it is wrong and should be fixed (I just wonder how do I simulate the problem and find out what is wrong).

Integrated into 'main-golden', will be available in build *201103110400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/da59d372501b
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: extra trace for #196425 -  AssertionError: no FileObject

Alexander, please, review the fix

QA, please, verify

Integrated into 'main-golden', will be available in build *201103120400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/c9988b90979b
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #196425: findResource does not refresh the state of disk to eliminate too often I/O

Fix safe and fixes the problem.
There are no performance problem.

terrible project (boost_1_43_0). Only on quick 4-core machine I can parse this project for about 1 hour.

Valera, does your comment means that the fix has been verified?

It is I can't reproduce this random bug in NetBeans trunk or rc1, but I missed a lot of time on project configuration

Verified
I can't reproduce exception in NetBeans IDE Dev (Build
cnd-main-5537-on-110315)

http://hg.netbeans.org/releases/rev/91e1ed437072
http://hg.netbeans.org/releases/rev/94f1a4fe2fa9
http://hg.netbeans.org/releases/rev/266da6ee7ccb

Verified in NetBeans IDE 7.0 RC1 (Build 201103180000)