Tomasz, it's probably yours

I notice that this happens regularly after I rename a file. The IDE becomes unusable then, because the exception is
thrown every time I switch to or try type in the editor.

It's even worse. Restart doesn't help. I hed to delete the GSF/PHP index to get rid of the exception coming up.

I think Tor fixed this issue last night, can you try to reproduce it with the latest builds? (hopefully this is not possible) 

I'll try later today.

Unfortunately it is still reproducible, I added assertion and now the message looks like this: 

java.lang.AssertionError: PHP Index is refering to a non-existing file file:/Users/tomslot/NetBeansProjects/PhpTESTProject/web/newEmptyPHP.php
	at org.netbeans.modules.php.editor.index.PHPIndex.isReachable(PHPIndex.java:385)
	at org.netbeans.modules.php.editor.index.PHPIndex.getFunctions(PHPIndex.java:244)
	at org.netbeans.modules.php.editor.PHPCodeCompletion.autoCompleteExpression(PHPCodeCompletion.java:282)

The line numbers in your new stacktrace don't match what is in the trunk. They do however match what is in the 6.1
branch.  Did you try to reproduce this in 6.1?  My GSF fix is not in 6.1 yet. However the fix has now been both code
reviewed and QA approved so I will transplant the fix into 6.1.

The problem is still there; I'm investigating.

I found the problem; my patch yesterday was modified from

    if (PHPLanguage.PHP_FILE_EXTENSION.equals(file.getExtension())) { // NOI18N

to
  
    if (file != null && file.getFileObject() != null && "php".equals(file.getExtension())) { // NOI18N

That won't work for the reasons stated in the comment right above it; file.getFileObject() will return null in this case
(the file has been deleted) yet this method should return true such that GSF knows this file is relevant to the index
and it should attempt to clean up the index. (Also, getFileObject() here will be expensive during startup indexing).

Also, the file parameter will never be null.  I'm going to annotate the GSF apis with more @NonNull and @CheckForNull
annotations to make this clear.

By the way, I verified that by modifying the line to

        if (/*file != null && file.getFileObject() != null &&*/ "php".equals(file.getExtension())) { // NOI18N

the bug is now fixed - after renaming code completion from other files not only do not throw assertions, they show the
same functions in the original file but coming from the renamed file.

Answering Tor's question regarding the branches: Yes, I've been encountering this with the release61 builds. I'm working
with the PHP support that is now being developed on top of NB 6.1.

Tor I am terribly sorry for the embarrassing merge. This issue is now fixed.

http://hg.netbeans.org/release61/rev/5a0e83cdb993

Unfortunately I am able to reproduce it again, although it got somewhat random. It was working after renaming the file but came back after restarting the IDE.  
I also managed to reproduce it without restarting the IDE. Now the problem seems to be on GSF side, Tor please have a look at it...

I've tried reproducing it - but I can't. 

I have two files in my project. I have functions in one of them. I tried code completion from inside the file with
functions - and it finds the functions along with the right filename displayed. It also works from the other file. Then
I rename the file - and code completion works for both. And I rename again - it still works. Then I restarted the IDE,
and everything still works.

Can you come up with a set of steps to reproduce this? How often does it occur? Can you attach your project to reproduce
in case there's anything special about the files you're using?

The only condition is that the file being renamed contains some indexable data e.g. a function. It happens quite often. Have you tried it with the release61 
build?

Steps to reproduce: 

- create a new PHP project 
- in the index.php file define a PHP function. A sample content of the file :

<?php
function foo(){}
?>

- rename the file
- an assertion error is thrown each time call code completion is called inside the php block. It might happen only after restarting the IDE

I can reproduce today as well. I use the latest sources from release61. 

Open a php file, do a some changes, save it and rename. 

I can reproduce it with release61 too. This is the AssertionException I got:

java.lang.AssertionError: PHP Index is refering to a non-existing file
file:/home/honza/AU-stats/webapp/dashboard-for-date-range-old_1.php
	at org.netbeans.modules.php.editor.index.PHPIndex.isReachable(PHPIndex.java:385)
	at org.netbeans.modules.php.editor.index.PHPIndex.getFunctions(PHPIndex.java:244)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.listFunctions(SemiAttribute.java:419)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.visit(SemiAttribute.java:317)
	at org.netbeans.modules.php.editor.parser.astnodes.Include.accept(Include.java:91)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:121)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.scan(SemiAttribute.java:131)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.visit(DefaultVisitor.java:234)
	at org.netbeans.modules.php.editor.parser.astnodes.ExpressionStatement.accept(ExpressionStatement.java:71)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:121)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.scan(SemiAttribute.java:131)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:128)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.visit(DefaultVisitor.java:160)
	at org.netbeans.modules.php.editor.parser.astnodes.Block.accept(Block.java:93)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:121)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.scan(SemiAttribute.java:131)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.visit(DefaultVisitor.java:290)
	at org.netbeans.modules.php.editor.parser.astnodes.IfStatement.accept(IfStatement.java:105)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:121)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.scan(SemiAttribute.java:131)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:128)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.visit(DefaultVisitor.java:346)
	at org.netbeans.modules.php.editor.parser.astnodes.Program.accept(Program.java:89)
	at org.netbeans.modules.php.editor.parser.astnodes.visitors.DefaultVisitor.scan(DefaultVisitor.java:121)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.scan(SemiAttribute.java:131)
	at org.netbeans.modules.php.editor.nav.SemiAttribute.semiAttribute(SemiAttribute.java:460)
	at org.netbeans.modules.php.editor.nav.OccurrencesFinderImpl.compute(OccurrencesFinderImpl.java:92)
	at org.netbeans.modules.php.editor.nav.OccurrencesFinderImpl.run(OccurrencesFinderImpl.java:84)
	at org.netbeans.modules.php.editor.nav.OccurrencesFinderImpl.run(OccurrencesFinderImpl.java:66)
	at org.netbeans.modules.gsfret.editor.semantic.MarkOccurencesHighlighter.processImpl(MarkOccurencesHighlighter.java:161)
	at org.netbeans.modules.gsfret.editor.semantic.MarkOccurencesHighlighter.run(MarkOccurencesHighlighter.java:122)
	at org.netbeans.modules.gsfret.editor.semantic.MarkOccurencesHighlighter.run(MarkOccurencesHighlighter.java:79)
	at org.netbeans.napi.gsfret.source.Source$CompilationJob.run(Source.java:1242)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[catch] at java.lang.Thread.run(Thread.java:619)

FWIW, I could not reproduce the issue with the latest trunk build. I don't know how up-to-date the PHP stuff is there
though.

I tried with release61 both on Saturday and today and wasn't able to reproduce --- until by accident just now I had a
typo. And then I discovered that if I try to autocomplete with no prefix (instead of "fo" in foo as I had been doing all
along) I can reproduce the problem.

I'm investigating now.

This should be fixed in the trunk now, with changeset 44212f5824b2 .

Explanation of the problem and the fix:
When a file is renamed or deleted, GSF should clean up index entries corresponding to the deleted file.  It did that -
but it didn't actually delete the index document corresponding to the file - it was just empty.  The Ruby and JavaScript
index queries never ran into this since they don't try to check file existence before iterating over map results - and
since the map is empty they will obviously never try to construct index objects for anything found in the map.

I have fixed this now in GSF such that in LuceneIndex, if the documents field passed in to store is null (as opposed to
an empty list), this means GSF should not attempt to create an empty document for the URL. I also made sure that if any
GSF client such as PHP returns null instead of an empty document list, it will be converted to an empty list since the
meaning of an indexer returning null is that there is nothing to be indexed for the file. I still want an empty document
in the index to record that, such that on IDE restart we don't attempt to reindex this file is the filestamp hasn't changed.

I've tested this a bit and I don't see any regressions in startup fast reindexing etc - but at this late stage I would
really like others to test the IDE a bit too after this change.

One thing you can do in the PHP code if you don't want to push for this change in RC1, is to simply not assert that the
index in every map will exist:

diff -r 688ae9b6bb2f php.editor/src/org/netbeans/modules/php/editor/index/PHPIndex.java
--- a/php.editor/src/org/netbeans/modules/php/editor/index/PHPIndex.java	Mon Apr 07 17:57:43 2008 +0400
+++ b/php.editor/src/org/netbeans/modules/php/editor/index/PHPIndex.java	Mon Apr 07 10:39:36 2008 -0700
@@ -382,6 +382,9 @@
                 PhpSourcePath phpSourcePath = project.getLookup().lookup(PhpSourcePath.class);
                 if (phpSourcePath != null) {
                     File file = new File(new URI(url));
+                    if (!file.exists()) {
+                        return false;
+                    }
                     assert file.exists() : "PHP Index is refering to a non-existing file " + url;
                     
                     FileObject fileObject = FileUtil.toFileObject(file);


Anyway, if you want this in 6.1, please code review changeset 
http://hg.netbeans.org/main/rev/44212f5824b2
and I also need QA approval.

I thought it would be quite bad to work it around on our side, as it could mean letting the index grow with obsolete data and finally causing performance 
problems. Is it actually an issue? Considering the schedule it might be better just to work around. Let me have a look at the change set.

I reviewed the changeset, I think it will not cause regressions. I recommend to integrate it into 6.1.

I doubt that it would be a big performance problem. How many times in a typical project will you delete/rename a file?
Lucene searches thousands of documents very quickly (and this will only be triggered if you do an empty prefix search).
On the other hand, the url-to-filename lookup (in your isReachable method) is probably expensive with the various things
it's doing - path lookup, url-to-file etc. When I was stepping through the code I noticed that it was looking up the
same url over and over (something in the php runtime). Caching the most recently converted url there might make a big
difference.

As soon as you rev your PhpIndexer's version number the user will start over with a fresh Lucene repository in their
user directory, so this isn't some data which will stay stale for a very long time.

I implemented the changes recommended by Tor on the PHP side:

http://hg.netbeans.org/release61/rev/106b9445793d

So there is no need to modify GSF code in release1.

The fix has been ported into the release61_fixes repository.

See Issue 135059 for more details.