Bug 210053

The place in the source code where this happens is the same across multiple clean-build-deploy-run processes. It just changes for the same project in multiple occurrences of this problem.

If I switch compile on save back on then the error is back.

This is now 100% reproducible - even after undeployemnt, server shutdown, IDE shutdown, deleting of var/cache/index. No errors in the IDE log. It must be a known issue, I just found bug 178597.

Could you try one thing please. Shut down the IDE and go to your IDE user directory (location can be found in IDE's Help->About dialog) and delete folder {your-user-dir}/var/cache/index and restart IDE. It will cause rescanning (and also recompiling) of all your project source files which can take a while but I wonder whether that will fix your problem or not. Thanks.

Hi David,

I did this twice because I did not believe the result the first time:
- Checked that COS is on
- undeployed web project
- shut down server
- shut down IDE
- deleted {user-dir}/var/cache/index
- started IDE
- cleaned and built project (yes it compiled files, I could see that not only the ant task was displayed but also the number of files compiled)
- deployed the project
- run the project

In this process there were no errors in any logs in addition to the runtime error on the server java.lang.RuntimeException: Uncompilable source code - Erroneous sym type:
(<any>)void

So from here it looks like the compilation creates the uncompilable class file again.

There are already multiple issues in the bug system one of which claims to have a reproducing testcase. I will get back with the ID later.

My Questions (I am trying to help):
- Can you suggest a tool that searches a directory for "uncompilable source" code files? I think we have to be unconventional - perhaps something like findbugs or anything that uses byte code engineering libraries?
- Can you suggest any strategy to produce a testcase from a large project for this? If I touch the source code now then I will change all sorts of conditions and the case might disappear quickly. This DOES reproduce at my end for now. I have copied a snapshot of the project and my userdir but I will have to move on.

I don't think I can make a case from other issues except if I search for "Uncompilable source code" there are a few, and also a few articles on the web.

If the IDE does not absolutely need these troublesome files to function correctly then I have some serious doubts about whether they should be written/deployed at all.

Honza is the best guy to have a look into this I think as it is directly related to javac used by the IDE.

In this project four class files out of 980 contain the string "Uncompilable source code". So I could try to shrink the project step by step until I get a testcase for you.

But this takes too long if for every step I have to delete var/cache/index and re-start, and who knows whether this will work?

That is why I am asking for a strategy. I am waiting ...

One of the key reasons for introducing Compile/Deploy on Save was to allow starting of unfinished applications ("This piece of code is not finished yet, but I would like to run the application and test some unrelated parts.").

So the root cause is that the compilation that is being done on IDE start (with clean caches) apparently thinks that some of the classes are uncompilable, even if they should be compilable. But its next to impossible to say why without a reproducible testcase. So, if you could produce a small testcase from the project (or share the whole project on which the bug can be reproduced), that would enable us to fix this bug.

As for the strategy - I am not sure if this can be made much simpler in general case, there are too many variants. You can try copying the suspect file into a new project along with its transient dependencies (making the method bodies in the dependencies empty, so that the closure is smaller). But I cannot guarantee this will lead to small reproducible testcase.

Your effort is much appreciated.

Thanks.

I cannot see any benefit in running an application that is partially broken. I have never used this feature. I thought that there might be benefits for the IDE to function e.g. for refactoring and code completion but this does not seem to be the case. Perhaps if a test class is full of syntactical errors, being able to run single working methods has become valuable.

I am not really sure whether I should be trying to fins this bug which is in a feature that I don't want in the first place.

Deploying uncompilable code is crazy! The more I think about it, the more this looks like a bad hack.

Can't you give me anything to speed up my testing efforts? Any technical background, any logging, any patches?

It is a nightmare having to shut down the IDE, delete the cache index and wait for the IDE to re-build the cache and open the project.

Having to go through this even once for fixing my work if bug fixing is no consideration is an insult - we know it doesn't even work in this case. There needs to be a IDE support for this type of situation whether the root cause is fixed or not. There might be others. Would you please open an issue for that and cc me. Or open an issue to remove compilation of uncompilable code.

Many thanks.

> Can't you give me anything to speed up my testing efforts? Any technical
> background, any logging, any patches?

It mostly depends on what exactly you see (what exactly is the error reported for the "Uncompilable" file, btw? E.g. in the Tasks window.) There are very many possible causes, including rare bug-to-bug incompatibility between your JDK and the internal compiler, which is based on JDK7 and adheres more strictly to the Java Language Specification than many versions JDK6 javacs.

But, I can do that work if I can get the code - I assume that either Marian or Jiri (both CCed) could help with the code transfer.

> 
> It is a nightmare having to shut down the IDE, delete the cache index and wait
> for the IDE to re-build the cache and open the project.

I can feel the pain - but it is very hard to me to say how to make it easier when all I know so far is basically "it is broken".

> 
> Having to go through this even once for fixing my work if bug fixing is no
> consideration is an insult - we know it doesn't even work in this case. There
> needs to be a IDE support for this type of situation whether the root cause is
> fixed or not. There might be others. Would you please open an issue for that

Well, if you can suggest how such support should look like (technically - what does it do and how?), then we can talk about that. But, as I don't have any really implementable idea on how to generally do that, I do not see much value in creating such issue. The only thing that came to my mind is to pack everything (the sources, IDE cache, dependencies, everything) and send that for analysis.

> and cc me. Or open an issue to remove compilation of uncompilable code.

Let me separate two things here:
-compilation of uncompilable code into the IDEs caches is mandatory, otherwise the uncompilable class wouldn't be visible to most IDE features. Problems like the one you are describing would still be visible because of the red error badges. So the bug you are describing is important anyway.
-running the code even though it is uncompilable (or the IDE believes so) can be eliminated. In fact, you had to accept (at least once) a dialog saying that the project was compiled with error and asking whether the project should be run anyway. There are basically two ways to achieve that:
--always run the build script and the external javac when (re-)deploying. On the user level, this can be (to some level) achieved by disabling CoS/DoS (more support on the project level might make that nicer for the user, though).
--not allowing the user to run projects with errors, but still using the classfiles produced by the IDE compiler - but note that in your case, you wouldn't be able to deploy (in CoS/DoS mode), because the IDE thinks there are errors in your code. Also, this was explicitly requested, so this is guaranteed to upset some/many users.

One more thing that came to my mind: inside the var/cache/index, there are segment directories, and a "segments" file, which maps source roots to the segment directories. If you know into which source root the offending file belongs, you can try deleting only the specific segment directory, which should make the reindexing faster.

What I did in the past in situations like this is to create a patched version of a module which dumps lots of logging and send it to user and asked for the logs. After several exchanges of patched modules you usually are able to understand what's happening.

After reading all comments, I think we may be getting a bit closer to a solution. Inevitably there are some misunderstandings, mainly because I am not familiar with the NetBeans code, but let me summarise from my perspective anyway. My hope is that more understanding will result in:

a) an efficient ad hoc process for pinpointing this issue
b) an acceptable future process that lets a user recover from this scenario
c) better support for handling potential future errors in this area

Some points to consider:

- The project does compile without errors. There is absolutely no indication of any kind visually, including potential error markers anywhere in the IDE, and the log files are absolutely clean. The error occurs at runtime AFTER successful deployment, and only if the application hits the bad code - which might or might not happen. My observation is consistent with other reports on the web. The confusion is that the code fails at runtime while there is no error in the source code whatsoever.

- The error is unrecoverable. After shutting dowen the IDE, deleting the cache, deleting the build dir, starting, the IDE re-creates the error if it is in COS mode

- We have a simple discriminator which is a text search for "Uncompilable source code" in the class files

- It would help if while the IDE is running, parts of the cache belonging to a project or or all of it could be cleared without having to re-start the IDE, and without having to figure out which part that is. For this exercise and generally.

- It would be very helpful if the IDE would write to the output whenever it writes a class that is classified "Uncompilable source code". If you could attach a patch to 7.1.1 that would do this, then we could use this case to test it.

After all, we might get a small reproducible testcase that would be valuable to test any new functionality in this area. Unfortunately, I cannot send all the source code. I also have some doubt that it will actually reproduce as easily as we think because of potential platform differences and many other unknowns such as compiler versions. Even then, it takes a lot of communication and work to achieve the desired result. I have been there.

I have tried to reproduce the problem in a separate directory with the same project setup. It did not reproduce there. So there is no hope at all that I can ever produce a testcase ...

Even more, when I re-opened the original set of projects without having touched them, the problem vanished. But I am sure the problem will be back. Hopefully by then, you will have a plan of attack and and some tools that can pinpoint the problem.

There is only a class file with "Uncompilable source code" in nbproject\private\org-nbheaven-sqe-core-java\CompileOnSaveHelper\7AF863D6\ - about 4 weeks old.

When searching the snapshot copy of var/cache/index dir for files containing "uncompilable source code", I found

.netbeans\7.1.1\var\cache\index\s2\java\14\classes\javax\swing\JComponent$AccessibleJComponent.sig
.netbeans\7.1.1\var\cache\index\s2\java\14\classes\java\lang\Enum.sig
.netbeans\7.1.1\var\cache\index\s2\java\14\classes\java\lang\Integer.sig

The entries causing the runtime exception had bad classes in a project used in libraries, too. However the library project did not have any errors. From this is apparent that there was a cascading effect (bad classes referencing other bad classes).

It appears there are always a few erroneous files in the cache, but sometimes the count grows quite a bit. In my case, now it contains 

.netbeans\7.1.1\var\cache\index\s5\java\14\classes\javax\swing\JComponent$AccessibleJComponent.sig
.netbeans\7.1.1\var\cache\index\s5\java\14\classes\java\lang\String$CaseInsensitiveComparator.sig
.netbeans\7.1.1\var\cache\index\s5\java\14\classes\java\lang\String.sig
.netbeans\7.1.1\var\cache\index\s5\java\14\classes\java\io\PrintStream.sig

Should these erroneous files be in there? Why dont't they get replaced with good ones? It looks like the cache is growing in an uncontrolled way.

(In reply to comment #16)
That sounds very strange. Could you please describe your project setup? Do you have JDK (sources/as a library) somewhere on classpath? What are the source roots corresponding to "s2" and "s5" in the ".netbeans\7.1.1\var\cache\index\segments"?

Generally, there are two variants of the .sig files:
-for project sources and its non-binary dependencies, these are compiled classfiles, possibly with repaired compilation errors (i.e. with the "Uncompilable ..." string).
-for binary dependencies (e.g. JDK's rt.jar, project libraries for which the binary is preferred over the source, etc.) these are mostly classfiles, but without any method bodies. These are created when the original classfiles do not contain parameter names, and are used as a cache so that the parameter names are not parsed from the (attached) sources each time they are needed. These files are not (or should not) be copied outside var/cache/index.

So it is strange that you have "Uncompilable" in any sig file with name resembling a JDK class, as those should not normally contain any method bodies.

Thanks.

Created attachment 117327 [details]
Patched java.source module.

To be placed into ${netbeans.install.dir}/java/modules. The jar has been patched for debugging - please retain the original jar.

Created attachment 117328 [details]
Patched javac-impl.

To be placed into ${netbeans.install.dir}/java/modules/ext. The jar has been patched for debugging - please retain the original jar.

I have attached a patched version of the java.source module and the internal version of javac. Please make sure you have a copy of the original jars, as the debugging ones are not thoroughly tested. The changes include:
1. prints a log message when a tree is repaired from uncompilable to compilable
2. prints a log message for files that have been compiled with errors (based on some of the comments the set of files from 1. and 2. may not be the same, which is very surprising, but rather printing it twice)
3. print a log message when a file containing "Uncompilable source code" is copied into build/classes - similar messages is added to the status line
4. a warning dialog is shown before any deployment of code that is considered uncompilable by the IDE (only those fitting under point "2." are considered). This warning exists since the first CoS/DoS enabled release, but this change disallows disabling it.

Note that it is very common for uncompilable classes to be compiled into var/cache/index during development (save of uncompilable file, focus lost while the file is uncompilable, etc.) So when you see something strange, you will probably need to say which of these lines are not correct.

Thanks Jan, I have installed the patches.

In reply to comment #17:
s2=jar\:file\:/C\:/Program%20Files/Java/jdk1.6.0_30/jre/lib/rt.jar\!/
s5=jar\:file\:/C\:/Program%20Files/Java/jdk1.6.0_30/jre/lib/jce.jar\!/

I don't think that I have JDK sources on the classpath. Could you suggest any safe way to verify this? I have ant and maven projects.

I searched uncompilable sources on a different computer with different project and found uncompilable jdk sig files, too. However one needs to tweak Windows to find these files using the ordinary search:

- Open explorer
- Right-click the mouse on "My Computer", and select, "Manage".
- Under "Indexing Services", select "Properties."
- Select "Index files with unknown extensions" and click "Apply."

(In reply to comment #21)
> Thanks Jan, I have installed the patches.
> 
> In reply to comment #17:
> s2=jar\:file\:/C\:/Program%20Files/Java/jdk1.6.0_30/jre/lib/rt.jar\!/
> s5=jar\:file\:/C\:/Program%20Files/Java/jdk1.6.0_30/jre/lib/jce.jar\!/
> 
> I don't think that I have JDK sources on the classpath. Could you suggest any
> safe way to verify this? I have ant and maven projects.

These are index segments for binary roots, so presumably no JDK sources on CP.

Thanks for the investigation, I think I know what is the cause of this specific behavior. These "Uncompilable" strings in the body-less sig files are generated for bridge methods. These are used when e.g. class implements Comparable<String> to generate a bridge from the original compare(Object, Object) method into the implemented compare(String, String). These are benign in the caches for JDK, to my knowledge. I tried a simple example in sources:

package javaapplication9;

import java.util.Comparator;

public class JavaApplication9 implements Comparator<String> {
    public static void main(String... args) {
        ((Comparator) new JavaApplication9()).compare("12", "23");
        new JavaApplication9().compare("12", "23");
    }
     
    public int compare(String s1, String s2) {
        new Exception().printStackTrace();
        return 0;
    }
}

and did not see a problem with running that in the CoS mode, so this is probably not the root cause of the main problem you are seeing.

The IDE is very stable with the patches. It compiles uncompilable source code and then repairs it as soon as the errors are fixed.

I created a new bug 210864 which might be related to the patches. Just in case.

Closing and re-opening all projects fixed the problem in my case while anything else including deletion of cache index and re-starting the IDE did not. I would suggest NOT to do this because it removes the evidence which could otherwise be analysed with patches as supplied here for 7.1.

I am afraid that we need something to begin with - either a reproducible test case or at least the logs produced by the above patches. Note bugs like this are generally being fixed as their root causes are being found (e.g. bug #209661).

Happens again with

Product Version: NetBeans IDE Dev (Build 201205200400)
Updates: Updates available
Java: 1.6.0_31; Java HotSpot(TM) Client VM 20.6-b01
System: Windows XP version 5.1 running on x86; Cp1252; en_NZ (nb)

2 class files found containing string "uncompilable". I have not applied the patches to this build because I don't know whether they are compatible. After the previous experience, I am waiting for your instructions what to do with my current configuration. As you can see there are updates available but I did not do anything with them yet.

Created attachment 120250 [details]
IDE log files zipped

First, could you please backup the cache dir? May turn out to be useless, but its the only thing we have. The cache dir moved for 7.2:
http://wiki.netbeans.org/FaqWhatIsUserdir

Then, please try to look for files with names ending with ".err" in the cache which would correspond to the Java files from which the broken classfiles (same name and package structure). If these exist, some errors should be listed in them, which might give some clue. If these do not exist, that would be an interesting information too.

Thanks.

I backed up userdir when this happened. There are no .err files. Also I could not find any files in the userdir that contain string "uncompilable" in usedir as last time - only in project dir. Also I have not cleaned the project, not changed any source code, not touched it at all. No idea if I get new errors if I do that. Waiting for new instructions.

I will attach a project that reproduces a bug analogous to the original report. It is unclear whether that relates to the last reported problem, that might be more related to the bridge methods problem described in #c22, but I am not sure as I didn't find any reproducible test case for that.

To reproduce:
-unpack the attached project, open.
-the Main class is without error badge, but is repaired (contains throw new RuntimeException)
-create the required "Undefined" class
-the Dep class is updated, but the Main class is not and the throw new RE remains.

Created attachment 120568 [details]
A test project.

This might be interesting: The test project causes an OutOfMemoryError when opened as the first project in a new userdir. See issue 213860.

The above problem is fixed by:
http://hg.netbeans.org/jet-main/rev/1908fee51970
http://hg.netbeans.org/main/nb-javac/rev/f3aef7a3f77b
http://hg.netbeans.org/main/nb-javac/rev/8c7740fb544f
http://hg.netbeans.org/jet-main/rev/1f73b5c2a68c

More cases leading to Uncompilable source code fixed under bug #182009.

verified in trunk

The fix seems to be OK.

release72, nb-javac:
http://hg.netbeans.org/releases/rev/fe85e2a03678
other changes in NB:
http://hg.netbeans.org/releases/rev/ce29161d34d4

Integrated into 'releases', will be available in build *201206212341* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/ce29161d34d4
User: Jan Lahoda <jlahoda@netbeans.org>
Log: #210053: need to record class usages for classes used as method return types and variable types.

Integrated into 'main-golden', will be available in build *201206220002* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/1908fee51970
User: Jan Lahoda <jlahoda@netbeans.org>
Log: #210053: need to record class usages for classes used as method return types and variable types.

verified

While this contains fixes, I still experience these errors which I reported in issue 210053 which is still open.