Bug 238543

there are multiple overheads here:
1. maven surefire plugin itself only counts the test execution itself, but it does more work before and after the test is executed.
2. then there is overhead of maven itself, which needs seconds to initialize the IoC container (that's an overhead that's most visible when simple goals are executed, not long running builds) and the MavenProject initialization and loading, preparing the plugin execution.
3. then there's the overhead of JVM startup.
4. And finally the overhead of the IDE doing work before and during build execution.

all of the above are most visible when single test or test method is executed. exactly because the test execution takes such a short amount of time.

the only possible shortcut in our current solution is to use Compile on Save and save on plugin executions related to the project lifecycle.

out of the current box, we could be executing tests differently, in IDE only, skipping 1-3 more or less, but when we did before with ant based execution, it was failing often as there's no easy way of 100% reproduction of execution environment within maven.

not something we will be looking into in the near future I'm afraid.

with forked tests, maven surefire plugin additionally starts yet another JVM to execute the single test. (unless one configures the plugin not to fork at all or to fork for each test)

Isn't there a way you maybe could provide some sort of  plugin that would basically do this?

Maybe other people would join the endeavour to have some sort of 'Infinitest or Quickstart tests' running that would rely on CoS?

In the end this would definately would be a major asset in imroving speed and efficiency of developing code :)

(In reply to tomzi from comment #3)
> Isn't there a way you maybe could provide some sort of  plugin that would
> basically do this?
> 
> Maybe other people would join the endeavour to have some sort of 'Infinitest
> or Quickstart tests' running that would rely on CoS?

setting "use.old.cos.execution" property in your pom to "true" will run the old ant-based CoS execution that is in-IDE JVM possibly faster. But for a significant portion of the userbase they execute the tests in a wrong way. So the tradeoff I've chosen is speed for accuracy as accuracy is IMHO more important and even 0.1% of tests failing differently in the IDE matters.

Oh yes - I agree it is important to start tests 'correctly' too, but it's pretty much unusable for immediate response in agile and test driven development, because you spend so much time just waiting for the result.

Many of the eclipse users use infinitest even in maven projects to get almost immediate response on unit test. But in NB this is just not possible - you loose A LOT time (90%) just waiting for the test to start up.

It would be a great addition to the test framework to have both possibilities - like CoS - you could compile the maven way to get correct results or a good approximation that works great with JRebel ... for continous development.

The same makes sense for tests - it would be great to have some 'infinitest' feature for near immediate unit tests result on a class you are working on, showing restults right after you saved the class

*If* the continuous re-running is to be implemented, it should be IMHO based on a surefire plugin execution that doesn't exit and waits for us manage it from the IDE's VM.
In a sense similar concept to what scala does with the compiler - http://www.scala-lang.org/old/sites/default/files/linuxsoft_archives/docu/files/tools/fsc.html

Or you can try convincing the infinitest people that netbeans should be supported as well. I haven't found such issue in their bug report system.

It is very unlikely that this is going to be implemented on netbeans side given the resource constraints we have. 
<ironic>it's easier to wait for moore's law to catch up and improve the experience.</ironic>

(In reply to tomzi from comment #5)
> Oh yes - I agree it is important to start tests 'correctly' too, but it's
> pretty much unusable for immediate response in agile and test driven
> development, because you spend so much time just waiting for the result.
> 
> Many of the eclipse users use infinitest even in maven projects to get
> almost immediate response on unit test. But in NB this is just not possible
> - you loose A LOT time (90%) just waiting for the test to start up.
> 

<opinionated>
personally I've never completely understood why seconds matter. If one has deep enough understanding of the code and is not just randomly doing changes in hope the tests pass, most of the time is spent in reading and writing code. Running tests every 2 minutes does not have any additional value. Additionally I like to save the files frequently, not expecting them to be "testable" or "runnable" everytime. In that sense, something like inifinitest would just add noise for me, not signal.

but my perspective might be skewed as I loose 1 hour every day just by building netbeans after hg fetch on the notebook provided by the company
</opinionated>

(In reply to Milos Kleint from comment #6)
> *If* the continuous re-running is to be implemented, it should be IMHO based
> on a surefire plugin execution that doesn't exit and waits for us manage it
> from the IDE's VM.
> In a sense similar concept to what scala does with the compiler -
> http://www.scala-lang.org/old/sites/default/files/linuxsoft_archives/docu/
> files/tools/fsc.html
That sounds great - that would mean that Cos Would probably pick up changes and together with jrebel you just keep starting the test, I guess

> 
> Or you can try convincing the infinitest people that netbeans should be
> supported as well. I haven't found such issue in their bug report system.
Well - I didn't mean that they have to write the plugin/feature...
I'd prefer that the NB community or the NB staff would have some prototype that maybe would be open sources so the community could kick in to continue developing the plugin. Or even better that this feature will be part of nb. Or maybe maven figures out a way to start test a lot faster ...

> <opinionated>
> personally I've never completely understood why seconds matter. If one has
> deep enough understanding of the code and is not just randomly doing changes
> in hope the tests pass, most of the time is spent in reading and writing
> code. Running tests every 2 minutes does not have any additional value.
> Additionally I like to save the files frequently, not expecting them to be
> "testable" or "runnable" everytime. In that sense, something like
> inifinitest would just add noise for me, not signal.
> 
> but my perspective might be skewed as I loose 1 hour every day just by
> building netbeans after hg fetch on the notebook provided by the company
> </opinionated>
Well, when some of the eclipse people here wanted to try out NB, literally one of the first things they complained, was that test take forever to start.

Until recently it was 'Find usage'/Open Type/Open Resource took forever (on an i5 Laptop, since these are our Dev machines), but NB did A LOT of performance improvements there - which is awesome.

You are right - seconds usually don't matter 'SO' much, but 'Distraction' from development does. And if you write code - you want to concentrate solely on that. So idealy I'd like to work on a class and see unit test result immediately *without* pressing magic buttons to start, but see an icon showing green, red, ... whatever. So I write the new business functionality in the test and write the implementation until its 'green'. If the source breaks the test doesn't have to start. Or if I write a bugfix I'd like NB to find 'all unit tests' and start them in the background, without me having to 'Find usage' them and start every single one of them on my own -> That would be the perfect world :)

In the CodeRetreat2013 while pairing with other developers (Eclipse, IntellJ, Python) my NetBeans 7.4 was the slowest thing in the room. We pair-developed in a test-first style. Test feedback on other IDE's was under one second. NetBeans tested single class in around 10 secs (with SSD drive). It is a huge distraction.

Compile on save was and still is buggy and does not provide 100% correct behavior anyway. But I do not care about that - it behaves well most of the time. Other times doing "mvn clean install" solves the problem.

Going back to use.old.cos.execution=true is not for me as I got infected by the new way NetBeans constructs classpath for test execution in multi-modular Maven projects with new CoS. That is a really good job on behalf of NetBeans team.

Still looking ahead for NetBeans to have sub-second or sub-two-seconds way of executing tests for a single class.

p.s. when I say "tests" I mean "specification of intentions using JUnit". I never test any more, just specify intentions of code in a form of JUnit test. Writing specification sentences (in a form of JUnit test case method names) requires to make pauses of 10 secs. This makes NetBeans very slow editor for spec-writing.

there's really a handful of possible solutions:

1. execute tests by the IDE infrastructure. Unfortunately there is no common infrastructure (as known from Eclipse/IDEA). We've done the IDE->Ant-> Compile on save build script thing, but that's been fairly limited in functionality and results often didn't match those of surefire execution. We could likely abuse surefire itself for it, but the surefire plugin itself appears to do quite some work on it's own. And we could only use one exact version of surefire within the IDE, thus risking incompatibilities with the maven build.


2. With maven builds, there's no easy shortcut. All solutions basically boil down to pre-heating the maven build, have all models resolved and then keep on executing tests in a loop.
Pro - the overhead would be fairly low.
Con - would likely work fine for a single project only, if tests or classes are changed in multiple projects, the preheated stuff would not work.

The cut itself can be done in maven core codebase or in surefire (by providing some sort of goal that accepts configuration via System.in or Socket)

Experimentally, I've tried the maven core codebase cut listening on localhost port for connections providing test parameter values. Appears to work and cuts down the test execution under 1s. However I haven't really done the IDE part of feeding the server with the right test parameter values.

Created attachment 143580 [details]
experimental code executing surefire:test in loop

*** Bug 239473 has been marked as a duplicate of this bug. ***

Why does it work so well in all the other IDE's?

(In reply to tomzi from comment #12)
> Why does it work so well in all the other IDE's?

well, i don't have enough experience with it to know if it really does, som problems will likely surface as well. the major difference is that netbeans doesn't have a generic infrastructure for test execution or any excution for that matter. the "we use the cmd line buildsystems for building" mantra does have some drawbacks as well. this is one of them.

I want to encourage you to improve the current situation - I'd suspect it to not be trivial, but it's a major improvement, for all developers and makes Netbeans shine even brighter if all these major performance issues are knocked out and development becomes even more fun, from release to release!
Thx again for the great work you invest in these things!

turned the experimental patch into a project at github: https://github.com/mkleint/netbeans-continuous-surefire

do you have a .nbm for this?

Do you use it the same way as currently?

(In reply to tomzi from comment #16)
> do you have a .nbm for this?

not finished yet, only can be built from sources.

> 
> Do you use it the same way as currently?

You would execute a special action on project popup that would handle the test re-executions within one project only.

Any news here ;)

> Why does it work so well in all the other IDE's?

It doesn't. It comes with a huge trade-off: maven-incompatibility. That's why I chose NetBeans. I never was able to use maven projects inside Eclipse in any satisfactory manner.

Regarding the surefire compatibility issue: why don't you guys just ask the maven developers for an API that provides the internally used configuration values like the constructed classpaths?

In the end, I think it all boils down to this:

maven can do unexpected things and you cannot account for that. NEVER. Except by running maven itself. So the practical solution is to ignore (as suggested before) everything until all is compiled and copied into target/classes/ and target/test-classes/ (by maven). From that point on you can use compile-on-save. As far as I understand it, you just need the information about the classpaths. But even that is not true: you already have that information because I can see the deps in the Projects list.

This issue is very critical for me. The CoS solution does not have to be perfect -- when I changes resources, I re-run maven, so what.

The argument about running test units every few secs is not a too good one: there are other use cases like developing graphical routines where you *want* to see the results immediately beause you might be experimenting.

BTW, pertaining to JUnit, if you run "mvn -X test" you get the command for running the forked test JVM. Maven surefire does not even remove the temporary files. You can rerun that command by hand and it seems to work. And MUCH faster. So that could be one solution for the tests. Running non-test code is a totally different beast of course, mainly because maven itself is not really designed to do that. So that classpath stuff should be enough. And even in Eclipse you need to adjust all sorts of stuff to make maven work with it.... and it still has lots of problems there.

Another point: one could have a look at Jenkins' maven integration. Jenkins is pretty tightly integrated with maven and uses an "interceptor" etc.

I use 

@REM See http://zeroturnaround.com/rebellabs/your-maven-build-is-slow-speed-it-up/ Section 5
set MAVEN_OPTS=-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -DdependencyLocationsEnabled=false -noverify -XX:MaxPermSize=256m %MAVEN_OPTS%

as Maven-Options to improve the startup-time of Maven

It's reduces the test-execution time by 12%. At least at my computer (Maven 3.2.5, JDK8uX @ Win7 64bit)

Quoting from:

"How about using test setup data from surefire forked execution?"

(https://github.com/mkleint/netbeans-continuous-surefire/issues/1)

You can use

mvn surefire:test@default-test -DforkCount=1 -X

to extract all information to run the surefire tests. So, first test would run slowly, the subsequent tests quite fast -- until there are chianges to the maven configuration.

The forked VM gets stared with:

java -jar target/surefire/surefirebooter4040048659501196593.jar target/surefire/surefire7931714325250446210tmp target/surefire/surefire_04776260083470312610tmp

First file contains system props, second contains junit test environment setup (classpaths, tests to run, test config)., the third is a manifest-only jar. The MANIFEST contains full classpath and main class org.apache.maven.surefire.booter.ForkedBooter.

So why not just use this the information dumped by the forcibly forked execution of the surefire plugin?

I would write a Netbeans plugin for that if I had experience in at last a few of the related areas required to do it (integrating junit runs with netbeans, checking if maven config has changed, avoiding memory leaks in NetBeans, setting up junit class loaders etc. etc.)

Or is there, maybe, some maven plugin that transforms a maven project into some other form, ie. a netbeans project with a more direct junit execution?

Created attachment 158781 [details]
Configure a goal for the "test file"/"test project" action

(In reply to netbeans.89423 from comment #24)
> Or is there, maybe, some maven plugin that transforms a maven project into
> some other form, ie. a netbeans project with a more direct junit execution?

You can create a maven mojo (a fork of the surefire plugin), which does the execution. In NB you can map the goal of the new mojo to the NB actions "test file"/"test project". See the attached screenshot to see what I mean!

Opened an issue https://issues.apache.org/jira/browse/SUREFIRE-1235

_This is a Java 8 issue_. Just switching the JDK for the project to Java 7 restores performances. With Java 8 my unit tests take a really long time just to get started. With Java 7 running the test is instantaneous.