I think this is very important.

During my use of GlassFish with NetBeans since version 2, I cannot remember any release where the server support wasn't broken. This includes 7.1.2 with all patches applied. Most of the problems would have been avoided had there not been a timeout.

This is more than a timing issue. The IDE actually prevents the server from starting up which it would normally do.

From the log:

INFO [glassfish]: __locations timed out. 3 of 3
java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
	at java.util.concurrent.FutureTask.get(FutureTask.java:119)
[catch] at org.netbeans.modules.glassfish.common.CommonServerSupport.isReady(CommonServerSupport.java:698)
	at org.netbeans.modules.glassfish.common.StartTask.startDASAndClusterOrInstance(StartTask.java:364)
	at org.netbeans.modules.glassfish.common.StartTask.call(StartTask.java:192)
	at org.netbeans.modules.glassfish.common.StartTask.call(StartTask.java:91)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)

Please consider asking the user in a dialog to confirm options a) kill the server or b) keep waiting. There is already a dialog, it is just the wrong dialog.

With this solution there is no need to add a timeout configuration parameter.

This code was completely changed in 7.3.
Can you try to reproduce this issue with last 7.3 build from
http://bertram2.netbeans.org:8080/job/web-main/8360/artifact/nbbuild/NetBeans-dev-web-main-8360-on-20120830-full.zip ?

If so, please send me strack trace from this version and I can try to find out how to configure some start timeout for local server startup im 7.3

That build does not have a timeout but I cannot see deployed applications from the services window. NetBeans asks me in a pop-up dialog for an admin user and password but I don't know what to enter.

It asks for administrator's user name and password to access GF admin interface. Try to supply them and let me know if it works.

I have already told you. I don't know what to enter. NetBeans has never before asked me for this. The server instalation is automatic, from NetBeans IDE 7.2.

I tried user admin and anonymous and tried 3 different passwords, admin, adminadmin and "" (nothing).

What do I need to enter for user id and password?

It looks weird. I was configuring GF 3.1.2 and 3.1.2.2 many times with NetBeans trunk last 2 months and it worked fine for me.
Default installation is without password and without SSL. With user/password enabled default user was "admin" and default password "adminadmin".

Try to log to GF admin console to check your real admin user and password.

Thanks for checking.

But this issue is about fixing the startup problem. Please, please create a dialog that provides the option to wait, with another interval, as many times as the user wants, or let the USER kill the server. Two options in a dialog. I have been filing bugs for the same thing many times. Why can't we have this commonsense solution? Everything else is a hack. It is not acceptable to have the IDE kill the server after a timeout. I don't know how someone can get this creazy idea. There is a need for waiting some reasonable time (timeout) before letting the user control the flow, but there is no need to configure this timeout.


Regarding the password issue:

I have two installations of build netbeans-trunk-nightly-201208300001-windows.exe on different computers. I have not been able to see deployed apps in the server node because of this password dialog. On one computer there is no problem with the same server when I use NetBeans 7.2 release. There was no problem with netbeans-trunk-nightly-201208290001-windows.exe, either.

Perhaps I should raise a separate bug for the password problem. But I don't think that I should because I would be very surprised if other people had not the same problem. Please ignore this problem for a while to save some time.

OK. I'll check what I can do for you. I'll try to make this timeout configurable value in server properties and adding kill command may also be quite easy change.

Just one more note to old problem. It's waiting for __locations command. It's very simple query which you can verify manually:

Start your glassfish manually:
nb:/users/tomas/WS/gfr3122/glassfish> bin/asadmin start-domain domain1
Waiting for domain1 to start .........
Successfully started the domain : domain1
domain  Location: /users/tomas/WS/gfr3122/glassfish/domains/domain1
Log File: /users/tomas/WS/gfr3122/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.

Then run locations command from your browser:
http://localhost:4848/__asadmin/__locations

It should return sg. like
Exit Code : SUCCESS
and small HTML table with PID, some server directories and uptime.

Please let me know how long did it take to start your server and how long did it take to get __locations response. Was it asking for some username/password?

Now back to CommonServerSupport.isReady code which is cause of timeout exception.
I'll be fixing it in 7.3 where locations call should be switched to new code.

We missed it somehow so I did it now. I'll push this change tomorrow after some additional testing. Please try that manual
 http://localhost:4848/__asadmin/__locations
test to see if it's working for you.

The server takes 48 seconds to start up and the location command succeeds immediately after that without asking for a password.

"C:\Program Files\glassfish-3.1.2.2\bin\asadmin" start-domain domain1
Waiting for domain1 to start ..............................................
Successfully started the domain : domain1
domain  Location: C:\Program Files\glassfish-3.1.2.2\glassfish\domains\domain1
Log File: C:\Program Files\glassfish-3.1.2.2\glassfish\domains\domain1\logs\server.log
Admin Port: 4848
Command start-domain executed successfully.

GlassFish __locations AdminCommand command report

Exit Code : SUCCESS
C:/Program Files/glassfish-3.1.2.2/glassfish/domains/domain1
Config-Dir 	C:\Program Files\glassfish-3.1.2.2\glassfish\domains\domain1\config
Instance-Root 	C:\Program Files\glassfish-3.1.2.2\glassfish\domains\domain1
Domain-Root 	C:\Program Files\glassfish-3.1.2.2\glassfish\domains\domain1
Uptime 	55875
Base-Root 	C:\Program Files\glassfish-3.1.2.2\glassfish
Restart-Required 	false
Pid 	2128

OK, thanks a lot!

changeset:   232470:49b7b5353b85
summary:     #199099 - __locations admin command call switched to GF Tooling SDK.

This small change will switch code responsible for __locations admin command call in 7.3.

this commit won't solve your problem but it's a good way to start. But think I can reproduce your problem now.

Integrated into 'main-golden', will be available in build *201209120001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/49b7b5353b85
User: Tomas Kraus <TomasKraus@netbeans.org>
Log: #199099 - __locations admin command call switched to GF Tooling SDK.

I see that Vince already did some change to fix this issue on 20th April.
It should be already fixed in 7.3 so all we need from you is to verify this with 7.3

Download last build from http://bits.netbeans.org/dev/nightly/latest/
It must be at least 201209140001 where I added possibility to change administrator's user name and password.
Install that and after GlassFish settings are imported, make sure that you have admin user and empty password in properties.

Actually there is 3 hours timeout in 7.3 so it should be enough for you. :)

How do I get the IDE to import GlassFish settings into an existing dev userdir? Or do I have to start a new userdir and get the IDE to import from 7.1.2?

Is it ok to work with an existing glassfish-3.1.2.2 installation?

Yes, you can register existing GlassFish using wizard from Services tab.

Created attachment 124406 [details]
IDE and server log files in zip file

I installed Build 201209150001 and updated the user id and password in properties as advised. Startup in debug mode failed 3 times (I did not try more times). Then I unregistered the server and re-registered it. Still startup failed.

I see nothing in the logs except
[#|2012-08-31T19:05:35.546+1200|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.container.common|_ThreadID=35;_ThreadName=Thread-2;|User [admin] from host localhost does not have administration access|#]

No timeout exception in the logs, ...there is nothing indicating that something went wrong.

What was IDE behaviour? Also please try to send me thread dump from NB JDK after GlassFish startup got stuck - maybe it will point to something.

Created attachment 124472 [details]
Thread dump

Created attachment 124473 [details]
Dialog

Created attachment 124474 [details]
log window screen shot

I see an exception in the IDE output window for the server log which I did not see before - very strange. I searched the entire server install directory for strings in this - no results.

Created attachment 124475 [details]
log files in zip file

I'm switching this to P2 to not have it as BETA1 blocker bug.

I believe that Vince already fixed this issue some time ago setting timeout to 3 hours and actual problem is how to verify it on bht's Glassfish where we are experiencing some issues.

As P2 it still has high enough priority for me to solve it in near future - in following week or two.

I've seen this problem on JB's laptop and it was definitely P1. It happened about two days ago running fairly recent dev build of IDE (I will ask JB to state which IDE build was it). We did clean start of the IDE with fresh GlassFish and all what happened was that after a minute or two of GF booting up the IDE would display above dialog - "the GF failed to start" and there was not error in GF log! Hard to fix it when you do not know what's wrong. I've never seen that before.

Was it happening with every single attempt to start GlassFish? What exactly did you see in the GF log? Was startup process finished?

I can add some logging into the whole startup task and following checks to see more information about the whole starup process.

But this issue will take longer time to resolve.

I still see this with the NetBeans 7.3 Dev nightlys (downloaded from the NB.org
site)
http://bits.netbeans.org/download/trunk/nightly/2012-09-19_00-01-28/

Installing the bundled GF and then trying to deploy a really simple REST
project will 4 out of 5 times, fail with nothing more than a message that
Glassfish Failed to Start. The App server will take up to 5 minutes to return
at times.

This is on a brand new Win7 laptop (Oracle issued Lenovo).

Trying to start Glassfish from the Services / Servers window results in the
same issue, so it's not tied to the Deploy task.

The interesting point for me is that if I download Glassfish separately, and
Netbeans from the internal Hudson, they work fine together.  It's only when
it's the complete install with .exe installer that seems to be display my
issues.

I'll see if I can get a log window copy the next time I try and run that
distribution.

I really like NetBeans and GlassFish and the NetBeans GlassFish server support. But it appears to me from the hundreds of bugs that I have filed so far, GlassFish bugs have had too much of a share. It is complex because GlassFish is not under NetBeans control and that perhaps more than doubles the chance of failures. Please consider that if not enough resources are spent on this then it is quite possible that the quality drops below acceptable levels. I keep thinking why I am always getting these bugs. My installations are always NetBeans full installation (everything) out of the box on Windows, on a computer with 3GBytes of RAM and 1.9 GHz single core CPU. My app is medium size JavaEE app so that would correspond with a largeer app on a faster computer. And it's been like that for the last few years. I don't do fancy things and I use sound routines e.g. re-starting the PC every day. Anything special would have been spotted long ago. 

### If you cannot see these failures while I always can then that fact should be examined whith high priority. QA.

> I can add some logging into the whole startup task and following checks to see
> more information about the whole starup process.

Yes, please do that Tomas so we can investigate fast. While this problem exists since NB 7.0, it seems to be more severe recently.

GlassFish Tooling SDK:
----------------------
changeset:   180:3f1a2dadd28f
summary:     Bug# 199099 - Logger renamed to org.glassfish.tools.ide and added finest logs to trace Runner execution.

changeset:   181:c2a858784ce2
changeset:   182:98975043d396
summary:     Added toString method to Java VM Version class instance.

changeset:   183:df52115f5ab5
changeset:   184:a96f71a8bea6
changeset:   185:f963da1ef7eb
summary:     Cosmetic logging changes.

NetBeans:
---------
changeset:   233957:af8890d5c00b
summary:     #199099 - Extended logging to trace Glassfish server startup.

FINEST logging for GlassFish module and GlassFish Tooling SDK library can be turned on by adding "-J-Dglassfish.level=0 -J-Dorg.glassfish.tools.ide.level=0" into netbeans_default_options in netbeans.conf

Integrated into 'main-golden', will be available in build *201209220001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/af8890d5c00b
User: Tomas Kraus <TomasKraus@netbeans.org>
Log: #199099 - Extended logging to trace Glassfish server startup.

Created attachment 124747 [details]
log files in zip file

netbeans-trunk-nightly-201209220001-windows.exe
Added "-J-Dglassfish.level=0 -J-Dorg.glassfish.tools.ide.level=0"
into netbeans_default_options in netbeans.conf

The extended logging (see comment #30) was transplanted to beta, because this changeset was needed as a part of fix for bug 217734.

Removing 73beta-stopper-fixed - this is NOT fixed in beta. Only additional logging is in beta right now.

From what I can remember - this is not recorded in the logs - the IDE kills a healthy server process.

I have seen this before in previous NetBeans versions.

From what I see there are multiple issues that cause this behaviour and that is why a normal human being cannot immediately figure out what is happening:

1) the IDE kills a healthy server process.
2) the IDE does not log the fact that it kills the server process
3) the IDE fails to detect that the server is running



If we only fix 3), and this is what has been happening many times before, then we will have to deal with 1) and 2) in the future many times more.

There could be any number of reasons why the IDE fails to detect that the server is not running.

Repoduces reliably at my end when my web application is already. In contrast, the server starts after I clean the project.

Repoduces reliably at my end when my web application is already deployed. In contrast, the server starts after I clean the project.

Extended logging should be in http://bits.netbeans.org/dev/nightly/latest/ available now.

Please add "-J-Dglassfish.level=0 -J-Dorg.glassfish.tools.ide.level=0"
into netbeans_default_options in netbeans.conf and attach IDE log to this bug.
We should see much more now.

Faulty healthy process kills shall be seen in the logs as well as whole starting process and server status detection result.

I do agree with you - there is an issue with server status check and also I exactly know where process is being killed and that there was no notification.
I would like to rewrite whole server status check code later (7.4 or 8.0 or whatever will come in the future) but for 7.3 it's too late to do such a large change.

Created attachment 125184 [details]
logs from Build 201210010929 in zip file

Created attachment 125218 [details]
Logs from my development environment

Here are logs form my local 7.3 build:
 * netbeans.log - full log from start trough GlassFish start and stop commands.
 * plugin.log - plugin related lines filtered using
                grep -E '\[org.glassfish.tools.ide\]|\[glassfish\]' \
                netbeans.log > plugin.log

And netbeans.conf contains my logging settings at the beginning of netbeans_default_options variable.

In your logs I see nothing related to GlassFish process startup.
I was expecting at least something like

FINEST [org.glassfish.tools.ide]: Java VM /usr/java/jdk1.6.0_30/bin/java executable version 1.6.0_30
FINEST [org.glassfish.tools.ide]: Starting GlassFish 3.1.2.2 using Java VM /usr/java/jdk1.6.0_30/bin/java and arguments  -Djava.awt.headless=true -Djava.security.policy=/users/tomas/WS/gfr3122/glassfish/domains/domain1/config/server.policy -Dosgi.shell.telnet.maxconn=1 -Dfelix.fileinstall.disableConfigSave=false -Dfelix.fileinstall.poll=5000 -Djava.endorsed.dirs=/users/tomas/WS/gfr3122/glassfish/modules/endorsed:/users/tomas/WS/gfr3122/glassfish/lib/endorsed -XX:MaxPermSize=192m -Dosgi.shell.telnet.port=6666 -XX:NewRatio=2 -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory -Djava.ext.dirs=/usr/java/jdk1.6.0_30/lib/ext:/usr/java/jdk1.6.0_30/jre/lib/ext:/users/tomas/WS/gfr3122/glassfish/domains/domain1/lib/ext -Dgosh.args=--nointeractive -XX:PermSize=64m -Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as -Dfelix.fileinstall.bundles.startTransient=true -Dfelix.fileinstall.dir=/users/tomas/WS/gfr3122/glassfish/modules/autostart/ -Dfelix.file.install.bundles.new.start=true -Djava.security.auth.login.config=/users/tomas/WS/gfr3122/glassfish/domains/domain1/config/login.conf -Dosgi.shell.telnet.ip=127.0.0.1 -Dfelix.fileinstall.log.level=2 -Djavax.net.ssl.keyStore=/users/tomas/WS/gfr3122/glassfish/domains/domain1/config/keystore.jks -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver -Djavax.net.ssl.trustStore=/users/tomas/WS/gfr3122/glassfish/domains/domain1/config/cacerts.jks -DANTLR_USE_DIRECT_CLASS_LOADING=true -client -XX:+UnlockDiagnosticVMOptions -Xmx512m -jar /users/tomas/WS/gfr3122/glassfish/modules/glassfish.jar --domain domain1 --domaindir /users/tomas/WS/gfr3122/glassfish/domains/domain1

You can see whole domain.xml arguments processing, Java VM call to start GlassFish and following checks to determine when server is up.

But in your log there are just periodic server state checks http://localhost:4848/__asadmin/__locations and http://localhost:4848/__asadmin/version in messages.log.2.

Created attachment 125243 [details]
ogs from Build 201210010929 in zip file. details turned on

Sorry, my previous log files were without glassfish logging options

changeset:   235891:a89c5e3b8e9b
summary:     #199099 - Better logging of location command response.

changeset:   235951:ba4f239196cb
summary:     #199099 - Keep retrying to read server status for up to 10 minutes with 2 seconds delays while servr is not ready yet.

This may fix your problem.
Thank you for the logs, I was finally able to reproduce this with breaking up GlassFish to make it keep sending "V3 cannot process this command at this time, please wait" messages in Manifest responses.

This response is sent while server is not fully initialised but administration interface is already accepting HTTP queries.
Related GlassFish V3 code is in core/kernel/src/main/java/com/sun/enterprise/v3/admin/AdminAdapter.java where this message is being generated.

Please let me know it that works for you. Our GA team will be unable to reproduce this issue without hacking GlassFish:

Index: src/main/java/com/sun/enterprise/v3/admin/AdminAdapter.java
===================================================================
--- src/main/java/com/sun/enterprise/v3/admin/AdminAdapter.java (revision 52756)
+++ src/main/java/com/sun/enterprise/v3/admin/AdminAdapter.java (working copy)
@@ -202,15 +202,16 @@
         Payload.Outbound outboundPayload = PayloadImpl.Outbound.newInstance();
 
         try {
-            if (!latch.await(20L, TimeUnit.SECONDS)) {
+              latch.await(20L, TimeUnit.SECONDS);
+//            if (!latch.await(20L, TimeUnit.SECONDS)) {
                 report = this.getClientActionReport(req.getRequestURI(), req);
                 report.setActionExitCode(ActionReport.ExitCode.FAILURE);
                 report.setMessage("V3 cannot process this command at this time, please wait");            
-            } else {
-                if (!authenticate(req, report, res))
-                    return;
-                report = doCommand(requestURI, req, report, outboundPayload);
-            }
+//            } else {
+//                if (!authenticate(req, report, res))
+//                    return;
+//                report = doCommand(requestURI, req, report, outboundPayload);
+//            }
         } catch(InterruptedException e) {
             report.setActionExitCode(ActionReport.ExitCode.FAILURE);
             report.setMessage("V3 cannot process this command at this time, please wait");                        

This will make GlassFish to respond with "V3 cannot process this command at this time, please wait" forever. For testing purposes I can make for example 2 minutes counter there. :)

Integrated into 'main-golden', will be available in build *201210130002* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/a89c5e3b8e9b
User: Tomas Kraus <TomasKraus@netbeans.org>
Log: #199099 - Better logging of location command response.

Thanks and congratulations. The startup worked first time.

Perhaps GlassFish needs to be fixed too? Is this timing issue recorded under the GlassFish project?

This seems to be correct behaviour of GlassFish.
Also NetBeans should work with older GF v3 so I can't change the way this response is handled in plugin.