Fixed at end of JavaOne

verified in build 060725

I am using the latest build and am still seeing these issues.

I was running a project through the debugger.  While my process was suspended 
at a breakpoint, I shutdown the IDE.

When I restart my IDE, the AppServer shows it is started, but all the folders 
in the JBI Manager fail to expand.  It seems that no service assemblies are 
deployed, no service engines or binding components are installed.

When I restart the AppServer everything then seems fine, the BCs, SE, and 
SErvice assembles are all up and running.

Solution:  Upon restart of the IDE, the AppServer should either be in 
the "stopped" stage, or should have actually restarted properly.

Ling there may need to be some clarification on this bug. It may not be the same
as the original description. 

For instance. Do you still get the InstanceNotFoundException when you try to
deploy a process following the shutdown and restart? If so, then this would be
the same as the original bug.

However, if you are able to deploy without exception then maybe this bug is not
the same as the original bug. Maybe this bug is closer to the other bug
described in bug 81516. Can you clarify please. 

I just watched this bug with APepin. We observed that the exception on
deployment is still occuring. So this bug is same as originally filed.

Also, we are looking to see if problem is perahps related to improper shutdown.
Normally, when you start AS from within IDE, the AS shuts down when IDE is shut
down. However, in this bug sceneario we observed that when the IDE is shut down
while attached to AS in debug mode, and then IDE is restarted, surprisingly the
AS is listed as still running in the runtime tab. This is part of possible
problem. Conducting more tests to confirm and clarify these observations.

Created attachment 32661 [details]
java processes which are still working after closing IDE

After closing IDE while debugging following processes are still working, see
attached java_processes file

It takes these processes long time to complete. After their completion user can
start IDE and AS will be stopped. After starting AS everything works fine. So
the only problem is to short these processes completion time to acceptable value.   

Unfortunately some problem occurs with undeploying of the bpel process which was
 deployed before closing IDE. Some exception could appear. It can be
IdexOutOfBoundsException 0>=0 or something like following:

JBIFW2460: Unexpected exception java.lang.NullPointerException occurred on
shutDown for Service Unit shutDown. The exception has no message text.
JBIFW2500: ServiceUnitManager stack trace follows:
java.lang.NullPointerException
        at
com.sun.jbi.engine.bpel.BPELSEDeployer.cleanUpAfterDeployment(BPELSEDeployer.java:340)
        at com.sun.jbi.engine.bpel.BPELSEDeployer.shutDown(BPELSEDeployer.java:272)
        at
com.sun.jbi.framework.ServiceUnitOperation.process(ServiceUnitOperation.java:159)
        at com.sun.jbi.framework.Operation.run(Operation.java:86)
        at java.lang.Thread.run(Thread.java:619)
|#]
 
javax.management.MBeanException: java.lang.Exception: <?xml version="1.0"
encoding="UTF-8" standalone="no"?><jbi-task
xmlns="http://java.sun.com/xml/ns/jbi/management-message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0"
xsi:schemaLocation="http://java.sun.com/xml/ns/jbi/management-message
./managementMessage.xsd"><jbi-task-result><frmwk-task-result><frmwk-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>FAILED</task-result></task-result-details><locale>en_US</locale></frmwk-task-result-details></frmwk-task-result><component-task-result><component-name>com.sun.bpelse-1.0-2</component-name><component-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>FAILED</task-result><message-type>ERROR</message-type><task-status-msg><msg-loc-info><loc-token>
</loc-token><loc-message>JBIFW2460: Unexpected exception
java.lang.NullPointerException occurred on shutDown for Service Unit shutDown.
The exception has no message
text.</loc-message></msg-loc-info></task-status-msg></task-result-details></component-task-result-details></component-task-result><component-task-result><component-name>com.sun.httpsoapbc-1.0-2</component-name><component-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>SUCCESS</task-result><message-type>INFO</message-type><task-status-msg><msg-loc-info><loc-token>JBIMA0500</loc-token><loc-message>Lifecycle
operation shutDown succeeded for Service Unit
SynchronousSampleApplication-com.sun.httpsoapbc-1.0-2.</loc-message><loc-param>shutDown</loc-param><loc-param>SynchronousSampleApplication-com.sun.httpsoapbc-1.0-2</loc-param></msg-loc-info></task-status-msg></task-result-details></component-task-result-details></component-task-result></jbi-task-result></jbi-task>
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.unwrapInvocationTargetException(MBeanIntrospector.java:283)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:210)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at javax.management.StandardMBean.invoke(StandardMBean.java:391)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:77)
        at $Proxy1.invoke(Unknown Source)
        at
com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:297)
        at
com.sun.jbi.management.system.EsbLifecycleAdapter.shutdownServiceAssembly(EsbLifecycleAdapter.java:305)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:77)
        at $Proxy1.invoke(Unknown Source)
        at
com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:297)
        at
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:287)
        at $Proxy13.shutdownServiceAssembly(Unknown Source)
        at
com.sun.jbi.management.system.EsbLifecycle.setServiceAssembly(EsbLifecycle.java:708)
        at
com.sun.jbi.management.system.EsbLifecycle.executeLifecycleCommand(EsbLifecycle.java:445)
        at
com.sun.jbi.management.system.EsbLifecycle.shutdownServiceAssembly(EsbLifecycle.java:374)
        at
com.sun.jbi.management.system.EsbLifecycle.shutdownServiceAssembly(EsbLifecycle.java:349)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:77)
        at $Proxy1.invoke(Unknown Source)
        at
com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:297)
        at
com.sun.jbi.esb.ui.admin.runtime.mbeans.AbstractUIMBeanESBImpl.invokeMBeanOperation(AbstractUIMBeanESBImpl.java:182)
        at
com.sun.jbi.esb.ui.admin.runtime.mbeans.AbstractUIMBeanESBImpl.invokeMBeanOperation(AbstractUIMBeanESBImpl.java:232)
        at
com.sun.jbi.esb.ui.admin.runtime.mbeans.JBIAdminCommandsUIMBeanCASImpl.shutdownServiceAssembly(JBIAdminCommandsUIMBeanCASImpl.java:688)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at javax.management.StandardMBean.invoke(StandardMBean.java:391)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:77)
        at $Proxy1.invoke(Unknown Source)
        at
com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:297)
        at
com.sun.enterprise.admin.jmx.remote.server.callers.InvokeCaller.call(InvokeCaller.java:56)
        at
com.sun.enterprise.admin.jmx.remote.server.MBeanServerRequestHandler.handle(MBeanServerRequestHandler.java:142)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.processRequest(RemoteJmxConnectorServlet.java:109)
        at com.sun.enterprise.admin.jmx.remote.serve|#]
r.servlet.RemoteJmxConnectorServlet.doPost(RemoteJmxConnectorServlet.java:180)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:397)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:278)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:240)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:179)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:73)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:182)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
com.sun.enterprise.web.VirtualServerPipeline.invoke(VirtualServerPipeline.java:120)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.invokeAdapter(ProcessorTask.java:667)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.processNonBlocked(ProcessorTask.java:574)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:844)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:287)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at
com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:75)
Caused by: java.lang.Exception: <?xml version="1.0" encoding="UTF-8"
standalone="no"?><jbi-task
xmlns="http://java.sun.com/xml/ns/jbi/management-message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0"
xsi:schemaLocation="http://java.sun.com/xml/ns/jbi/management-message
./managementMessage.xsd"><jbi-task-result><frmwk-task-result><frmwk-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>FAILED</task-result></task-result-details><locale>en_US</locale></frmwk-task-result-details></frmwk-task-result><component-task-result><component-name>com.sun.bpelse-1.0-2</component-name><component-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>FAILED</task-result><message-type>ERROR</message-type><task-status-msg><msg-loc-info><loc-token>
</loc-token><loc-message>JBIFW2460: Unexpected exception
java.lang.NullPointerException occurred on shutDown for Service Unit shutDown.
The exception has no message
text.</loc-message></msg-loc-info></task-status-msg></task-result-details></component-task-result-details></component-task-result><component-task-result><component-name>com.sun.httpsoapbc-1.0-2</component-name><component-task-result-details><task-result-details><task-id>shutDown</task-id><task-result>SUCCESS</task-result><message-type>INFO</message-type><task-status-msg><msg-loc-info><loc-token>JBIMA0500</loc-token><loc-message>Lifecycle
operation shutDown succeeded for Service Unit
SynchronousSampleApplication-com.sun.httpsoapbc-1.0-2.</loc-message><loc-param>shutDown</loc-param><loc-param>SynchronousSampleApplication-com.sun.httpsoapbc-1.0-2</loc-param></msg-loc-info></task-status-msg></task-result-details></component-task-result-details></component-task-result></jbi-task-result></jbi-task>
        at
com.sun.jbi.management.system.DeploymentService.changeSAState(DeploymentService.java:2375)
        at
com.sun.jbi.management.system.DeploymentService.shutDown(DeploymentService.java:1859)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:589)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        ... 103 more
  

Created attachment 32702 [details]
ArrayIndexOutOfBoundsException

I run app server on windows and really can not reproduce this bug.
App server : not in debug mode
bpel debugger : attached.

I tested shutting down IDE in 2 scenarios: 
1. No debugger attached
2. Debugger attached and a breakpoint is hit.

I watched the java processes, and when when IDE shuts down, the 3 processes stay
for about 10 sec and go away and I restared IDE and do undeploy and redeploy the
app and it works just fine. There is no difference between attaching debugger
and not attaching debugger.

I use Synchronous sample an set breakpoint at the first activity "sequence". Are
you using the 08/10's build -- The one shipped later which fixed the 2
showstoppers ? Unless there is a difference between platforms.

I run app server on windows and really can not reproduce this bug.
App server : not in debug mode
bpel debugger : attached.

I tested shutting down IDE in 2 scenarios: 
1. No debugger attached
2. Debugger attached and a breakpoint is hit.

I watched the java processes, and when when IDE shuts down, the 3 processes stay
for about 10 sec and go away and I restared IDE and do undeploy and redeploy the
app and it works just fine. There is no difference between attaching debugger
and not attaching debugger.

I use Synchronous sample an set breakpoint at the first activity "sequence". Are
you using the 08/10's build -- The one shipped later which fixed the 2
showstoppers ? Unless there is a difference between platforms.

The bug is still reproducible in RC1 build on both windows and linux platforms.
The one and only scenario is:
- create a new Synchronous sample application
- start AS
- deploy the application
- start debug session
- set a breakpoint in the BPEL source
- run test scenario 
- when the breakpoint is reached close IDE (click on a cross in the right-top
corner of the window)
Result: IDE closes but three java processes are still working. It takes them lon
time to complete (> 3 minutes). If user starts IDE before these processes
complete AS will look like working but JBI will not be alive.

After working with Mei on my machine and her machine, we found the problem.

Mei's machine is much faster then my machine and Sasha's machine. That is why
she can not reproduce this issue.

The reason Sasha and I keep seeing this problem is, when we restart IDE, the App
Server is started already. If we expand all the subnodes under JBI, we see
nothing. But the truth is, the app server is not really started. After a couple
of minutes, if refresh the app server, will see it is stopped. I found a couple
of exceptions in server.log file. I sent it to Mei. Will attach it in this
report later. After seeing the app server is stopped, restart the app server,
everything works fine again.

Created attachment 32905 [details]
attach the server.log file

The problem is that the App Server is taking MINUTES to shut down. Don't worry
about refreshing the IDE. You do not even need to restart the IDE. Follow the
scenario, after IDE shutdown, observe how long it takes for AS java processes to
disappear. 
Just observe the amount of time it takes for the AS java processes to
dissappear. I don't understand how the manchine speed can be so different that
for Sasha Pepin, the AS processes "hang around" for 5-6 minutes longer. 

Tried steps outlined above and at one point, I was able to replicate the issue 
where the Appication Server visual node contained the JBI sub-node but when 
expanded, it did not contain any information.  On my machine, it took about 
20+ seconds for the java processes to completely shutdown.   I believe the IDE 
when re-started, might just be reflecting the state of the AppServer and it's 
service engine(s) at the time it is creating the nodes (which may not reflect 
the actual state once the processes have time to completely shutdown).  Talked 
to Mei Wu and she thinks this might be in the backend side.  Note that the 
steps are not consistently reproducible.

Re-assigning to Mei Wu.

Using the original steps I was not able to reproduce:
- create a new Synchronous sample application
- stert AS
- deploy the application
- start debug session
- set a breakpoint in the BPEL source
- run test scenario and wait until the breakpoint is achieved
- close IDE
Result: After starting the IDE, the Appserver is shutdown.  Upon starting the 
Appserver, I am able to deploy new projects successfully.

Through analyzing server.log, BPELSELifeCycle has a while loop that could make
the shutdown process to hang until the maximum time allowed for component
shutdown expires, removed this loop and asked Hong to retest the binary, it
works fine.

Now it takes 20-25 sec. to shutdown AS, so I think the bug can be closed.
Verified in build 060905