More Power! — The Grid Compute Server Plug-in for NetBeans IDE
Jennifer Kotzen is a Sr. Product Marketing Manager for Sun Microsystems.
Jennifer, in a few words, what is Compute Server technology?
A compute server is a cluster (or a "grid") of distributed CPUs that operate in unison to solve parallel computing problems. The Compute Server project supports distributed parallel computing on the Sun Grid and offers the Compute Server technology as a download to support this usage model.
And you now offer a Grid Compute Server Plug-in for NetBeans IDE? What does this mean for Java developers?
Yes, the plug-in is a key deliverable of Sun's Compute Server project, which aims to help Java developers use the Sun Grid Compute Utility for the distributed execution of parallel computations. For Java developers this means, if you're trying to build a new application, and if significant parts of that application could be modeled as independent tasks that execute in parallel, then this plug-in may be of interest to you. It can help you build that application and also leverage the Sun Grid Compute Utility as a platform for its execution.
Can you refresh me briefly on the Sun Grid Compute Utility? What is it and why would developers want to take advantage of it?
The About Sun Grid page summarizes this pretty well. In a nutshell, the Sun Grid Compute Utility is a grid of servers accessible through the Network.com portal for the affordable price of $1/CPU-hr.
With this vast computing resource available as a pay-for-use utility, you don't need to buy, maintain, and keep your own grid of machines busy around the clock in order to be able to run jobs that require lots of CPU power. Once you register for an account at Network.com you can zip up and post your application and data, run a job, then download your results. It's really easy.
The Compute Utility, and indeed the Grid Compute Server Plug-in for NetBeans IDE as well, are best used today for grid-based parallel computing applications. That's a computing architecture that many industries are already benefiting from - it's widely used for modeling, simulation, data analysis and processing, image rendering and so on. For CPU-intensive batch-style applications Sun Grid is pretty compelling because the price point is so low and it's accessible over the Internet.
By the way, if developers are even just a little bit curious about the Sun Grid Compute Utility, there are a number of great programs underway right now to make this a really good time to do some experimenting.
What kind of programs, for example?
Well, there's a contest going on right now with big cash money prizes that will be awarded to developers of innovative applications for the Sun Grid - what's more, $25,000 US of that prize money will be awarded to developers who use Compute Server technology to build their applications. In addition, qualified developers may also be eligible to receive 100 free CPU-hours on the Sun Grid to test and run their apps. So, right now you could get free time on the grid, and win prize money for using that time to create an innovative grid app. If you want to learn more about what utility computing is really like, now is truly a prime time to do some exploration.
You mentioned that the Grid Compute Server Plug-in for NetBeans IDE is a key deliverable of Sun's Compute Server project - can you explain what you meant by that?
Sure. Compute Server technology includes three fundamental elements, and those are a programming model, an application execution framework, and a NetBeans plug-in module. The programming model defines a set of Java interfaces and interactions that are supported by the execution framework; together they enable developers to create quite sophisticated and highly performant applications for the Sun Grid using relatively simple Java code. Complementing this, the plug-in extends the NetBeans IDE with specialized tools that are useful for developers of Compute Server applications; you can think of the plug-in as a toolset that simplifies the processes involved in application development, testing, and debugging.
And what kinds of support does the plug-in provide to simplify the development of Compute Server applications?
Well, when you think about it, we're talking about creating an application that conforms to a particular programming model, one that you'd probably prefer to debug locally, even though it will eventually run on-grid, and one that requires uploading to the Sun Grid for execution and downloading of results later - that's a scenario that presents lots of opportunity for developer support! The plug-in helps with all of these things, and as a result makes it dramatically easier for Java developers to successfully create Sun Grid applications.
For instance, the plug-in's online help features a section on Compute Server design patterns to assist in application design. The plug-in also includes example applications and a Compute Server project creation wizard that automatically generates needed class templates to simplify application coding. These tools help developers use the Compute Server programming model effectively.
In terms of debugging support, the plug-in includes a special version of the execution framework that enables you to test and debug your code locally, using all of the standard NetBeans debugging tools, before trying to run it on the actual Sun Grid utility. Related to that, the NetBeans build tools are also customized by the plug-in to simplify the management of separate on-grid and off-grid build configurations. So you can, for example, easily use different native libraries on-grid than you use when debugging locally.
And finally, the plug-in facilitates both the packaging of Compute Server applications so they can be uploaded to the Sun Grid, and also the processing of output downloaded from the completed Sun Grid job. With the push of a button in the NetBeans IDE, everything that is needed to run your application on the Sun Grid -- that is, your code, the Compute Server execution framework, any input data, and even a job startup script -- it's all packaged automatically into a zip file for easy uploading to the Sun Grid. And once your job has completed and you've downloaded your results, the plug-in helps with unpacking and processing those from within NetBeans IDE too.
So, clearly Compute Server application developers should get the plug-in, but I assume that they will also need the programming model and execution framework too - how do they get those pieces of the technology?
All Compute Server technology is included in the Grid Compute Server Plug-in for NetBeans IDE download. So, that means you get the programming model interfaces and API documentation, the execution framework, and the NetBeans plug-in module, as well as documentation and example applications. Everything that you need to develop a Compute Server application is in there. All code is available in both source and binary form, and everything is released under the Apache License, v2.0.
You can download the technology from the Compute Server project homepage at computeserver.developer.network.com.
And the price?
No charge - Sun is contributing Compute Server technology to the Developer Community to encourage experimentation with the Sun Grid and development of applications that will run on it.
The technology includes the Compute Server programming model, the Compute Server execution framework, and the NetBeans plugin module. Note that all three components, as well as examples and documentation, are included as part of the Grid Compute Server Plug-in for NetBeans IDE download.
There is, of course, a cost associated with running your Compute Server applications on the Sun Grid Compute Utility, and that is figured at the $1/cpu-hour rate.
What if my processes run on a large data set; can my Compute Server application access data that is stored elsewhere from the grid?
Given that an application executing on the Sun Grid cannot currently access data off-grid, and that, alternatively, the time needed to move hundreds of gigabytes of data onto the grid would in most cases be prohibitive, applications that demand access to such large data sets are not well suited to run on the Sun Grid Compute Utility at this time. The utility is currently intended to be used primarily to run jobs that require no more than 10GB of Sun Grid resource to run, including both applications and data.
That said, if you need to create applications that will operate over such large volumes of data, you should not be deterred from investigating the Sun Grid and also the Compute Server project. In fact, our hope is quite the opposite: we are not deterred by hard problems and we want to work with developers who are like-minded. So, if you are pushing the envelope today - not only in terms of computing over very large volumes of data, but in other dimensions too - then you are the sort of developer we most want to work with. Go join the Sun Grid Developer Community and help us understand how to make utility computing work for you!
So the Compute Server project is pretty new - it was just started up with the release of the 0.1 version of the technology in March, coincident with the Sun Grid Compute Utility launch - but I understand that you've already put out three releases, the most recent of which came out just last week. This rapid release model is a bit different from what most of us are accustomed to - can you tell us what that's all about?
Yes, frequent technology releases are an integral part of the Compute Server project endeavor. We know that utility computing is a very new and innovative idea, and we know from our experience with many innovations at Sun that people will end up doing extraordinary things with new grid computing technology infrastructures, utility computing business models, and distributed applications architectures - people will soon be using these technologies in ways that we are not even imagining today. We acknowledge this with an eager desire to support the rapid evolution of ideas that we anticipate, and so, with the Compute Server project we are inviting developers into our thought process early and openly, so that we can learn together how best to evolve the technology as we move it forward.
And is this approach working?
Yes, we are very pleased with the way the model has been working so far. In the space of 3 months, we've been able to share 3 early access releases with folks in the Sun Grid Developer Community. We expect to put out a fourth release within the next month. The project mail alias is carrying lively discussion around work that people are currently doing, or work that they would like to do with the technology. As a result, we've already been able to incorporate feedback from developers into both the 0.2 and 0.3 releases, and we have new ideas for what we might do going forward.
So what's the latest, greatest news with the 0.3 release?
The 0.3 release is a perfect example of the kind of rapid and user-directed advancement that the Compute Server project engenders. In this release we introduced significant improvements to the programming model and the NetBeans user interface. Perhaps most significantly, we added new support for something we call "multi-phase jobs" which enable applications to be constructed as sequences of parallel execution phases, with the output of one phase becoming the input to the next. We believe that this enhancement will greatly expand the range of applications for which the programming model can be used, and will satisfy a broad range of user needs. We've also introduced a new, unified on- and off-grid I/O model, including support for off-grid input production. This I/O model complements the multi-phase enhancements and simplifies the programming model more generally. Also, responding to feedback from developers who have been using earlier releases, we further simplified development by removing type parameters (generics) from the APIs, integrating database extraction with the off-grid input production, and by supporting the use of different native libraries on- and off-grid.
To wrap up, is there anything else you'd like to tell us about the Grid Compute Server plug-in for NetBeans IDE?
Yes, in closing I'd like to reiterate that at Sun we believe that utility computing presents an incredible opportunity for developers. We want to make that opportunity as accessible as possible, and the Grid Compute Server Plug-in for the NetBeans IDE delivers it right to the doorstep of Java developers today. I'd like to encourage developers to take a look at it, try it out, experiment and have fun with it, to think of it as an entry point into the world of utility computing and to use it as a vehicle for learning about that, and of course to share their feedback with Sun and others in the Sun Grid Developer Community.
The Grid Compute Server Plug-in for the NetBeans IDE is available under the Apache License v2.0, and can be downloaded for free from the Compute Server project site at computeserver.developer.network.com.
More information about the Sun Grid Developer program, including applications for free CPU-hours and the Cool Apps Challenge, is available at www.developer.network.com.
Jennifer, thank you for the interview!