I suggest that a modifier key be used to determine if the dialog would
open as a modal or non-modal dialog.  If the dialog opened as a modal
dialog, the dialog would also have an apply button to apply changes
without closing the dialog.

Changing category to UI temporarily - this should get reviewed by HIE.

Note this is not as simple as it sounds - one would have to
handle cases such as:
 - Underlying property/node is deleted or otherwise ceases
   to exist - how to gracefully handle the situation
 - How the window system manages modeless property editor
   windows (workspace change, or in new window system,
   window group change)
 - No viable API contract for how a custom property editor
   should behave (how should it supply its value to the 
   property editor, when should it be applied, how should
   errors be handled, etc.).  Apply/Cancel dialogs are also
   difficult due to this - the API allows a custom editor
   to directly update the underlying property, and there
   is no contract as to its behavior.  The end result is
   that it would be easy to produce custom property 
   editors that fail in strange ways (like trying to
   modify attributes of a deleted file - how should an
   open custom property editor indicate that it wants to
   go away?, etc.).  An API 
   should provide contracts that make it hard to fail.
   Support for custom property editors in 
   java.beans.PropertyEditor is, unfortunately, provided
   more as an afterthought bolted onto the property editor
   concept.

Opinion:
Does this request represent a real need?  Absolutely.
Should the issue be resolved via property editor infrastructure? 
Probably not.

In the interest of the long-term longevity of our codebase,
further overloading the property editor concept is probably
not the way to go - as has been noted, over-reliance on 
the property sheet reduces usability.  We could go in the
direction Steve suggests, but it would require a more
robust API for custom property editors, at the very least.

Could we approach the problem from a different direction - 
perhaps we need a well-defined means of providing something
akin to "floating palettes" (ala Photoshop or other apps)
which provide usable access to commonly reconfigured 
properties?  It's also worth asking the question of why
we have properties that a user must repeatedly reconfigure - 
that may be the underlying problem.

In other words, if the user will need to reconfigure 
something frequently, why put the property sheet in between
them and a UI to reconfigure it?

In effect, we already have the "floating palette" concept - 
the property sheet is one.  Perhaps an alternative solution
would allow some component other than the property sheet to
be provided in the property sheet area, which would allow
the user to easily configure things that they will need to
reconfigure repeatedly.

Steve, I'm sorry to not be agreeing wholeheartedly that
non-modal property editors are the solution to the problem.
Could you attach some more details of the specific use case
 - what the property is, why it needs to be reconfigured 
often?  And do you know of other similar cases that would
help make the case that the problem is
common enough that we ought to provide a better 
infrastructure than property editors for doing it?

I filed this issue after working on the a generic vcs profile
(Subversion).  I spent so much time clicking on a property, resizing
the property editor, editing a property in the editor, closing it,
testing, and repeating the process over and over.  If the property
editor was non-modal, it would have saved so much time.

I recalled using the Form editor in a similar way - especially with
custom code (pre initialize, etc) as well as with custom javabeans. 
Actually, take a simple example like the color property editor.  You
have to close the editor to see the changes.  If it was non-modal,
these sort of rapid changes would be much easier to do.

I understand all of the issue you have raised.  Maybe there is a way
to address them?  Could the non-modal dialog listen for property state
changes?  If the property (or whatever the dialog is for) is deleted,
then the property editor is closed - maybe with a prompt to the user
that the dialog is no longer valid.  

I have seen this behavior in some application and it was elegant but I
can't place where!  I can't emphasize emough how much time this could
save.

Most properties allow in-line editing with the need for modal customizer.