Currently, the Editor Code Completion SPI does not allow a developer to customize the sorting, selection, and filtering algorithms. The attached patch provides flexibility in these areas while fully preserving both the original behavior as the default and avoids any breaking API changes.
This patch introduces the CompletionController interface (provided by CompletionControllerProvider) to provide much of the behavior of the code completion drop down. When no CompletionControllerProvider is explicitly registered for the current mime type, DefaultCompletionControllerProvider.INSTANCE is used internally, and this provider creates a BaseCompletionController instance that uses the original IDE behavior in its implementation of each method.
Conceptually, the CompletionController alters the previous notion of "selected" as it applies to the completion list. A completion list now has 3 properties to represent the selection state.
1. (CompletionItem or Integer index) Best Match: When the completion drop down appears, this element is the "best" item in the completion drop down. In the previous implementation, this would simply be referred to as the selected item (at a selected index).
2. (Boolean) Selected: This flag specifies whether or not the Best Match is initially selected in the completion drop down. In the previous implementation, this was always true.
3. (Boolean) Unique: This flag specifies whether or not the Best Match is a unique match, and is used for triggering the instantSubstitution feature. The default implementation comes from the original in CompletionImpl: a match is unique if there is only 1 CompletionItem in the drop down, the prefix is a case-sensitive match to the CompletionItem, and the caret is at the end of the identifier.
The following methods are delegated through the CompletionController:
* sortItems: Sorts a list of CompletionItem items for display in a drop down.
* getSelection: Gets the initial selected item for the completion drop down. The return value is a SelectedCompletionItem containing the 3 properties described above. SelectedCompletionItem.DEFAULT can be used when no good match is available and is configured as index 0, not selected, and not unique.
* defaultAction: Executes the default action associated with the selected item. In the default implementation, this method calls CompletionItem.defaultAction for the best match item.
* processKeyEvent: In the default implementation, this method calls CompletionItem.processKeyEvent for the best match item.
* render: In the default implementation, this method calls CompletionItem.render.
* instantSubstitution: In the default implementation, this method calls CompletionItem.instantSubstitution for the unique item.
Created attachment 113028 [details]
3 patches applied in order to main-silver
To ensure that these changes do provide the flexibility my platform application demands, I used the modified version of the Editor Code Completion API to implement my full completion filtering/selection/replacement algorithm as described in the following message:
NB 7.1 is already feature frozen, so we could think about your changes for NB 7.2.
BTW: Is there an implementation of your enhanced 'filtering, selection, and replacement algorithms' for Java code completion available somewhere for download? Your idea seems interesting and I would like to test the prototype. Thanks.
Created attachment 113239 [details]
Apply after the previous 3 patches to fix NPE
Created attachment 114584 [details]
Apply after above to fix another NPE
Created attachment 114586 [details]
Apply after above to add CompletionProvider.USER_QUERY_MASK (no breaking changes)
Without introducing any breaking changes to the current API, I added the ability to specify user information in the value returned from CompletionProvider.getAutoQueryTypes. Any bits from CompletionProvider.USER_QUERY_MASK which are returned from getAutoQueryTypes will be passed to CompletionProvider.createTask. I'm currently using this feature to (manually) track the value of CompletionImpl.explicitQuery, which is critical information for my code completion algorithm.
Created attachment 114896 [details]
I figured out how to use MQ... yay! Attached a single patch of all my changes to the API, applied against a recent rev of main-silver.
Created attachment 115396 [details]
Updated code completion support
An extra function call from a separate test made it into the previous patch, fixed in this one.
Created attachment 116107 [details]
Updated code completion support
Updated to address a compatibility issue with 7b4644780ee1 (which fixed #208223).
1) CompletionControllerProvider - Why the CompletionTask is passed as an argument to the createController(...) method?
- Is CompletionController supposed to call query(...), refresh(...) or cancel() on the passed CompletionTask instance?
- For each code completion invocation, each active CompletionProvider creates its own CompletionTask instance. Looking at your implementation inside CompletionImpl.getController() - only the CompletionTask instance created by the first CompletionProvider is passed. Why the CompletionTask instances created by other CompletionProviders are ignored?
2) CompletionController - What is the use case for the sortItems(...)? Is it a possibility to implement case-insensitive sorting that you mentioned in some of your previous e-mails?
- Is sorting method really tied to the particular mime-type? Isn't it rather a user preference?
- Concerning getSelection(...) method - isn't the CompletionProvider itself a better authority to mark one of the created CompletionItems as "best matching"? CompletionProvider knows the language semantics, it can resolve particular context in which code completion is invoked, it knows all the items created by it? What additional information a CompletionController could have to better decide about the "best matching" item?
3) BaseCompletionController - Why is it an SPI class? Is it supposed to be subclassed?
4) CompletionItemComparator - Why is it an SPI class? Is it supposed to be subclassed?
5) CompletionProvider - What is the particular use case for COMPLETION_QUERY_MASK, USER_QUERY_MASK, and RESERVED_QUERY_MASK?
- What particular information needs to be transferred from CompletionProvider.getAutoQueryTypes(...) to CompletionProvider.createTask(...)?
6) CompletionResultSet - What is the use case for getAnchorOffset(), and getItems() methods?
- Currently, for each code completion invocation - every CompletionProvider active at the given context obtains its own CompletionResultSet instance (via CompletionTask) to collect its own results (items, anchor offset, etc.). Which additional API client (besides CompletionProviders) needs an access to these data and how does it get access to the CompletionResultSet instances?
7) AsyncCompletionTask - What is the use case for getQuery() and getComponent() methods?
- Which API client needs an access to AsyncCompletionQuery? What about queries created by other CompletionTask implementations?
I'll hold off on making the changes (and updating the patch) proposed by this comment until you get a chance to consider the proposals.
3) BaseCompletionController: Due to the extent of my modifications, I actually implement the CompletionController interface in its entirety. For slight alterations it would be easier for a user to extend BaseCompletionController. Due to the complexity of even a basic implementation of this class, I thought it appropriate to give users a starting point.
4) CompletionItemComparator: I create an instance of CompletionItemComparator with modified sub-Comparator instances. CompletionItemComparator implements the proper sequence of primary/secondary comparators for sorting completion items. If CompletionItemComparator is not an SPI class, then users will likely have to duplicate its functionality (including the PriorityComparator and TextComparator nested classes) if they want to extend this functionality. Obviously as long as they have a way to implement CompletionController.getComparator(int) then they'll have some option available to address requirements in this area.
- As long as the PriorityComparator and TextComparator classes remain accessible, any generic "sort by / then by" Comparator<T> wrapper would work. I don't know of one in the standard library, but I've only been programming in Java for a few months so I'm not an expert on its libraries.
5) CompletionProvider: I needed a way to get a flag from getAutoQueryTypes to createTask. In particular, I currently need to know if the user explicitly invoked the completion or if it was triggered automatically. Rather than restrict the implementation to only providing this boolean value, I reserved the high 16 bits for user information in the result of getAutoQueryTypes. As an example, someone may want to additionally distinguish between completions automatically triggered by a '.' operator and those triggered by typing a letter.
- The USER_QUERY_MASK shows the bits the user is free to pass from getAutoQueryTypes which then are passed back to createTask.
- The RESERVED_QUERY_MASK shows the bits reserved for the implementation.
- The COMPLETION_QUERY_MASK is a subset of RESERVED_QUERY_MASK showing flags used by a completion query (as opposed to documentation or tool tips).
2) CompletionController: The sorting, filtering, and selection features of code completion popups are fundamentally independent operations, although different implementations may link one or more of them together in various ways.
- Sorting could be a user preference, but it assumes the IDE implementation won't overlook necessary features. In the future, I will likely create an extension which overrides the default implementation for all languages unless overridden by one specific to a language. See  and  for another sorting algorithm that someone may want to implement.
- Only one item can be selected, but there can be multiple instances of CompletionProvider. There clearly needs to be a central controller for calculating the selection. Under the current policy of selecting the first item in sorted order, this controller is implicitly defined.
1) CompletionControllerProvider: I actually need to call getQuery() on the provided task. Considering the component is provided as another argument to createController(...), it might make sense to change the task parameter to a query parameter (of type AsyncCompletionQuery).
- I agree, it's appropriate and more complete to pass a List of the tasks (or queries based on the previous point).
6) CompletionResultSet: If the Completion SPI is further extended to allow an extension to change the way the completion popup is presented, the getAnchorOffset() and getItems() methods will be necessary. I don't currently use them and they could probably be removed for now.
7) I needed the getQuery() method as part of my answer to #1. I added getComponent() for completeness. If we change the parameter list of CompletionControllerProvider.createController as described in #1, I won't need either of these methods with my current code.
 "Learning from Examples to Improve Code Completion Systems" http://www.monperrus.net/martin/Learning-from-Examples-to-Improve-Code-Completion-Systems.pdf
 "Towards A Better Code Completion System by API Grouping, Filtering, and Popularity-Based Ranking" http://serl.clarkson.edu/site/wp-content/uploads/2011/08/BCC-RSSE10.pdf
ad 1, 7) CompletionControllerProvider: What particular method on the AsyncCompletionQuery instances you want to call from the CompletionControllerProvider??? (AFAIK - the only public method is isTaskCancelled()).
Another problem is that not all CompletionTasks have to be instances of AsyncCompletionTask (either a CompletionProvider running synchronously could exist or it can ensure asynchronous execution in its particular way).
BTW - if you say you need an access to AsyncCompletionQueries - isn't it another argument for CompletionProviders to mark their best matching items?
ad 2) Concerning sorting - we already thought about a possibility to register custom CompletionItemComparators via MimeLookup. However, there is a question how many different comparators could be implemented based on the CompletionItem's getSortPriority() and getSortText() methods?
Concerning 'best matching item' selection - is your CompilationController implementation for ANTLRWorks 2 able to select the 'best matching' item using the CompletionItem's public API only? (I mean without any additional information on the CompletionProvider and/or CompletionItem implementation)?
ad 5) CompletionProvider - please note that ANY of the active CompletionProvides can initiate automatic code completion invocation - without setting any user query bits.
ad 6) CompletionResultSet - we should not change SPI without the actual usecase, so let's ignore these changes for now.
ad 5) CompletionProvider: If the user query bits are not set, my completion controller uses a default behavior which works like the current defaults. The special bits are used when a triggered completion is ambiguous between a declaration and a reference to prevent the standard completion selectors from completing a new definition with a reference to an existing item. If the flags are missing, it treats the completion as a reference only. This behavior matches the previously supported behavior which means an CompletionProvider added in an extension that doesn't support these flags will behave as expected.
ad 2) Sorting: My completion controller works with any CompletionItem implementation. The only behavior which is not part of the Editor Code Completion SPI is related to setting the Selection.isSelected(). The only behavior supported before this patch is isSelected()==true, and my controller only changes this for a CompletionItem which implements an optional non-SPI interface providing other information about the item's state and behavior.
ad 1, 7) CompletionControllerProvider: The fact that not all CompletionTask instances must be AsyncCompletionTask clearly shows why CompletionControllerProvider.createController(...) must take a List<CompletionTask> instead of List<AsyncCompletionQuery>. My controller does not call methods directly on AsyncCompletionQuery, but allows for some optional behavior if (query instanceof MyExtendedQueryInterface).
If two CompletionProvider instances each provided their "best matching item", how do you determine which to select?
This old bug may not be relevant anymore. If you can still reproduce it in 8.2 development builds please reopen this issue.
Thanks for your cooperation,
NetBeans IDE 8.2 Release Boss