When I first saw the package javadoc of the lexer
package, I had
     problems to find out what classes are
designed for what. I knew that
     there are two things - the lexer and the
incremental algorithm. But the
     TokenConstants class was totaly unrelated to
anything - now there is a
     comment in Javadoc, but I am still not able
to realize the reason of
     putting this into separate class...

     The second problem I had was with
TokenIterator. It starts with word
     Token, so I immediatelly connected it with
Token, TokenCathegory and
     TokenId - but it is not true, it is connected
with the algorith. Please
     for the sanity of future readers of the API
do something with this. That
     is what I want to emphsize.

     I suggested you to make the TokenIterator
inner class of the algorithm
     or merge it with the algorithm to make the
clear cut between lexer and
     algorithm part.