lucene - Generating a custom Tokenizer for new TokenStream API using JFlex/ Java CC

Question

We are currently using Lucene 2.3.2 and want to migrate to 3.4.0 . We have our own custom Tokenizer generated using Java CC which has been in use ever since we started using Lucene and we want to continue with the same behavior. I appreciate pointers to any resources that deal with building a Tokenizer for new TokenStream API from grammar.

UPDATE:

I found the grammar used to generate StandardTokenizer at http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex?view=log&pathrev=692211. Modified grammar to suit to our requirements and generated java code using jflex http://jflex.de/

Answer 1

I found the grammar used to generate StandardTokenizer at http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex?view=log&pathrev=692211. Modified grammar to suit to our requirements and generated java code using jflex http://jflex.de/