Uses of Class
uno.anahata.asi.gemini.tokenizer.LocalTokenizerProcessor

Packages that use LocalTokenizerProcessor

Package	Description
uno.anahata.asi.gemini.tokenizer	Provides the custom Byte Pair Encoding (BPE) and merges tokenizer engine for Gemini.

Uses of LocalTokenizerProcessor in uno.anahata.asi.gemini.tokenizer

Fields in uno.anahata.asi.gemini.tokenizer declared as LocalTokenizerProcessor

Modifier and Type	Field	Description
private final LocalTokenizerProcessor	LocalTokenizer.tokenizer

Fields in uno.anahata.asi.gemini.tokenizer with type parameters of type LocalTokenizerProcessor

Modifier and Type	Field	Description
private static final Map<String, LocalTokenizerProcessor>	LocalTokenizerLoader.localTokenizerProcessorCache

Methods in uno.anahata.asi.gemini.tokenizer that return LocalTokenizerProcessor

Modifier and Type	Method	Description
static LocalTokenizerProcessor	LocalTokenizerLoader.getSentencePiece(String tokenizerName)

Constructors in uno.anahata.asi.gemini.tokenizer with parameters of type LocalTokenizerProcessor

Modifier	Constructor	Description
(package private)	LocalTokenizer(com.google.genai.proto.SentencepieceModel.ModelProto modelProto, LocalTokenizerProcessor tokenizer)