tokenizeKeywords
function tokenizeKeywords(text): string[];Shared keyword tokenizer — exported so Fts5KeywordIndex queries are
tokenized identically to BM25Index for ranking parity.
Keeps combining marks (\p{M}) as token characters — Indic scripts
(Tamil, Sinhala, Hindi, …) encode vowel signs as combining marks, and
treating them as separators splits every word mid-syllable.
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
Returns
Section titled “Returns”string[]