Skip to content
whispcatPublic

About

A static site search engine built for content that mixes English with Chinese or Japanese

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

kensaku

CI Demo License: MIT

kensaku (検索, "search") is a static site search engine built for content that mixes English with Chinese or Japanese. A command-line tool turns JSON or Markdown into one index file. A 147 KB (gzipped) WebAssembly runtime then searches that file in the browser, with no server involved.

Try the live demo, which searches 2,000 Kubernetes docs pages in English, Chinese, and Japanese.

The kensaku demo showing results for the query 'Servoce 载均', with 'Service' and '载均' highlighted in Chinese titles and excerpts

The query Servoce 载均 has a typo and only the middle two characters of 负载均衡 ("load balancing"). Typo tolerance matches Service, and the bigram index finds 载均 inside 负载均衡, giving the full results as in Service 负载均衡.

Why kensaku

Chinese and Japanese are written without spaces between words. Search engines usually rely on a dictionary to guess word boundaries, and dictionaries disagree. When a query crosses a boundary the dictionary chose differently, the page can go missing from the results.

kensaku skips word segmentation for these scripts and indexes bigrams instead: every pair of adjacent characters, stored with its position. 负载均衡 becomes 负载, 载均, and 均衡. A query is split the same way, and kensaku checks that its bigrams appear next to each other and in order. So every page containing the exact query text, at least two characters long, is a top-tier candidate before ranking, wherever a dictionary would have put the word boundaries. The test suite checks this exhaustively for every substring of its fixture text.

Latin text gets the features you'd expect from search-as-you-type:

  • exact word matches,
  • prefix matching on the word you're still typing,
  • typo tolerance of one edit for words of five or more letters.

A mixed query such as Service 负载均衡 needs both parts to match. Ranking uses BM25, with title words weighted three times as heavily as body text, and matches are highlighted in titles as well as excerpts.

Numbers

Measured with the demo's benchmark page on an Apple M4 Pro running macOS 26.6 and Chrome 153. Your numbers will vary with hardware.

Measure Result
WASM runtime 147 KB gzipped (366 KB raw)
JavaScript loader and wrapper 5 KB gzipped
2,000-page index 6.2 MB raw, 4.1 MB gzipped (3.1 KB and 2.0 KB per page)
Index load and validation about 115 ms
Search latency 0.2 ms median, 0.4 ms at p95

The index stores titles, URLs, and short excerpts, but not full page bodies. bench/README.md explains how the corpus is built and how to reproduce these measurements.

Quick start

Download the CLI for your platform from the latest release, or build it with Cargo:

cargo install --git https://github.com/whispcat/kensaku kensaku-cli

Build an index and try a query:

kensaku build content/ -o public/search/index.bin --stats
kensaku search public/search/index.bin "Rust 検索"

kensaku build accepts any mix of the following:

  • JSON files holding an array of { "id"?, "title", "url", "body", "excerpt"? } records.
  • Markdown files.
  • Folders of Markdown, scanned recursively in path order.

For Markdown, single-line YAML front matter can set id, title, url, excerpt, and draft: true. Without it, the title comes from the first heading or the file name, and the id and URL come from the file's path. Drafts, hidden files, code blocks, HTML, and Hugo shortcode markup are left out of the index. The visible text of links, inline code, image alt text, and shortcode text and title attributes stays searchable. Excerpts are capped at 300 bytes, and bodies are indexed but not stored.

Run kensaku --help to see the other commands, inspect and dump-terms.

Add search to a site

Copy the files from the browser archive (kensaku-browser-VERSION.tar.gz on the releases page) into your site, next to the index:

public/search/
├── index.bin
├── kensaku.css
├── kensaku.js
├── kensaku_wasm.js
└── kensaku_wasm_bg.wasm

Then add the search elements to a page. The stylesheet is optional.

<link rel="stylesheet" href="/search/kensaku.css" />
<input
  id="kensaku-input"
  class="kensaku-input"
  type="search"
  placeholder="Search"
/>
<p id="kensaku-status" class="kensaku-status" role="status"></p>
<div id="kensaku-results" class="kensaku-results"></div>

<script type="module">
  import { mountKensaku } from "/search/kensaku.js";
  await mountKensaku();
</script>

mountKensaku accepts these options:

Option Default Meaning
indexUrl index.bin next to kensaku.js Where to fetch the index
input, results, status the ids above CSS selectors or elements
limit 10 Results per query, from 1 to 100
debounce 100 Milliseconds to wait after typing
resultBaseUrl the current page Base for relative result URLs
onSearch none Called with { query, payload, elapsedMs } after each search

The server must send .wasm files as application/wasm. Many static hosts, including GitHub Pages, don't compress binary files. If yours doesn't, run gzip -9 index.bin and point indexUrl at index.bin.gz; kensaku detects gzip and decompresses it in the browser. Results are rendered as text only, and only http and https links become clickable.

How it works

flowchart LR
  src["JSON / Markdown"] --> cli["kensaku CLI"]
  cli --> core["core: analyze and index"]
  core --> file[("index.bin")]
  file --> wasm["WASM runtime"]
  wasm --> js["kensaku.js"]
  js --> page["search box"]
Loading

The native CLI and the browser runtime share the same Rust core, so a query returns the same results in both. docs/design.md covers text analysis, ranking, the binary format, and the reasoning behind the main design choices.

Path Contents
core/ Text analysis, index builder, binary format, query engine
cli/ The kensaku command and JSON and Markdown loading
wasm/ WebAssembly bindings
web/ Browser wrapper, stylesheet, and demo page
bench/ Test fixtures, benchmark corpus report, benchmark page
xtask/ Build, test, demo, corpus, and release automation

When to use something else

kensaku loads the whole index up front, which suits sites with a few hundred to a few thousand pages. It has no filters, facets, or index sharding. It doesn't stem words or support Korean, and it indexes Markdown rather than built HTML. If you need those, or integrations with site generators, a mature tool such as Pagefind is a better fit. kensaku makes sense when you want one small, inspectable index and predictable matching for mixed Latin and CJK text.

Contributing

CONTRIBUTING.md covers setup, the cargo xtask commands, tests, and releases.

License

kensaku is available under the MIT license.

The demo and benchmark corpus use pages from the Kubernetes website, by the Kubernetes contributors, under CC BY 4.0.

About

A static site search engine built for content that mixes English with Chinese or Japanese

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages