Skip to content

issue/251 - change nt config to op config for kv caching #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wooway777 merged 1 commit into from
Mar 4, 2026
Merged

issue/251 - change nt config to op config for kv caching #252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ python scripts/test_ppl.py --model-path MODEL_PATH [--ndev NDEV] [--max-batch MA
```


- 选择是否使用九齿计算路径，默认为false，即不依赖九齿算子
- 选择是否使用kv caching，默认为false；在支持了此算子的平台(英伟达、阿里、天数、沐曦、海光、QY)可以使用
```bash
xmake f --ninetoothed= [true | false] -cv
xmake f --use-kv-caching= [true | false] -cv
```


Expand Down
2 changes: 1 addition & 1 deletion csrc/cache/kv_cache.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ StaticKVCache::update(size_t layer_idx,

auto device = k_cache_layer->device();

#ifdef ENABLE_NINETOOTHED
#ifdef ENABLE_KV_CACHING
infinicore::op::kv_caching_(
k_cache_layer,
v_cache_layer,
Expand Down
8 changes: 4 additions & 4 deletions xmake.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ set_toolchains("gcc")
add_includedirs("third_party/spdlog/include")
add_includedirs("third_party/json/single_include/")

option("ninetoothed")
option("use-kv-caching")
set_default(false)
set_showmenu(true)
set_description("Whether to complie NineToothed specifc path")
set_description("Whether to compile the path using the kv caching operator")
option_end()

if has_config("ninetoothed") then
add_defines("ENABLE_NINETOOTHED")
if has_config("use-kv-caching") then
add_defines("ENABLE_KV_CACHING")
end

target("infinicore_infer")
Expand Down