Release v0.7.0 · ggml-org/llama.cpp
github.com
Overview llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory (--moe-cache-mib, up to ~2.2x decode speedup wh...

--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

  • 9tr6gyp3@lemmy.worldEnglish
    1·
    14 days ago

    Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.