--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.