So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?
What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model
There are thousands of resources for this information
https://youtu.be/ugKAkCVBcz0
https://www.fitmyllm.com/
https://meattacker.github.io/local-ai-model-recommender/best-local-ai-models-4gb-vram.html
Wow amazing stuff! Goes to show the gulf of knowledge between a noob just getting into local llama and you. Ty! I dont even know what to search for lol
Fit llm recommended me Qwen2.5 Coder? Come on!
No system is perfect, a good place to jump off from I use Qwen Flash Next because it would recommend 70B models for my hardware