So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?
What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model
I’ve experimented with a lot of that range of models. The sweet spot is 35-50k of context. Anything more and the response quality degrades, or the model times out trying to answer, or just keeps churning till it becomes nonsense. I had a model respond with “SimiSimiSimi” for 3 pages of content because I overloaded the context to about 130k tokens. Was really disappointing to see.
I was asking Kimi-K3 that very question and one of the possible things was a local model for VTT. I think looking at specialised models like that might be the best use case.
Another one I was thinking about last night was local semantic search through embeddings. I’m not sure if that’s actually feasible, but it would certainly be useful for when you have terabytes of PDF.
There are thousands of resources for this information
https://meattacker.github.io/local-ai-model-recommender/best-local-ai-models-4gb-vram.html
Wow amazing stuff! Goes to show the gulf of knowledge between a noob just getting into local llama and you. Ty! I dont even know what to search for lol
Fit llm recommended me Qwen2.5 Coder? Come on!
No system is perfect, a good place to jump off from I use Qwen Flash Next because it would recommend 70B models for my hardware
I believe Pocock only recommends going up to 100k now, last I heard. But for a lot of simpler tasks 10-20k will be plenty if you don’t have a super long prompt + documentation for it to parse first. Your inputs are really what decide how much context you need.