A distributed, content-addressed filesystem across the internet, not just in a home LAN, than can be mounted (fuse or whatever else), doesn’t require storing data twice, has a reasonable API and acceptable documentation.
Does something like that exist?
256 KB is the chosen default. so this is the best choice. however… i wonder…
if you share all 3900 finetune versions of llama 3.1 8b or all 2900 finetune versions of qwen 2.5 7b instruct
there is likely alot of matching chunks if you break the chunk size super small.
so it may not be the most optimal chunk size for sharing… but you could (potentially have waay more peers (because peeps are all hosting diff models with some shared matches between them)
** a more difficult question is wether to create ,efficient, highly compressed archive of specific popular sets of related models(“most-popular, starter packs”). this requires communication,coordination,effort to get it EXACTLY right.
IMO this is a trade for using centralized trackers and random uploaders.