A distributed, content-addressed filesystem across the internet, not just in a home LAN, than can be mounted (fuse or whatever else), doesn’t require storing data twice, has a reasonable API and acceptable documentation.

Does something like that exist?

    • refalo@programming.dev
      0·
      1 year ago

      My experience with IPFS over the years has been abysmal, and I think people have said the protocol design cannot sustain any more growth, which is not even that big yet at all.

      You also cannot realistically search for files reliably by its hash, because of how files are divided into smaller pieces, whereby the method of dividing can change between clients, making the hashes incomparable. BitTorrent v2 solves this to my understanding, but almost nobody uses it for some reason.

      Often times you need to wait several minutes for IPFS to find a file, assuming it ever finds it, which sometimes fails even on two boxes next to each other.

      • leanleft@lemmy.mlEnglish
        1·
        3 days ago

        256 KB is the chosen default. so this is the best choice. however… i wonder…
        if you share all 3900 finetune versions of llama 3.1 8b or all 2900 finetune versions of qwen 2.5 7b instruct

        there is likely alot of matching chunks if you break the chunk size super small.

        so it may not be the most optimal chunk size for sharing… but you could (potentially have waay more peers (because peeps are all hosting diff models with some shared matches between them)

        ** a more difficult question is wether to create ,efficient, highly compressed archive of specific popular sets of related models(“most-popular, starter packs”). this requires communication,coordination,effort to get it EXACTLY right.
        IMO this is a trade for using centralized trackers and random uploaders.