Avalon local LLM

[-rick-]

11 minutes ago, SteamyTea said:

I am quite interested in this.

I can understand needed a fair bit of RAM to run the LLM.

But is there something special about the OS that it needs a massive amount.

I know OSs are RAM hungry these days, unlike 35 years ago when an image file was larger in RAM than the OS took up.

Can the OS be stripped back to the very basics of what it was originally designed to do?

 

MacOS will use less than 4GB IIRC. But decent LLMs are very memory hungry so I think @Pocster was more focussed on the RAM needed by the LLM not the OS. The smallest 'useful' models might be 16GB and then you need a ton more for context windows + LV cache. These models will still be far behind anything from the main commercial providers. More memory = better model + more context.

[SteamyTea]

16 minutes ago, -rick- said:

MacOS will use less than 4GB IIRC

Less than I thought, and similar to Windows, which is a bit messy with RAM.

Many years ago I used Puppy Linux, that ran totally in RAM, including all the applications. Was pretty good on my old laptop that had only 4 MB. Screen died eventually.

Often wonder how it would perform on a more modern machine.

I think DSL works the same and that was only about 50 MB for everything.

[MikeSharp01]

9 hours ago, Pocster said:

I’ll say it again , it’s insane ! 

Yes but not, as yet, unlike most technological innovations in the history of technology in that it enables people to do more. flint, the railways, the jet engine, the mobile phone, the LLM, the next thing. What will change that is AGI and although that could happen tomorrow I am not sure the underlying technology we have (digital) will get us there.