Avalon local LLM

[Pocster]

8 hours ago, -rick- said:

@Pocster How desperate are you?

 

 

Well ! Unfortunately as I presume you now realise the ram shortage is a massive problem - yes , even for Apple .

It did look obvious tbh - so now I have to wait for any indicator of ultra appearing and jump on it ! I guess it could be announced at wwdc - maybe have a date for pre orders . Buys Apple time to acquire ram whilst still be bigging up their product . That’s the best I can hope for - a definite date to refresh Apple Store !

[-rick-]

https://arstechnica.com/gadgets/2026/05/apple-may-take-several-months-to-catch-up-to-mac-mini-and-studio-demand/

 

Interestingly it mentions TSMC being a bottleneck. TSMC doesn't make RAM.

Edited Friday at 14:19 by -rick-

[Pocster]

2 hours ago, -rick- said:

https://arstechnica.com/gadgets/2026/05/apple-may-take-several-months-to-catch-up-to-mac-mini-and-studio-demand/

 

Interestingly it mentions TSMC being a bottleneck. TSMC doesn't make RAM.

TSMC doesn’t make RAM, no one said they do. TSMC is the SoC bottleneck.

 

The RAM shortage is a separate bottleneck, and Apple clearly isn’t immune. Other manufacturers have openly said memory shortages are affecting them, and Cook has said Mac mini / Mac Studio supply-demand balance will take several months.

 

High-memory Mac mini / Studio configs are also exactly where the worst delays and unavailable options have shown up.

 

There are also multiple reports suggesting the memory shortage could last well into 2027 and possibly beyond, especially as AI demand keeps soaking up supply.

 

So pointing out that TSMC doesn’t make RAM doesn’t disprove the RAM issue; it just identifies a different bottleneck.

[-rick-]

9 minutes ago, Pocster said:

So pointing out that TSMC doesn’t make RAM doesn’t disprove the RAM issue; it just identifies a different bottleneck.

 

Sure, I just thought it was interesting to highlight.

[Pocster]

Even if Mac unified memory and iPhone RAM aren’t identical final packages, they come from the same constrained LPDDR supply chain. Apple then has to allocate that supply across products, and iPhone will obviously take priority over niche high-memory Mac configs.

[MikeSharp01]

3 hours ago, Pocster said:

Even if Mac unified memory and iPhone RAM aren’t identical final packages, they come from the same constrained LPDDR supply chain. Apple then has to allocate that supply across products, and iPhone will obviously take priority over niche high-memory Mac configs.

I suppose that consumer high end local LLM schemes need large memory but you can do the same thing with a cloud based one while you await delivery of enough RAM, or whatever other semiconductor you are waiting on. Trouble is that the end of the proverbial rainbow is moving away from you faster than you can stuff memory into machines anyway, so, when your RAM arrives there will be plenty of models it won't run because they have grown, bloated, faster than you can add RAM to your machine. 

[-rick-]

12 minutes ago, MikeSharp01 said:

I suppose that consumer high end local LLM schemes need large memory but you can do the same thing with a cloud based one while you await delivery of enough RAM, or whatever other semiconductor you are waiting on. Trouble is that the end of the proverbial rainbow is moving away from you faster than you can stuff memory into machines anyway, so, when your RAM arrives there will be plenty of models it won't run because they have grown, bloated, faster than you can add RAM to your machine. 

 

Does look that way. New Deepseek one for example, and it's been touted has having huge optimizations for reducing memory usage. It might be much better at storing context but if the model is so huge in the first place it's not easy to run at home.

 

Google seems to have done well with Gemma, but of course that is a lot cut down from the leading models and they aren't releasing those. Doubt any western firm will for 'safety' reasons. So far the Chinese govt doesn't seem to care about that so long as their models spout Chinese propoganda at the relevant moments.

[Pocster]

8 hours ago, MikeSharp01 said:

I suppose that consumer high end local LLM schemes need large memory but you can do the same thing with a cloud based one while you await delivery of enough RAM, or whatever other semiconductor you are waiting on. Trouble is that the end of the proverbial rainbow is moving away from you faster than you can stuff memory into machines anyway, so, when your RAM arrives there will be plenty of models it won't run because they have grown, bloated, faster than you can add RAM to your machine. 

I’ll use chat as my reasoner but I need local to  do my local actioned stuffed until I can source enough ram . Bigger model did mean better but now smaller better optimised models can out perform a larger one . Obviously there’s a point when you can’t go smaller without too much compromise . If you ask ChatGPT itself how long before a model matches its capabilities of today in a ‘ reasonable ‘ size it reckons within 2 years a 256gb model will be comparable to ChatGPT 5.4. That’s insane ! To have that capability sat on your desk in a relatively low ram config.

Moe and potentially’ bolt on ‘ specialists seems to be the next level . A medical specialist , maths specialist etc plugged in to the reasoner.

[Pocster]

After lots of reading ( for probably months ) , YouTube videos etc etc . Enough to confuse anyone I decided to start Avalon .

 

chatgpt and I are architects 

 

we setup llama etc , vs studio , ssh into m3 etc etc , even parsec screen share 

 

Deepseek is our reasoner . Qwen coder next our sweat shop “ just do it ! “ coder !

 

chat managed to create me some scripts so we create a brief for today’s task . Get Deepseek to take that and produce a detailed instruction list for qwen .

 

So process is

 

Create task 

Give to Deepseek 

Approve Deepseek output 

Tell qwen to do it ( for reasons I’m unsure of we don’t use git diffs but json - I didn’t argue as it seems to work )

Then some kind of test . Chat gets excited about this ; me less so when terminal window says “ it worked ! “ 😊

But we are building framework and structure first .

 

I’m blown away by this all locally !

llm , creating document for llm to produce detail for another llm 

Crazy !!

90% of programmers obsolete ! 
Just todays tasks would have been weeks of learning and work .

 

[Pocster]

Notice Mac minis basically out of stock and even a m3 ultra 96gb ( base model ) no where to be seen . I got mine from curry’s of all places ! , check it every few days to judge demand - now they have none !

I think when I sell the m3 I’ll get a pretty good price for it even with m5 released because it’s all going to sell out fast !

[JohnMo]

6 hours ago, Pocster said:

Notice Mac minis basically out of stock and even a m3 ultra 96gb ( base model ) no where to be seen

Bought mine the other week from Amazon, just looked and still available.

Edited 20 hours ago by Nickfromwales

[Pocster]

5 hours ago, JohnMo said:

Bought mine the other week from Amazon, just looked and still available. Look harder

Yeah it said “ out of stock “ so that’s a clue .

Screen grab is from now !

Edited 20 hours ago by Nickfromwales

[Pocster]

4 hours ago, Pocster said:

Yeah it said “ out of stock “ so that’s a clue .

Try (expletive deleted)ing reading . Screen grab is from now !

what is it with pricks on this forum !

I’ll wait for apology 

For completeness yes it’s on Amazon apparently at an inflated price . 

Edited 20 hours ago by Nickfromwales

[MikeSharp01]

Why not get an RTX 5090 card for you desktop pc. True they don't have the shared ram but they should do all you need. I have a 3090 with just 24Gb of ram but it does all I need. 

[JohnMo]

Mac mini, M4 is what I quoted and yes it's for sale, delivery on Wednesday.

 

Edited 20 hours ago by Nickfromwales

[Pocster]

38 minutes ago, MikeSharp01 said:

Why not get an RTX 5090 card for you desktop pc. True they don't have the shared ram but they should do all you need. I have a 3090 with just 24Gb of ram but it does all I need. 

A sane person . Problem is unified ram . Best on Nvidia is 96gb on rtx6000 that’s a 10k gpu card !!!!

This is why Apple have ( unintentionally) jumped to number 1 for local llm .

Edited 23 hours ago by Pocster

[SteamyTea]

I am waiting for the ultimate answer.

[Pocster]

8 minutes ago, SteamyTea said:

Don't lock it down, I am waiting for the ultimate answer.

I’ll give you a 69 

[Nickfromwales]

Thread tidied. As you were people.

 

The management.

[MikeSharp01]

3 hours ago, Pocster said:

Don't lock it down, I am waiting for the ultimate answer.

Like we already said the answer is moving all the time so 42 won't cut it in today's race. 

 

FWIW I think this thread is interesting because this technology has so many implications for us all and although we are just experimenting the opportunity to do amazing things is tantalising and, frankly, we / they have not found the killer application yet and that may come almost as probably from amateurs playing as professionals trying to crack it. The local platform arms race to top or bottom is really an aside. As you say you can already spend 10K on a card always assuming your rings (mains) can supply the power. Pretty soon fan coils and UFH won't be the only things your average ASHP will be cooling!

[Pocster]

13 hours ago, JohnMo said:

Mac mini, M4 is what I quoted and yes it's for sale, delivery on Wednesday.

 

Right , now, let’s do this slowly 

 

I didn’t check every website on Earth .

You need enough ram to be useful . 24gb once booted won’t leave a lot for a useful llm .

local llm can even be installed on a pi for example .

The keyword is useful - in essence you’d want 64gb min on a Mac as it uses unified ram . People run these things on much less .

These units and above have all but disappeared. Ultra m3 ( go check ) on eBay ( assuming they have real stock ) are on for 5k base model prices . So that’s 1k more than Apple retail them for . Openclaw etc has made a lot of people punt on a relatively cheap Mac mini to experiment with .

[Pocster]

9 hours ago, MikeSharp01 said:

Like we already said the answer is moving all the time so 42 won't cut it in today's race. 

 

FWIW I think this thread is interesting because this technology has so many implications for us all and although we are just experimenting the opportunity to do amazing things is tantalising and, frankly, we / they have not found the killer application yet and that may come almost as probably from amateurs playing as professionals trying to crack it. The local platform arms race to top or bottom is really an aside. As you say you can already spend 10K on a card always assuming your rings (mains) can supply the power. Pretty soon fan coils and UFH won't be the only things your average ASHP will be cooling!

For a programmer / SE certainly junior roles have been reduced significantly. Why employ someone when a llm can do it ? . There’s virtually no need to write code anymore 🤯.

No need to learn all the libraries .

Now it is orchestration for the human . I discussed this with ChatGPT . It worded it as coding is now cheap and virtually no labour required ! 
I’m in a fortunate position as I can experiment, spend money , no boss , endless end goal , no risk .  Chat agrees that my full project would require pre llm a small team of programmers and would take years to develop . Now there’s one , me , zero code to do . 🤯🤯🤯.

For reference as I have only 96gb I have the smallest possible context windows. This apart from reducing ram requirements also avoid hallucinations. Deepseek and qwen are fed fresh prompt each time not historical reference required . ChatGPT of course needs the history / context - but produces no code . I’ll say it again , it’s insane ! 

[JohnMo]

40 minutes ago, Pocster said:

Right , now, let’s do this slowly 

That's a bit more specific. You simply said Mac mini. But like asking for a BMW when you are only looking for an M5 BMW. But hay ho. Was only trying to be helpful, but get a world of scorn for my efforts... Won't bother in future. Don't bother replying because I won't see it

[Pocster]

31 minutes ago, JohnMo said:

That's a bit more specific. You simply said Mac mini. But like asking for a BMW when you are only looking for an M5 BMW. But hay ho. Was only trying to be helpful, but get a world of scorn for my efforts... Won't bother in future. Don't bother replying because I won't see it

Good ! This thread is about llm’s - they require Ram thanks though @JohnMo fo 

[SteamyTea]

8 hours ago, Pocster said:

You need enough ram to be useful . 24gb once booted won’t leave a lot for a useful llm .

I am quite interested in this.

I can understand needed a fair bit of RAM to run the LLM.

But is there something special about the OS that it needs a massive amount.

I know OSs are RAM hungry these days, unlike 35 years ago when an image file was larger in RAM than the OS took up.

Can the OS be stripped back to the very basics of what it was originally designed to do?