Avalon local LLM

[SimonD]

2 hours ago, Pocster said:

After paying 90 quid even 5.5 thinking seems considerably better - funny that.....

Wouldn't put it past them.

[Pocster]

6 minutes ago, SimonD said:

Wouldn't put it past them.

Dont trust any of the AI firms.... Apparently GLM5.2 local is really good - of course hardly anyone can run it ....

[Pocster]

Really awful bug. Chat 5.5 thinking kept patching and we kept rolling back. I kept trying to think of other ways to deal with it so we can try different approaches. Been at it for 45 minutes. Rolled it back. clicked "pro" gave it all the info I could. Pro then thinking for 14 minutes!. Found a really obscure issue - MAGIC! FIXED!

Edited Thursday at 16:00 by Pocster

[Pocster]

WOW oh wow!

Never really looked into how a LLM generates its output i.e. the cost. Assumed its just generated at the end but it isn't. It's generated as it goes ! So each token passes through the model. Never thought of that!

SO! 5 seconds with a moderately complex phrase after json compaction is now 1.3 seconds! BOOM! WHO"S THE MOFO!

[Pocster]

2 hours ago, Pocster said:

WOW oh wow!

Never really looked into how a LLM generates its output i.e. the cost. Assumed its just generated at the end but it isn't. It's generated as it goes ! So each token passes through the model. Never thought of that!

SO! 5 seconds with a moderately complex phrase after json compaction is now 1.3 seconds! BOOM! WHO"S THE MOFO!

Saved another 250ms ... yeah I know. I'll stop now! sad.

[Pocster]

Chat has been SO good today I might give it a promotion - nothing to do with me spending 90 quid...... 

[SimonD]

1 minute ago, Pocster said:

Chat has been SO good today I might give it a promotion - nothing to do with me spending 90 quid...... 

Bastard! 😉 I'm having a shit day today. Realised it had left a glaring security hole in the Auth model. Decided to fix it, but instead has broken the (expletive deleted)ing app. It fixed part of the problem and then tried to tell me the rest wasn't important until I told it that I could grab an id and post it into the browser in a string and it would expose the entire records for a user, even when not logged it! What was supposed to be a couple of hours at most has ended up taking all bloody day and I'm still trying to explain to it what's going wrong and it still misunderstands me! Thank (expletive deleted) I'm on a dev server. 

[Pocster]

13 minutes ago, SimonD said:

Bastard! 😉 I'm having a shit day today. Realised it had left a glaring security hole in the Auth model. Decided to fix it, but instead has broken the (expletive deleted)ing app. It fixed part of the problem and then tried to tell me the rest wasn't important until I told it that I could grab an id and post it into the browser in a string and it would expose the entire records for a user, even when not logged it! What was supposed to be a couple of hours at most has ended up taking all bloody day and I'm still trying to explain to it what's going wrong and it still misunderstands me! Thank (expletive deleted) I'm on a dev server. 

Oh yes!. Well as you know you get good days and bad days! Mines been epic. Massive speed increases. Local llm "whats the capital of france?" working. Current affairs " whats the news?" gives headlines and options verbally if you want more detail. Will add history so you can have a conversation. Gained a further 82ms saving on STT (I know, I know !). Honestly now its so fast to respond to even complex stuff I'm well impressed. Started on timers like Alexa (a SWMBO requirement!). TBH if I coded this by hand that's weeks of work for sure.  But of course I never look at the code! G n T time now!

Edited yesterday at 16:36 by Pocster

[SimonD]

17 hours ago, Pocster said:

Oh yes!. Well as you know you get good days and bad days!

Oh yes. Finally got it sorted by about 9pm!!!!!

 

 

17 hours ago, Pocster said:

But of course I never look at the code! G n T time now!

It's interesting this and token burn rate. This morning I needed to do some financial modelling and analysis so selected Opus 4.8 for the conversation. 30 min later I looked at my session stats and I'd gone through nearly 80% of my session allocation. Then I select Sonnet 4.6 for another conversation - involves maths but nowhere near as complex and the conversation only burns 3% of session allocation but provides the answers I need. It certainly pays to switch models for different needs.
 

[Pocster]

4 minutes ago, SimonD said:

It certainly pays to switch models for different needs.

Though cloud tokens arent an issue for me now I use ministral mlx for fast language parse. If its a knowledge question it passes it to qwen. I suspect I'll be passing other stuff around to various models depending on the requirement. I'm having a break from that part and looking at realtime 3d scenery for the backdrop - I like my eye candy!

[Pocster]

 

v1

[SimonD]

4 minutes ago, Pocster said:

 

v1

That's awesome!

[Pocster]

2 minutes ago, SimonD said:

That's awesome!

All of 2 minutes work! 😀

[Pocster]

But I expect I'll go Minecraft style!

[Pocster]

Welcome to my BS YouTube channel. Today we are going to create a minecraft inspired game in 10 seconds!. Hit subscribe!, it's you viewers I need! - thanks to our sponsor buildhub.org.uk.

[Pocster]

 

SWMBO made me do some outdoor work! . Anyway welcome back to my youtube channel. Chat not very good at optimising drawcalls. We started with 50k!!!!!. Now down to 1. It needed a word. Struggling with the river at the moment. Hit that subscribe button NOW!

Edited 2 hours ago by Pocster