Avalon local LLM

[SimonD]

2 hours ago, Pocster said:

After paying 90 quid even 5.5 thinking seems considerably better - funny that.....

Wouldn't put it past them.

[Pocster]

6 minutes ago, SimonD said:

Wouldn't put it past them.

Dont trust any of the AI firms.... Apparently GLM5.2 local is really good - of course hardly anyone can run it ....

[Pocster]

Really awful bug. Chat 5.5 thinking kept patching and we kept rolling back. I kept trying to think of other ways to deal with it so we can try different approaches. Been at it for 45 minutes. Rolled it back. clicked "pro" gave it all the info I could. Pro then thinking for 14 minutes!. Found a really obscure issue - MAGIC! FIXED!

Edited yesterday at 16:00 by Pocster

[Pocster]

WOW oh wow!

Never really looked into how a LLM generates its output i.e. the cost. Assumed its just generated at the end but it isn't. It's generated as it goes ! So each token passes through the model. Never thought of that!

SO! 5 seconds with a moderately complex phrase after json compaction is now 1.3 seconds! BOOM! WHO"S THE MOFO!

[Pocster]

2 hours ago, Pocster said:

WOW oh wow!

Never really looked into how a LLM generates its output i.e. the cost. Assumed its just generated at the end but it isn't. It's generated as it goes ! So each token passes through the model. Never thought of that!

SO! 5 seconds with a moderately complex phrase after json compaction is now 1.3 seconds! BOOM! WHO"S THE MOFO!

Saved another 250ms ... yeah I know. I'll stop now! sad.

[Pocster]

Chat has been SO good today I might give it a promotion - nothing to do with me spending 90 quid...... 

[SimonD]

1 minute ago, Pocster said:

Chat has been SO good today I might give it a promotion - nothing to do with me spending 90 quid...... 

Bastard! 😉 I'm having a shit day today. Realised it had left a glaring security hole in the Auth model. Decided to fix it, but instead has broken the (expletive deleted)ing app. It fixed part of the problem and then tried to tell me the rest wasn't important until I told it that I could grab an id and post it into the browser in a string and it would expose the entire records for a user, even when not logged it! What was supposed to be a couple of hours at most has ended up taking all bloody day and I'm still trying to explain to it what's going wrong and it still misunderstands me! Thank (expletive deleted) I'm on a dev server. 

[Pocster]

13 minutes ago, SimonD said:

Bastard! 😉 I'm having a shit day today. Realised it had left a glaring security hole in the Auth model. Decided to fix it, but instead has broken the (expletive deleted)ing app. It fixed part of the problem and then tried to tell me the rest wasn't important until I told it that I could grab an id and post it into the browser in a string and it would expose the entire records for a user, even when not logged it! What was supposed to be a couple of hours at most has ended up taking all bloody day and I'm still trying to explain to it what's going wrong and it still misunderstands me! Thank (expletive deleted) I'm on a dev server. 

Oh yes!. Well as you know you get good days and bad days! Mines been epic. Massive speed increases. Local llm "whats the capital of france?" working. Current affairs " whats the news?" gives headlines and options verbally if you want more detail. Will add history so you can have a conversation. Gained a further 82ms saving on STT (I know, I know !). Honestly now its so fast to respond to even complex stuff I'm well impressed. Started on timers like Alexa (a SWMBO requirement!). TBH if I coded this by hand that's weeks of work for sure.  But of course I never look at the code! G n T time now!

Edited 2 hours ago by Pocster