S2D2

The reply it gave is complete nonsense, it didn't even use the dateutil method it's saying it did.

Nope. Great if it works for you but hopefully this exercise has proven care is needed and dont trust the code "AI" generates.

It added 37 lines to fix the time parsing and did indeed do so. It only needed one of those lines.

Thanks - it disappears on mobile due to the lack of space!

The first failure is incorrect timestamp parsing, i.e. it will throw an error. It's at this point no value is being added imo, far quicker to just fix the timestamp parsing than convince gpt it needs to. Then you can move on to the other bugs.

Personal highlight is how it's started cramming multiple env entries onto one line to get the line count down 😂. It still fails to parse the response in the same way as before.

Nope, yours is far, far worse. 222 lines 😂. It gets the right endpoint at least which mine didn't, then fails completely when it comes to parsing the response.

Dunno, the original prompt is gone. Here's one that spits out non-functioning code: Here's some info on data collection: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance

I disagree, but I didn't use the exact same prompt so it could be pure coincidence. Not that comes to mind and I'm not logged in, so you could see what I do with a logged out session.

Apologies for the formatting and you'll have to sanity check the output for me!

This is no longer possible as it has seen my working code and will now produce a working example - you're welcome ChatGPT! Previously it accessed into JSON data using an invalid key and tried to store cumulative data as hourly totals (PV forecast from forecast.solar). The former is easy to spot, an outright error. The latter will look like it has worked but the values are nonsense, much more dangerous. I appreciate I'm being overly negative, it's a useful tool for getting started or finding relevant information but a lot of care is needed when using the output code.

To check I'm not talking nonsense I asked GPT-4.1 to replace a 30 line Python file I have to access a public API and store the returned data in a database. Took 10 minutes to write and test. Given no context, it produces code that looks fine. It doesn't even vaguely work as it gets the API syntax (which is publicly documented) very wrong. Fixing it requires replacing many lines of code (which are now 56, not 30). Given my existing script, it rehashes it to 76 lines without adding value and introducing a few unwanted side-effects. When told what the problems are, the code works, because the meat of it is the existing code. When asked to improve some timestring parsing that was written in a hurry it happily introduces bugs whilst claiming it has been improved. It will get you 80% of the way there, but so will copy-pasting the documented example from your provider's readme. The rest is from you if you actually want it to work, at which point you might as well just write the code.

Sources like stack overflow, which helped software engineers to share knowledge on a platform immediately available to everyone. AI companies see that data as fair game, scrape it all and provide a more user friendly interface to obtain it. The end result is playing out as we speak, contributing posts to stack overflow are tanking and will dry up. No new data to feed the models, things go south fast as technology moves on.