“AI solutions that are almost right, but not quite” lead to more debugging work.
Read in full here:
1 Like
Might take a couple of years more, but it will get better.
1 Like
‘Survey shows that when Devs start to use the tools, they realize they are unreliable’.
Assuming the same architecture, it will probably get better… but not because LLMs stop hallucinating. It’ll get better because they’ll get larger context windows, and they’ll start putting the docs for the frameworks and languages into that context window before starting. It’ll get better when they can reliably test their own work before submitting. And it’ll get better when they have better processes around workflow incorporated into their programming system prompts.
These are all low hanging fruit, so I expect this to be rapidly improving in the coming months.
One thing that I have learned is that documenting what is not supported is often just as important as documenting what is supported, but it’s a commonly overlooked task in documentation leading to both human and AI hallucinations. For example, if a language structure only supports && conditions an LLM might hallucinate a solution that utilizes || conditions. Improving the documentation for LLMs will also help humans.
1 Like
Probably the best way to use them now is not to trust 100% the code that they generate.
2 Likes
hard agree. the way i ask the agent is to just focus on a very specific lines in a file and my use ccase is letting it solve trivial problems. as long as these specific lines are sufficient in terms of the picturing-out-the-whole-thing problem, the agent will have no problem solving it and tried it few times and it worked well in my case.
1 Like
I sometimes rely on them without truly understanding the code being generated. I don’t really trust, so, I do some tests later on.
1 Like
Yup. I agree.
1 Like
Which one are you using right now?
1 Like
What I realise is that, you can get the draft from the tools, but we should use our expertise to make it work. It’s not 100% ready to use blindly actually.
1 Like
100% agree 
1 Like