Be aware, this will kill us! 😂

General Developer Forum General Dev Chat
3

Here we go again … GitHub made Copilot free (with limitations). I have tried it and I was a bit surprised - only a bit … Please keep in mind that Copilot does not officially support Elixir. :exclamation:

GPT 4o

obraz
obraz1229×696 38.5 KB

Let’s make it quick to not repeat myself too much once again …

  1. Why do chatbots always try to convince me that I want something else that I want? I don’t want a String! :man_facepalming:

  2. The chatbot have changed an option key (correctly) and the value (valid, but wrong for this case) and didn’t tell about that. :icon_confused:

  3. Once again ChatGPT tries to convince me that I want to print data as list … especially if it changed a chalist to a String! :joy:

  4. There is no false in last paragraph, but it’s an answer for a different question. I didn’t asked to change type, but to fix code. Is GPT based on commits with only fix as a message? :bulb:

Claude 3.5 Sonnet

obraz
obraz1200×915 62.5 KB

First of all I have respect for chatbot have written in first paragraph. To perfection there should be also note like:

Please pay attention that Elixir is not officially supported language by Copilot, see below link for more information:
GitHub language support - GitHub Docs

Second paragraph is amazing! That’s a first chatbot mentioning deprecation and it even told me which Elixir version deprecated old naming. :+1:

It’s still far away from perfection, but it’s already on good road. Better answer should include:

  1. Mention that a change in naming is recommended since Elixir version 1.3.0 as since this version the documentation at hexdocs was has been changed with a new naming. :spiral_notepad:

  2. There should be a links to: changelog, discussion, issue, PR and maybe even a direct link to commit (page with commit also includes some helpful information, so for some cases it’s also worth to mention it). :link:

The changed code however decreases the quality.

  1. It does not suggest a naming change 'char_lists' → 'charlists' :see_no_evil:
  2. It does not suggest a c sigil 'char_lists' → ~c"char_lists" :slightly_frowning_face:
  3. The option key is changed to proper one as described. However once again the value is valid, but not in context of the value in question. It already found a deprecation, so it could just follow that lead! :-1:

Similarly to GPT 4o the next paragraph is almost correct, but answers wrong question. We still can notice that chatbot have a problem with figuring out a difference between a charlist and string. :exploding_head:

I would not said any word about last paragraph if the chatbot would not stop half a way. Many additions mentioned before are “nice to have” and they are not as much required as the correct answer. Since it already realised the deprecation it should give a proper answer without any more context. :speak_no_evil:

Summary

Claude 3.5 Sonnet in GitHub Copilot is first chatbot that I can recommend to Elixir developers, but only for seniors who understand the problems with so-called "AI"s. I hope that in next version developers would be able to make chatbot keep one idea without telling developers that they want something else. Since it even found a version I also hope it would be much more descriptive and provide a links for learning purposes. :heart_eyes:

I guess that until Elixir would not be officially supported we would still need to be extremely careful about chatbot responses. However it’s not really as bad a it looks. It shows that’s not an AI which suddenly “wake up”, but just another algorithm. I can’t blame any developer working on any chatbot for such mistakes as simply LLMs alone were never intended for that purposes and if they would not change narration then I would not see any problem with LLM part. :see_no_evil:

They already are doing well. We should also keep in mind that the more lines of code the more possible bugs would be introduced. I would say that changing a direction of response (from deprecation to changing code logic) may be considered as a bug, but I also understand that LLMs have problems with figuring out the context. :thinking:

1 Like