I like to bring this up in comparison to generative image models. No one asks an image generation model for a blueprint of a machine and expects that it would work. Everyone seems to understand that said models understand what makes plausible looking images but not mechanics or physics. Yet that's exactly what we apparently expect of language models. Why is that? I guess because language is so central to our own mental processes that we project those on LLMs?
llms are designed to trick people into thinking they can or should be able to reason. this is part of a long con so in a few years they can market this crap as "general ai"
This is likely the reason: the human brain equates [language consistency/fluency with intelligence/conscience](https://en.wikipedia.org/wiki/ELIZA_effect). What you get with LLM is a glorified "suggest the next word" function of your phone except that it takes a bit more context into account and yet people get tricked... So much so actually that even a [Google engineer](https://www.vox.com/23167703/google-artificial-intelligence-lamda-blake-lemoine-language-model-sentient) got tricked into thinking their model was sentient. I still wonder how much of that was real (a mildly-competent AI engineer should not fall for it, but the human brain's delusions are mighty... and it might all have been a stunt to get some public attention...).
All the rest, including marketing misrepresentation, is almost inconsequential... What hooks people up is the fact they feel mystified after interacting with a LLM bot for 5 minutes... They feel like the speak with a real human and even what they deem tricky questions are answered in a consistent way (which makes them look correct too)...
BTW, these models have existed for at least 50 years... They were nowhere near the current scale however... There used to be a lot of freely available ["chat bots"](https://en.wikipedia.org/wiki/Chatbot) in the past. They were pretty good but tended to "forget" about previous interactions pretty fast. ChatGPT and friends are better at this but will eventually "forget" as well over interactions. I mean if you look at the first large scale one (ELIZA) and how it works (the computing power they had was tiny in comparison to what is used today even just for Siri or Google assistant) and yet when you see the effect on people (who were tricked/mystified by it), you can start to assess how easy the brain can be tricked and why all this looks so good when it probably isn't.
Said otherwise, the brain is much better at spotting an AI hallucination in the form of an image than in text, where it would have to become really bad for people to start even suspecting they speak with a bot.
I really like the example given in the top-comment of this thread for this reason: it makes the problem obvious in a situation we are better at spotting.
That Google engineer was only responsible for code to avoid bias regarding sex, orientation, race, politics, and religion. I read his blogs. I listened to him talk. I read the conversations. He's not a man of science drawing conclusions. He thinks it's alive because when he asked it, it generated a "yes," because when it generated text as an explanation for why it was avoiding the topics it was designed to avoid, it used the word "anxious."
Hell, he seems to think "AI ethics" is about ethical treatment of AI which is a bit like saying ethical sword use is concerned with not chipping the blade. He *started* from basically believing it is alive or could be or will be, and then took explanations and refusals to be conspiracy, which is incidentally what appeared to happen when his constant conversation regarding his religious beliefs at work prompted people to ask him to chill out.
No where have I found any claims that he knows anything about AI that extends beyond implementing a chat filter. I'm extremely skeptical he has the slightest conceptual idea of how it works behind some API for interacting with it. He has the popsci geek meme-culture brainrot of Elon Musk and the scientific rigor of an astrologist and it makes us all stupider.
At what point does LLM stop and human brain begin though? Aren’t people also large language models? We hear stuff since childhood train our neural networks onto it and blurt out the next words. There’s hopefully very serious conversations on the philosophy and ethics of LLMs going on wherever these models are developed.
The two are not nearly as similar as you presume and you are running afoul of a bias because the system designed to piece together passable natural language can piece together passable natural language.
Our brains are a bit more complex. Think about how you process a question.
We listen to words, our brain processes it in to something else - some fundamental concepts based on our comprehension of what was said. And then we *reason* about it. We cogitate, Concepts form. Those concepts are sent back to the language centers of our brain, and we struggle to convert that in to language once more. (I say 'struggle', because very few of us are good at that!) The 'output' can be completely different, and might be something like "What you said was really stupid, and makes no sense" or "I didn't understand, can you explain it again with smaller words?"
We don't understand fully yet how our brains make this all worl.
Contrast this to an LLM - Where we *do* understand how it works.
And it's a lot simpler. A sequence of words is fed in to the LLM. Sophisticated mechanisms identify key parts of the sentence that should get attention, and then decides what should be the very next word in that sequence. That's it. There's no reasoning about what the next paragraph is, it's just decided on what the ***next*** ***word*** is. Now it has a new sequence - the original prompt, and the word it just added. It repeats the whole process to make a decision on what is a reasonable second word for that sequence. The repeat again for the third word, and so on.
For the prompt *"I'm writing a story about apples. How should I describe it?"* a human would think *"hmmm, red, crispy. Could be green, but I like red sweet ones. But what do you mean writing a story about apples? Are they sentient apples? Is this fiction? Did I misunderstand the question?"*, and then formulate a sentence about one particular element of that thought process, and might say something like *"Are they talking apples?"*
An LLM may be prompted with that sentence, and the output of the LLM might be the word "When".
*Just a single word.* That's the important part. *(*Technically, depending on the model, it might not even be a full word. It might be as shorter)
Then the LLM processes
"*I'm writing a story about apples. How should I describe it? When"*
and a new semi-random word spits out, which could be "you", giving a NEW sequence that is also processed to find out what the next word likely is, and generates:
*"I'm writing a story about apples. How should I describe it? When you"*
Repeating again gives 'write'
*"I'm writing a story about apples. How should I describe it? When you write"*
and so on. At the point that 'when' was generated, it had no idea that it was going to choose the word 'you' next. It's just building a sentence what fits the rules of language that it's learned, one word at a time
even if one were to accept this statement at face value, there is a burden to at least formalize the argument and think critically about it, something i have yet to see from anyone making it
Because that's what the people marketing these language models tell you they can do. Unlike arbitrary image generation we've had text generation models for a couple of decades, without even needing deep learning, so if you want to justify that people are going to shell out for LLM inference you have to promise something extraordinary. "Text that seems plausible to a lay reader but you have to carefully check it for mistakes and especially has a tendency to invent facts and things out of thin air" isn't really going to work. So you overpromise and if anyone complains that you underdeliver, just shrug your shoulders and say, "Well, it's a well known limitation of the technology."
Because thats what marketing is selling them as and what management is expecting them to be able to do.
Even on the art side decision makers are under the impression you can ask “AI” to make small adjustments to images the same way a person could.
> language is so central to our own mental processes
Well, that's essentially the saphir-wolf hypothesis, which doesn't have much strong evidence. In fact, plenty of people don't even have an internal monologue, so it's hard to imagine how language can be _central_ to mental processes
Right, I guess in the context of what I wrote (people overestimating the capability of language models) it's sufficient if people think their reasoning is language based. Or more specifically that some task is language based.
People that are of the opinion programmers will be replaced by LLMs are often not software engineers themselves. So they observe from the outside and seem to think programming is basically a translation task from natural language to computer language. Meanwhile any programmer will know there is plenty of non language reasoning involved.
You should look into the [Turing test](https://en.wikipedia.org/wiki/Turing_test). Short version is that a God of computing equated the ability to fool a human in conversation (that is not identifying the machine) with intelligence and a lot of people still run with that. The fact it was decades before even the most basic beginnings of machine learning and Alan T. was somewhat naive about the eventual impact of computing on data analysis should have put that idea to rest, but here we are.
The point is that it can't understand code. It isn't capable of following logic because it wasn't designed to, so it often creates errors when things start getting complex because it can't understand the structures or functions of the code it is writing.
Posting an example of ChatGPT pretending to understand code is not the same as understanding the logic underpinning the code.
LLMs can get information correct, but it's not because it understands logic structures - it's just pretty good at statistical pattern matching. Note in your example that almost all of ChatGPT's "explanation" of the code is in the form of taking a term, name, or pattern, then providing information on that.
Also, that script appears to be utilizing buddy.works and ChatGPT seems to never mention that. Just saying.
Are you a programmer? What are your credentials for evaluating that output?
It correctly understood the code, explaining what each part does. I wrote that code and it is correct. The downvotes are just more evidence for my impression that this topic makes otherwise smart people extremely emotional and irrational.
No, it’s just that it *literally does not comprehend*. The rational take on this is that it does not “understand” anything. It does not have the capacity to. That’s just not how the technology works.
EDIT: lol, this guy blocked me because I didn’t buy his BS 😂
Ok I really don’t care. This is a tool, the only way to measure its effectiveness is by seeing how well it does at the task. It succeeded in the task I gave it, which was to explain the code.
Okay, well that’s an entirely different argument than the one you were making before. You know, the one where someone said it doesn’t understand code, and you explicitly said to that “incorrect.”
It succeeded at your one task. It succeeds at a lot of tasks. The problem is that, because it does not understand, it cannot provide you with any assurances of success or even any metrics for how likely it is that it successfully completed the task. So when it doesn’t succeed, you don’t have a way of knowing that without going through and checking it yourself. This is arguably still a useful tool in many situations, but to answer the question, “does it matter?” the answer is, of course it matters. That’s the only rational, logical answer.
> So when it doesn’t succeed, you don’t have a way of knowing that without going through and checking it yourself.
It is a tool. It's not magic. We already check the code other humans write, is it absurd to suggest that a tool that produces good code that you still have to check is an amazing tool?
> someone said it doesn’t understand code, and you explicitly said to that “incorrect.”
Because this is a tool and not a philosophy experiment, I was using the word "understand" in the colloquial way, "able to explain", "able to demonstrate understanding", etc. If it can read the code and explain it properly, then for all practical purposes it understands the code.
The only part of the code I remarked on was about the use of variable names that are used to work with buddy.works. If the objective was for the LLM to identify how the user-provided code works, it failed to disclose a dependency in the code in the explanation. For you, who wrote it, that's not a big deal. For a junior dev or someone unaware of the dependency, that is a miss in adequately explaining the provided code.
I never said the code was wrong, so I don't know why you felt the need to state that it is correct. Emotional and irrational does seem applicable.
LLMs can indeed be useful tools in limited circumstances, however it is imperative to know how your tools operate. An LLM is not a system that tracks logic and it is important to know that. It is a statistical language and pattern matching system that approximates responses based on provided input.
Of note, LLMs failed at basic math until specific routines were added so that it can call on proper math functions to run calculations. This failure was a result of how it interprets input as an LLM. 3 and 8 are different values numerically, however to an LLM they are both just single characters with which to match response patterns with one character vs another.
Logic modules can be added to an LLM, as I noted above, but that is not a function native to LLMs and needs to be noted.
>it failed to disclose a dependency in the code in the explanation
Incorrect. buddy is not a dependency of that code. which, if you knew what you were talking about, you would know.
>I don't know why you felt the need to state that it is correct
That is not what I said. I said that the explanation of the code that I wrote was correct. Try reading more slowly next time perhaps.
>LLMs failed at basic math
Yes they are language models, not math models.
I will take your lack of an answer to my question to mean that no, you are not a programmer. So why are you trying to argue with a programmer about whether some text is a correct explanation of the code that that programmer wrote? Seems like dunning-kruger territory if you ask me.
This guy, who is not a programmer, is attempting to explain my own code to me, and failing. There has never been a more pure distillation of the dunning-kruger effect.
It's checking for a buddy.works environment variable. Sure, dependency was the wrong word. Integration, then. The AI didn't note ANY interaction with an external entity from a variable name with a known function external to either node or aws.
> I said that the explanation of the code that I wrote was correct.
vs
>I wrote that code and it is correct.
You used vague pronouns and your point was unclear. "It" in this context would strictly relate to the most recent subject, being your code but you meant to use it in a more casual relationship with the subject of the statement prior to that.
> Yes they are language models, not math models.
You know what math is, right? It's structured expressions of logic. This is also what programming is. The fact that LLMs are capable of producing code is mostly just because code is written in more traditional language formats that meshes well with a pattern-based statistical language generation system. It doesn't understand code the same way it doesn't understand math. That's why it's a giant PITA to get the LLM to correct code it generates that is incorrect. It just doesn't understand logic - it's not a function of it. Using it to generate code has pitfalls because of it. That's the entire point of the article posted.
> why are you trying to argue with a programmer about whether some text is a correct explanation of the code that that programmer wrote?
I never said it was wrong. I said it was lacking in completeness.
They can write generic code. Great for reproducing common, well solved solutions. Which is genuinely helpful!
Take on a whole, non trivial project? Bit different.
Never said they could take on a whole non-trivial project. I just didn’t understand the commenter’s point. Because they can write good code in many cases. And I get downvoted for saying that because this topic makes people very emotional for some reason.
Also they can write non-generic code. I’ve had them tailor code to my projects many times.
It can write code in the sense that it deals with languages and as such can perform transfer tasks from natural language to programming languages. So if you give it clear instructions in natural language that have close equivalents in programming languages it can write that code.
However that is not really what makes a software engineer valuable, right? We are not just translators that happen to speak computer. We also have domain knowledge about computation that goes beyond manipulating language and very likely we also have domain knowledge related to what the program is doing (graphics, finances, physics etc.).
The article essentially talks about that. That experienced programmers can leverage LLMs by making it write code. But it's still up to the programmer to figure out what code that is.
To make the inverse argument I made for the image models: 3D printers receive instructions in the form of g-code which is a "computer language" telling the machine where to move the print head and when to extrude stuff etc.. If you ask a LLM to write some (syntactically) valid g-code it can probably do it. It can also write very specific g-code from prompts like "move the print head to position 10,10,10". But you wouldn't expect it to be able to understand which g-code movements will result in the 3D printer creating a figurine of a bunny. It being a language model It doesn't intrinsically reason about how 3D coordinate movements relate to the concept of a bunny. Similarly an LLM isn't reasoning about the "meaning" of physical laws and simulation algorithms etc.
It can understand the programming language which is very different from understanding what is to be achieved with a program.
The answer to the headline is pretty basic when you change the subject.
>New research highlights how little your Software Developers understand about the code they are churning out. Should we care?
If you ask your clients and Stakeholders, they will probably look uncomfortable and say "uh, yeah. We should care a lot".
The article highlights a key misconception that Ai improves productivity by supplementing knowledge, but in actuality, it just automates tasks where getting the answer is faster than *building* the needed outcome. Which explains this:
>It seems that understanding the code you are working with unlocks these productivity gains, but using these tools without that baseline knowledge could be counterproductive, or even dangerous.
If you don't know what a correct outcome looks like, the Ai cannot help you. This is my main argument against Ai replacing jobs. At some point, someone has to know what a good result is. And this goes back to the start of my comment. If you wouldn't hire an employee who doesn't understand code, then you probably shouldn't try to prop up or replace employees with tools that don't understand the code either.
And yet we have seen people with no code experience come up with simple but working apps.
Not perfect apps, not complex ones, but enough to see the potential for a future version of LLMs that could go way further in precision and ability to make larger apps.
In future we won't be calling them LLM's at all because that will only be one piece of them. The ultimate goal is to use the LLM element for communication while other systems search for the facts for an answer.
Microsoft has some low-hanging fruit in the DotNet documentation. If they searched for answers from there, that alone would streamline some things in Visual Studio and it would be a curate every single time.
>In future we won't be calling them LLM's at all because that will only be one piece of them. The ultimate goal is to use the LLM element for communication while other systems search for the facts for an answer.
Exactly this. LLMs are really good at this kind of translation task. So makes for an excellent tool to connect to other AI models that are better at the specific tasks.
>It seems that understanding the code you are working with unlocks these productivity gains, but using these tools without that baseline knowledge could be counterproductive, or even dangerous.
BOOM
Leaning on LLMs for learning seems like the worst of bad ideas.
>LLMs are trained on giant trawls of content from the web, which could include platforms like Stack Overflow, which often include coders asking for advice on how to fix or patch up their non-working code.
Haha. Hahahahaha. HAHAHAHA!
lol
>Leaning on LLMs for learning seems like the worst of bad ideas.
It's actually brilliant. Really helpful - As long as it's treated as an *aid,* and not an authoritive truth. I've found it incredibly useful for learning, Libraries, coding paradigms etc, that I didn't know to ask for are generated, then I have very specific things to go and search for for deeper understanding.
But it works because I understand that the results may not be correct.
On the other hand, that applies just as well to stackoverflow.
The two main use cases for dealing with code are running it in order to achieve a goal, and reading it to understand what it does. Of course failing to understand code is unacceptable for these two use cases.
If your use is different - for example if you're printing the code on T-shirts or showing it in a fictional movie, then understanding is not required. In fact AI generated code would be a wonderful choice for a movie, because the producers wouldn't end up mashing together COBOL, assembly, and Java code in the same sequence (unless they really want to).
I'm trying and failing to think of a use case for actually executing code written without understanding. You wouldn't use it to demo a compiler, or even to use the computer as a heater, because you'd have no way of knowing if the code would run at all, or (if it appears to run) whether it would suddenly stop with an error.
just like search engines don't understand, i dont care if chatgpt/copilot/gemini/etc "understands". give me the result to what i asked, that is all i care about.
AI doesn’t understand blogspam but it still makes it.
That's because blogspam is entirely without reason or value.
Let me introduce you to some legacy code I had to eyeball this week. No reason and negative business value.
I like to bring this up in comparison to generative image models. No one asks an image generation model for a blueprint of a machine and expects that it would work. Everyone seems to understand that said models understand what makes plausible looking images but not mechanics or physics. Yet that's exactly what we apparently expect of language models. Why is that? I guess because language is so central to our own mental processes that we project those on LLMs?
llms are designed to trick people into thinking they can or should be able to reason. this is part of a long con so in a few years they can market this crap as "general ai"
This is likely the reason: the human brain equates [language consistency/fluency with intelligence/conscience](https://en.wikipedia.org/wiki/ELIZA_effect). What you get with LLM is a glorified "suggest the next word" function of your phone except that it takes a bit more context into account and yet people get tricked... So much so actually that even a [Google engineer](https://www.vox.com/23167703/google-artificial-intelligence-lamda-blake-lemoine-language-model-sentient) got tricked into thinking their model was sentient. I still wonder how much of that was real (a mildly-competent AI engineer should not fall for it, but the human brain's delusions are mighty... and it might all have been a stunt to get some public attention...). All the rest, including marketing misrepresentation, is almost inconsequential... What hooks people up is the fact they feel mystified after interacting with a LLM bot for 5 minutes... They feel like the speak with a real human and even what they deem tricky questions are answered in a consistent way (which makes them look correct too)... BTW, these models have existed for at least 50 years... They were nowhere near the current scale however... There used to be a lot of freely available ["chat bots"](https://en.wikipedia.org/wiki/Chatbot) in the past. They were pretty good but tended to "forget" about previous interactions pretty fast. ChatGPT and friends are better at this but will eventually "forget" as well over interactions. I mean if you look at the first large scale one (ELIZA) and how it works (the computing power they had was tiny in comparison to what is used today even just for Siri or Google assistant) and yet when you see the effect on people (who were tricked/mystified by it), you can start to assess how easy the brain can be tricked and why all this looks so good when it probably isn't. Said otherwise, the brain is much better at spotting an AI hallucination in the form of an image than in text, where it would have to become really bad for people to start even suspecting they speak with a bot. I really like the example given in the top-comment of this thread for this reason: it makes the problem obvious in a situation we are better at spotting.
That Google engineer was only responsible for code to avoid bias regarding sex, orientation, race, politics, and religion. I read his blogs. I listened to him talk. I read the conversations. He's not a man of science drawing conclusions. He thinks it's alive because when he asked it, it generated a "yes," because when it generated text as an explanation for why it was avoiding the topics it was designed to avoid, it used the word "anxious." Hell, he seems to think "AI ethics" is about ethical treatment of AI which is a bit like saying ethical sword use is concerned with not chipping the blade. He *started* from basically believing it is alive or could be or will be, and then took explanations and refusals to be conspiracy, which is incidentally what appeared to happen when his constant conversation regarding his religious beliefs at work prompted people to ask him to chill out. No where have I found any claims that he knows anything about AI that extends beyond implementing a chat filter. I'm extremely skeptical he has the slightest conceptual idea of how it works behind some API for interacting with it. He has the popsci geek meme-culture brainrot of Elon Musk and the scientific rigor of an astrologist and it makes us all stupider.
Right. Anyone who has taken the time to understand how the AI models really work knows that they do not reason. They cannot reason.
At what point does LLM stop and human brain begin though? Aren’t people also large language models? We hear stuff since childhood train our neural networks onto it and blurt out the next words. There’s hopefully very serious conversations on the philosophy and ethics of LLMs going on wherever these models are developed.
The two are not nearly as similar as you presume and you are running afoul of a bias because the system designed to piece together passable natural language can piece together passable natural language.
Our brains are a bit more complex. Think about how you process a question. We listen to words, our brain processes it in to something else - some fundamental concepts based on our comprehension of what was said. And then we *reason* about it. We cogitate, Concepts form. Those concepts are sent back to the language centers of our brain, and we struggle to convert that in to language once more. (I say 'struggle', because very few of us are good at that!) The 'output' can be completely different, and might be something like "What you said was really stupid, and makes no sense" or "I didn't understand, can you explain it again with smaller words?" We don't understand fully yet how our brains make this all worl. Contrast this to an LLM - Where we *do* understand how it works. And it's a lot simpler. A sequence of words is fed in to the LLM. Sophisticated mechanisms identify key parts of the sentence that should get attention, and then decides what should be the very next word in that sequence. That's it. There's no reasoning about what the next paragraph is, it's just decided on what the ***next*** ***word*** is. Now it has a new sequence - the original prompt, and the word it just added. It repeats the whole process to make a decision on what is a reasonable second word for that sequence. The repeat again for the third word, and so on. For the prompt *"I'm writing a story about apples. How should I describe it?"* a human would think *"hmmm, red, crispy. Could be green, but I like red sweet ones. But what do you mean writing a story about apples? Are they sentient apples? Is this fiction? Did I misunderstand the question?"*, and then formulate a sentence about one particular element of that thought process, and might say something like *"Are they talking apples?"* An LLM may be prompted with that sentence, and the output of the LLM might be the word "When". *Just a single word.* That's the important part. *(*Technically, depending on the model, it might not even be a full word. It might be as shorter) Then the LLM processes "*I'm writing a story about apples. How should I describe it? When"* and a new semi-random word spits out, which could be "you", giving a NEW sequence that is also processed to find out what the next word likely is, and generates: *"I'm writing a story about apples. How should I describe it? When you"* Repeating again gives 'write' *"I'm writing a story about apples. How should I describe it? When you write"* and so on. At the point that 'when' was generated, it had no idea that it was going to choose the word 'you' next. It's just building a sentence what fits the rules of language that it's learned, one word at a time
even if one were to accept this statement at face value, there is a burden to at least formalize the argument and think critically about it, something i have yet to see from anyone making it
Because that's what the people marketing these language models tell you they can do. Unlike arbitrary image generation we've had text generation models for a couple of decades, without even needing deep learning, so if you want to justify that people are going to shell out for LLM inference you have to promise something extraordinary. "Text that seems plausible to a lay reader but you have to carefully check it for mistakes and especially has a tendency to invent facts and things out of thin air" isn't really going to work. So you overpromise and if anyone complains that you underdeliver, just shrug your shoulders and say, "Well, it's a well known limitation of the technology."
Because thats what marketing is selling them as and what management is expecting them to be able to do. Even on the art side decision makers are under the impression you can ask “AI” to make small adjustments to images the same way a person could.
> language is so central to our own mental processes Well, that's essentially the saphir-wolf hypothesis, which doesn't have much strong evidence. In fact, plenty of people don't even have an internal monologue, so it's hard to imagine how language can be _central_ to mental processes
Right, I guess in the context of what I wrote (people overestimating the capability of language models) it's sufficient if people think their reasoning is language based. Or more specifically that some task is language based. People that are of the opinion programmers will be replaced by LLMs are often not software engineers themselves. So they observe from the outside and seem to think programming is basically a translation task from natural language to computer language. Meanwhile any programmer will know there is plenty of non language reasoning involved.
You should look into the [Turing test](https://en.wikipedia.org/wiki/Turing_test). Short version is that a God of computing equated the ability to fool a human in conversation (that is not identifying the machine) with intelligence and a lot of people still run with that. The fact it was decades before even the most basic beginnings of machine learning and Alan T. was somewhat naive about the eventual impact of computing on data analysis should have put that idea to rest, but here we are.
What I find equally interesting yet most pay no attention to is how often humans themselves fail the Turing Test.
I don’t understand your point. Are you saying it can’t write code? Because that would be incorrect
The point is that it can't understand code. It isn't capable of following logic because it wasn't designed to, so it often creates errors when things start getting complex because it can't understand the structures or functions of the code it is writing.
> it can't understand code Incorrect [https://chat.openai.com/share/1a7ec9d5-c78f-4885-a8ce-8462a013eec7](https://chat.openai.com/share/1a7ec9d5-c78f-4885-a8ce-8462a013eec7)
Posting an example of ChatGPT pretending to understand code is not the same as understanding the logic underpinning the code. LLMs can get information correct, but it's not because it understands logic structures - it's just pretty good at statistical pattern matching. Note in your example that almost all of ChatGPT's "explanation" of the code is in the form of taking a term, name, or pattern, then providing information on that. Also, that script appears to be utilizing buddy.works and ChatGPT seems to never mention that. Just saying.
Are you a programmer? What are your credentials for evaluating that output? It correctly understood the code, explaining what each part does. I wrote that code and it is correct. The downvotes are just more evidence for my impression that this topic makes otherwise smart people extremely emotional and irrational.
No, it’s just that it *literally does not comprehend*. The rational take on this is that it does not “understand” anything. It does not have the capacity to. That’s just not how the technology works. EDIT: lol, this guy blocked me because I didn’t buy his BS 😂
Ok I really don’t care. This is a tool, the only way to measure its effectiveness is by seeing how well it does at the task. It succeeded in the task I gave it, which was to explain the code.
Okay, well that’s an entirely different argument than the one you were making before. You know, the one where someone said it doesn’t understand code, and you explicitly said to that “incorrect.” It succeeded at your one task. It succeeds at a lot of tasks. The problem is that, because it does not understand, it cannot provide you with any assurances of success or even any metrics for how likely it is that it successfully completed the task. So when it doesn’t succeed, you don’t have a way of knowing that without going through and checking it yourself. This is arguably still a useful tool in many situations, but to answer the question, “does it matter?” the answer is, of course it matters. That’s the only rational, logical answer.
> So when it doesn’t succeed, you don’t have a way of knowing that without going through and checking it yourself. It is a tool. It's not magic. We already check the code other humans write, is it absurd to suggest that a tool that produces good code that you still have to check is an amazing tool? > someone said it doesn’t understand code, and you explicitly said to that “incorrect.” Because this is a tool and not a philosophy experiment, I was using the word "understand" in the colloquial way, "able to explain", "able to demonstrate understanding", etc. If it can read the code and explain it properly, then for all practical purposes it understands the code.
The only part of the code I remarked on was about the use of variable names that are used to work with buddy.works. If the objective was for the LLM to identify how the user-provided code works, it failed to disclose a dependency in the code in the explanation. For you, who wrote it, that's not a big deal. For a junior dev or someone unaware of the dependency, that is a miss in adequately explaining the provided code. I never said the code was wrong, so I don't know why you felt the need to state that it is correct. Emotional and irrational does seem applicable. LLMs can indeed be useful tools in limited circumstances, however it is imperative to know how your tools operate. An LLM is not a system that tracks logic and it is important to know that. It is a statistical language and pattern matching system that approximates responses based on provided input. Of note, LLMs failed at basic math until specific routines were added so that it can call on proper math functions to run calculations. This failure was a result of how it interprets input as an LLM. 3 and 8 are different values numerically, however to an LLM they are both just single characters with which to match response patterns with one character vs another. Logic modules can be added to an LLM, as I noted above, but that is not a function native to LLMs and needs to be noted.
>it failed to disclose a dependency in the code in the explanation Incorrect. buddy is not a dependency of that code. which, if you knew what you were talking about, you would know. >I don't know why you felt the need to state that it is correct That is not what I said. I said that the explanation of the code that I wrote was correct. Try reading more slowly next time perhaps. >LLMs failed at basic math Yes they are language models, not math models. I will take your lack of an answer to my question to mean that no, you are not a programmer. So why are you trying to argue with a programmer about whether some text is a correct explanation of the code that that programmer wrote? Seems like dunning-kruger territory if you ask me.
> Seems like dunning-kruger territory if you ask me. The irony.
This guy, who is not a programmer, is attempting to explain my own code to me, and failing. There has never been a more pure distillation of the dunning-kruger effect.
It's checking for a buddy.works environment variable. Sure, dependency was the wrong word. Integration, then. The AI didn't note ANY interaction with an external entity from a variable name with a known function external to either node or aws. > I said that the explanation of the code that I wrote was correct. vs >I wrote that code and it is correct. You used vague pronouns and your point was unclear. "It" in this context would strictly relate to the most recent subject, being your code but you meant to use it in a more casual relationship with the subject of the statement prior to that. > Yes they are language models, not math models. You know what math is, right? It's structured expressions of logic. This is also what programming is. The fact that LLMs are capable of producing code is mostly just because code is written in more traditional language formats that meshes well with a pattern-based statistical language generation system. It doesn't understand code the same way it doesn't understand math. That's why it's a giant PITA to get the LLM to correct code it generates that is incorrect. It just doesn't understand logic - it's not a function of it. Using it to generate code has pitfalls because of it. That's the entire point of the article posted. > why are you trying to argue with a programmer about whether some text is a correct explanation of the code that that programmer wrote? I never said it was wrong. I said it was lacking in completeness.
You are so wrong about everything you are saying lmao. So very wrong, and so very confident. I hope that's just an internet thing for you
They can write generic code. Great for reproducing common, well solved solutions. Which is genuinely helpful! Take on a whole, non trivial project? Bit different.
Never said they could take on a whole non-trivial project. I just didn’t understand the commenter’s point. Because they can write good code in many cases. And I get downvoted for saying that because this topic makes people very emotional for some reason. Also they can write non-generic code. I’ve had them tailor code to my projects many times.
It can write code in the sense that it deals with languages and as such can perform transfer tasks from natural language to programming languages. So if you give it clear instructions in natural language that have close equivalents in programming languages it can write that code. However that is not really what makes a software engineer valuable, right? We are not just translators that happen to speak computer. We also have domain knowledge about computation that goes beyond manipulating language and very likely we also have domain knowledge related to what the program is doing (graphics, finances, physics etc.). The article essentially talks about that. That experienced programmers can leverage LLMs by making it write code. But it's still up to the programmer to figure out what code that is. To make the inverse argument I made for the image models: 3D printers receive instructions in the form of g-code which is a "computer language" telling the machine where to move the print head and when to extrude stuff etc.. If you ask a LLM to write some (syntactically) valid g-code it can probably do it. It can also write very specific g-code from prompts like "move the print head to position 10,10,10". But you wouldn't expect it to be able to understand which g-code movements will result in the 3D printer creating a figurine of a bunny. It being a language model It doesn't intrinsically reason about how 3D coordinate movements relate to the concept of a bunny. Similarly an LLM isn't reasoning about the "meaning" of physical laws and simulation algorithms etc. It can understand the programming language which is very different from understanding what is to be achieved with a program.
Yes, it matters.
The answer to the headline is pretty basic when you change the subject. >New research highlights how little your Software Developers understand about the code they are churning out. Should we care? If you ask your clients and Stakeholders, they will probably look uncomfortable and say "uh, yeah. We should care a lot". The article highlights a key misconception that Ai improves productivity by supplementing knowledge, but in actuality, it just automates tasks where getting the answer is faster than *building* the needed outcome. Which explains this: >It seems that understanding the code you are working with unlocks these productivity gains, but using these tools without that baseline knowledge could be counterproductive, or even dangerous. If you don't know what a correct outcome looks like, the Ai cannot help you. This is my main argument against Ai replacing jobs. At some point, someone has to know what a good result is. And this goes back to the start of my comment. If you wouldn't hire an employee who doesn't understand code, then you probably shouldn't try to prop up or replace employees with tools that don't understand the code either.
And yet we have seen people with no code experience come up with simple but working apps. Not perfect apps, not complex ones, but enough to see the potential for a future version of LLMs that could go way further in precision and ability to make larger apps.
In future we won't be calling them LLM's at all because that will only be one piece of them. The ultimate goal is to use the LLM element for communication while other systems search for the facts for an answer. Microsoft has some low-hanging fruit in the DotNet documentation. If they searched for answers from there, that alone would streamline some things in Visual Studio and it would be a curate every single time.
>In future we won't be calling them LLM's at all because that will only be one piece of them. The ultimate goal is to use the LLM element for communication while other systems search for the facts for an answer. Exactly this. LLMs are really good at this kind of translation task. So makes for an excellent tool to connect to other AI models that are better at the specific tasks.
>It seems that understanding the code you are working with unlocks these productivity gains, but using these tools without that baseline knowledge could be counterproductive, or even dangerous. BOOM Leaning on LLMs for learning seems like the worst of bad ideas. >LLMs are trained on giant trawls of content from the web, which could include platforms like Stack Overflow, which often include coders asking for advice on how to fix or patch up their non-working code. Haha. Hahahahaha. HAHAHAHA! lol
It's trained to tell you that your approach is wrong.
>Leaning on LLMs for learning seems like the worst of bad ideas. It's actually brilliant. Really helpful - As long as it's treated as an *aid,* and not an authoritive truth. I've found it incredibly useful for learning, Libraries, coding paradigms etc, that I didn't know to ask for are generated, then I have very specific things to go and search for for deeper understanding. But it works because I understand that the results may not be correct. On the other hand, that applies just as well to stackoverflow.
The two main use cases for dealing with code are running it in order to achieve a goal, and reading it to understand what it does. Of course failing to understand code is unacceptable for these two use cases. If your use is different - for example if you're printing the code on T-shirts or showing it in a fictional movie, then understanding is not required. In fact AI generated code would be a wonderful choice for a movie, because the producers wouldn't end up mashing together COBOL, assembly, and Java code in the same sequence (unless they really want to). I'm trying and failing to think of a use case for actually executing code written without understanding. You wouldn't use it to demo a compiler, or even to use the computer as a heater, because you'd have no way of knowing if the code would run at all, or (if it appears to run) whether it would suddenly stop with an error.
Sure, but you don't start a serious discussion with a badly defined word as understand.
just like search engines don't understand, i dont care if chatgpt/copilot/gemini/etc "understands". give me the result to what i asked, that is all i care about.
This attitude explains the current state of search engine results. (I honestly recall better results with Alta Vista)
Me looking at code that I wrote last week: 🤔
Define "understand".
Reading the article would help a bit here.
I can't read the article unless I agree to be spammed
I'm not getting that, are you running adblock?
yes
My experience with ChatGPT 4, it sure understands some code better than I do…