Return to Answer

Because the goal is to produce output that looks like native English text, the models are trained to assign high probabilities to existing text samples, and evaluated based on how well they predict other (previously unseen) samples. Which, for a language model, is a fine objective function. It will favor models that produce syntactically correct text, use common idioms over semantically similar but uncommon phrases, don't shift topics too often, etc. Some level of actual understanding does exist in these models, but it's on the level of knowing that two words or phrases have similar meanings, or that certain parts of a paragraph relate to each other. There is understanding, but no capacity for reasoning.

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody who actually understands NLP is claiming otherwise.²

There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work³). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

² Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.

³ If trained on well-written text

Because the goal is to produce output that looks like native English text, the models are trained to assign high probabilities to existing text samples, and evaluated based on how well they predict other (previously unseen) samples. Which, for a language model, is a fine objective function. It will favor models that produce syntactically correct text, use common idioms over semantically similar but uncommon phrases, don't shift topics too often, etc. Some level of actual understanding does exist in these models², but it's on the level of knowing that two words or phrases have similar meanings, or that certain parts of a paragraph relate to each other. There is understanding, but no capacity for reasoning.

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody who actually understands NLP is claiming otherwise.³

There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work⁴). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

² Where "understands" is shorthand for "encodes the information in such a way that it can condition its decisions (i.e. probability distribution functions) upon it"

³ Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.

⁴ If trained on well-written text

A key thing to understand here is that the question is not, as some have suggested in the comments, whether any AI model can produce correct code. It's whether this one can be trusted to do so. The answer to that question is an unqualified "NO". GPT-3 is a language model. Language models are an essential part of tools like automatic translators; they tell us how probable it is that any given sentence is a valid English (or whatever language) sentence written as a native speaker would (1), which lets us favor translations that are idiomatic over ones that just translate individual words without considering how the sentence flows. The systems can be trivially modified to generate text, if instead of looking up the word you have in the probability distribution it provides, you instead select the next word according to that distribution, which is how these chat bots work.

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody who actually understands NLP is claiming otherwise (2). There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work (3)). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

1. More formally, it gives the probability that a uniformly randomly selected English sentence of a specific length would be this one, but that gives the same ordering over sentences as long as we make some fairly reasonable assumptions.
2. Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.
3. If trained on well-written text

A key thing to understand here is that the question is not, as some have suggested in the comments, whether any AI model can produce correct code. It's whether this one can be trusted to do so. The answer to that question is an unqualified "NO". GPT-3 is a language model. Language models are an essential part of tools like automatic translators; they tell us how probable it is that any given sentence is a valid English (or whatever language) sentence written as a native speaker would¹, which lets us favor translations that are idiomatic over ones that just translate individual words without considering how the sentence flows. The systems can be trivially modified to generate text, if instead of looking up the word you have in the probability distribution it provides, you instead select the next word according to that distribution, which is how these chat bots work.

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody who actually understands NLP is claiming otherwise.²

There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work³). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

¹ More formally, it gives the probability that a uniformly randomly selected English sentence of a specific length would be this one, but that gives the same ordering over sentences as long as we make some fairly reasonable assumptions.

² Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.

³ If trained on well-written text

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody (2) who actually understands NLP is claiming otherwise. There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

1. More formally, it gives the probability that a uniformly randomly selected English sentence of a specific length would be this one, but that gives the same ordering over sentences as long as we make some fairly reasonable assumptions.
2. Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.

Trying to use a language model to generate code is like trying to use a submarine to fly to the moon. That's not what it's for; why are you trying to use it for that? Stop doing that. But at the same time, arguing that the submarine is bad at flying is rather missing the point. Nobody who actually understands NLP is claiming otherwise (2). There do exist systems that are designed to produce code, and trained to optimize correctness. (e.g. Genetic Programming). That's a bit too far outside my area of expertise for me to make any claims as to where the state of the art is on those, so I'm not sure whether answers generated by them should be allowed or not. But if you were to use an AI tool to generate code, that's the sort of thing you should be looking at; they're designed for the task. Similarly, you could ask if language models could be used as a tool to edit questions you've written by hand, perhaps to check the grammar or recommend new ways to phrase answers so they flow better. They'd be fairly good at that sort of thing (probably. I haven't used any of those tools myself (the rambling, stream-of-consciousness answer might have given that away), but the math supports the idea that they should work (3)). Translation is another task where (similar) systems work fairly well. (Machine translations still aren't perfect, but they're much better than they were 10 years ago, and improvement in language models is a big part of that.) Just always be aware of what tool you're using, and whether it's the right one for the job.

1. More formally, it gives the probability that a uniformly randomly selected English sentence of a specific length would be this one, but that gives the same ordering over sentences as long as we make some fairly reasonable assumptions.
2. Well, not many. There'll always be a few who get caught up in the hype. They shouldn't.
3. If trained on well-written text