Why OpenAI’s Codex Won’t Replace Coders:
This summer, the artificial intelligence company OpenAI released Codex, a new system that automatically writes software code using only simple prompts written in plain language. Codex is based on GPT-3, a revolutionary deep learning platform that OpenAI trained on nearly all publicly available written text on the Internet through 2019.
As an early Beta tester, I've had extensive opportunities to put both GPT-3 and Codex through their paces. The most frequent question I'm asked about Codex is "Will this replace human programmers?" With world powers like the United States investing billions into training new software developers, it's natural to worry that all the effort and money could be for naught.
If you're a software developer yourself—or your company has spent tons of money hiring them—you can breathe easy. Codex won't replace human developers any time soon, though it may make them far more powerful, efficient, and focused.
Why isn't Codex an existential threat to human developers? Years ago, I worked with a high-level (and highly compensated) data scientist and software developer from a major American consulting firm on a government database project. Our task was to understand how a state agency was using its database to assign grants to organizations, and then to advise the agency on how to improve the database.
[...] He later explained to me that actually writing code and running analyses occupies about 1 percent of his time. The remainder is spent working with clients to understand their problems, determining the right software and mathematical models to use, gathering and cleaning the actual data, and presenting results. In most cases, the coding and math itself is a tiny, almost rote, part of the software development process.
[...] The day when a non-coder can sit down with Codex, write up a spec sheet, and crank out a working piece of software is still far away.
[...] Systems like Codex may fail when they're pitted against a skilled human developer. But as Codex and its ilk improve, humans who transform themselves into centaurs by combining their skills with advanced AI are likely to become a powerful—and perhaps unstoppable—technological force.
So what do you people think, will Codex live up to it's hype or is this another technology that is perpetually 20 years away !!
Related Stories
The Q&A site has been flooded with ChatGPT coding answers that look correct but often aren't, with moderators calling for a halt:
Stack Overflow, a site where developers can ask and answer coding questions, has temporarily banned the use of text generated from ChatGPT, a chatbot released by Open AI last week.
[...] Since launching, it's been prompted in numerous ways, including to write new code and fix coding errors, while the chatbot can ask for more context when a human asks it to resolve coding problems, as OpenAI sets out in examples. But Open AI also notes that ChatGPT sometimes writes "plausible-sounding but incorrect or nonsensical answers."
This appears to be a key cause of its impact on Stack Overflow and its users who are seeking correct answers to coding problems. Additionally, because ChatGPT generates answers so quickly, some users are supplying lots of answers generated by it without parsing them for correctness.
[...] Stack Overflow says that ChatGPT answers have "swamped" its volunteer-based quality curation infrastructure because there are so many poor quality answers pouring in.
So far, Stack Overflow has detected posts generated by ChatGPT in the "thousands". The other problem is that many answers require a detailed analysis by someone with experience in the subject to determine if the answer is bad.
Previously:
- OpenAI's New ChatGPT Bot: 10 Dangerous Things it's Capable of
- Why OpenAI's Codex Won't Replace Coders
OpenAI, the company behind the chatbot ChatGPT, has ramped up its hiring around the world, bringing on roughly 1,000 remote contractors over the past six months in regions like Latin America and Eastern Europe, according to people familiar with the matter:
About 60% of the contractors were hired to do what's called "data labeling" — creating massive sets of images, audio clips, and other information that can then be used to train artificial intelligence tools or autonomous vehicles.
The other 40% are computer programmers who are creating data for OpenAI's models to learn software engineering tasks. OpenAI's existing Codex product, launched in Aug. 2021, is designed to translate natural language into code.
[...] Previously, OpenAI trained its models on code scraped from GitHub, a repository site owned by its largest investor, Microsoft, which last week confirmed multi billion dollars in new funding first reported by Semafor. But in this case, OpenAI appears to be building a dataset that includes not just lines of code, but also the human explanations behind them written in natural language.
[...] Sam Altman, OpenAI's CEO, recently put the company's headcount at 375 people, a tiny number compared to the thousands of staff at tech giants like Google and Facebook working on artificial intelligence. "I know I'm not supposed to brag about OpenAI," he tweeted, touting the company's "talent density."
Originally spotted on The Eponymous Pickle.
Previously: Why OpenAI's Codex Won't Replace Coders
Related: OpenAI and Microsoft Announce Extended, Multi-Billion-Dollar Partnership
(Score: 2) by sonamchauhan on Thursday September 30 2021, @02:35PM
Microsoft has an exclusive license for OpenAI's GPT-3 engine, on which Codex is built. They are using that technology to power GitHub Copilot, which requires Microsoft development tools.
In the meantime, OpenAI offers Codex beta access but only via an API, for which you got to get on a waitlist.
Anyone know of any half-decent open-source alternatives to Codex?
(Score: 4, Informative) by martyb on Thursday September 30 2021, @02:47PM
-- D.T. Ross; SofTech November 1978
Wit is intellect, dancing.
(Score: 5, Insightful) by DannyB on Thursday September 30 2021, @02:53PM
A developer also needs to work with QA people and others.
Developers need to estimate the size of issues for management planning. Is this a minor quick fix, vs this is a major six week project.
Developing is much more than writing code that will compile.
In fact, the compiler is not your target audience. Here is a tip that I figured out a few decades back. Realize that the code you write is being written for a human being to read. The compiler is not your target audience. A human reader is your target audience.
How is an AI going to comprehend the ideas behind existing code in order to revise it with a new feature?
This might happen some day. But I am not holding my breath. When that happens, it is not only developers whose jobs would be automated.
The lower I set my standards the more accomplishments I have.
(Score: 5, Interesting) by sjames on Thursday September 30 2021, @04:13PM (5 children)
MBAs have been trying to replace software developers with silver bullet software for decades. Remember when CASE tools was going to replace programmers? It turned out that MBAs couldn't replace programmers with CASE tools because CASE tools requires a precise and detailed description of the goal and MBAs weren't able to produce that.
Then UML was supposed to be the silver bullet. Then it was noticed that producing sufficiently detailed UML to produce non-trivial source code took more effort and was more error prone than simply writing the code.
Copilot is also based on GPT-3 but even though it works with a (hopefully) skilled developer, most of the code it produces has security flaws.
I suspect it will always come down to the old joke about the TV repairman. The TV picture won't stop rolling so they call the TV repairman in. He sizes it up for a moment and slaps the TV hard on the side and the picture is perfect. He writes a bill for $75.
The customer says "You want $75 just for slapping the TV!!?!.
The repairman replies "Slapping the TV is free. The $75 is for knowing where to slap it.
(Score: 2) by Thexalon on Thursday September 30 2021, @04:25PM (1 child)
And of course No Silver Bullet [worrydream.com] should be required reading for anybody considering themselves a programmer or developer.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Friday October 01 2021, @03:55AM
It's been a long time since I read Fred Brooks. Thanks for the link and the memories (although that version seems to be full of typos). I met him once, a friend from undergraduate years managed to get Brooks for a PhD advisor.
Most of it is still spot on, although it was first written in 1987 https://en.wikipedia.org/wiki/No_Silver_Bullet [wikipedia.org]
One place where "No Silver Bullet" seems to miss is his discussion of AI, starting at the bottom of page 7 and top of page 8. As I read it, he didn't anticipate the rise of "machine learning".
Discuss!!
(Score: 0) by Anonymous Coward on Thursday September 30 2021, @05:23PM (2 children)
It always fascinated me how slapping the side of a TV could stop the vertical rolling. I have employed that trick many times.
(Score: 0) by Anonymous Coward on Friday October 01 2021, @08:10AM
It makes the problem worse in the long run. The real solution is to either readjust the tuner or fix the cracked joint.
(Score: 2) by sjames on Friday October 01 2021, @09:21AM
In many cases, it was dirt in the tuner and slapping it could potentially improve the electrical contact for a while. Generally a liberal application of contact cleaner to the tuner was the real fix, but slapping was a lot faster.
(Score: 3, Interesting) by Thexalon on Thursday September 30 2021, @04:22PM
"Natural" human language is necessarily imprecise: Humans need to be able to get points across quickly without having to think too hard about it. Coding is necessarily extremely precise: Computers by their very nature don't tolerate ambiguity well, since each possibility multiplies the memory and CPU requirements accordingly. The developer's real job isn't to know coding syntax, but to bridge that gap between ambiguous fuzzy human thought and the precision of computers.
My usual example for describing this to non-coders: Imagine that you are moving your furniture, and you have a bunch of people around to help you. You tell two of them "I want you to move this dresser to that corner of the bedroom right there." That seems like a perfectly clear and precise instruction, right? Wrong. The developer's job is to take that and turn it into, in advance, something much more like: "Note where everything is on top of the dresser, and remove it and set it aside somewhere convenient. If the dresser is too heavy to lift easily, try to take the drawers out to move. Turn the dresser 90 degrees so it fits through the door into that room, move past the bed, turn it 90 degrees again to make it the way that will fit between the bed and the corner with the drawers facing away from the wall, except if you can't make that turn because there isn't enough space go back to where there is enough space and turn 180 degrees so you can then make the 90 degree turn in the right direction, except if there's still not room because of the bed then shift the bed enough to make the turn and then shift the bed back to where it was. Then move the dresser as close to both walls as possible, bearing in mind radiators and other obstructions that might prevent you from putting it exactly in the corner. Once the frame is in the right spot, put the drawers back in if you took them out, and put any stuff that was on top back where it was relative to the corners of the dresser-top when you started."
Once you've figured that all out, translating it into code is the comparatively easy part.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 1, Insightful) by Anonymous Coward on Thursday September 30 2021, @05:51PM (3 children)
0) Most of them don't even know what they really want.
1) And if somehow they do most of them won't know how to specify exactly and reasonably unambiguously what they want even in the language they are most fluent in.
2) And even if they finally manage to do so, what they specified is often wrong, because they're ignorant and incompetent idiots at such stuff.
For stuff that's complicated enough for them to need some other entity's help, there'd be plenty of stuff they don't know they don't know, that they need to know.
Let them talk to the AI directly and literally nothing good will come out of it.
Yeah you'll have the salesman do the equivalent of having the PHBs tell the AI to produce a toy kennel for a dog. Whoopee it looks passable. Then the PHBs spec a shopping mall without the required plumbing, vents etc for restaurants. Or an airport with two of world's tallest skyscrapers...
(Score: 2) by istartedi on Thursday September 30 2021, @09:14PM
So it'll be like having a genie or visiting Fantasy Island. You'll get your wish, sort of. Maybe you'll learn a lesson, but you might also just be disgusted with yourself and the process.
Appended to the end of comments you post. Max: 120 chars.
(Score: 0) by Anonymous Coward on Friday October 01 2021, @08:15AM (1 child)
My favorite request I've ever heard was to "make it do exactly what it does now, but faster." Stupid me must have forgotten to #include <red.h> because everyone knows that red goes fasta!
(Score: 0) by Anonymous Coward on Friday October 01 2021, @01:06PM
> "make it do exactly what it does now, but faster."
We get this request all the time -- working with realtime simulation software (the industrial version of simulation video games). In our case the definition of "exactly" means the same inputs give the same outputs--the program internals are changed to meet the request.
Often there is one bottleneck (or a few) that needs to speed up. We've had limited success with profiling tools over the years.