Amazon CEO Andy Jassy revealed in a LinkedIn post on August 22 that the company was able to integrate Amazon Q, its generative AI assistant, into its internal systems to update its foundational software.
The result has been a "game changer," Jassy said.
"The average time to upgrade an application to Java 17 plummeted from what's typically 50 developer-days to just a few hours," he wrote. "We estimate this has saved us the equivalent of 4500 developer-years of work (yes, that number is crazy but, real)."
The AI is not only fast but seems pretty accurate, too, according to his post. Amazon developers shipped 79% of the AI-generated code reviews without any additional changes, Jassy wrote.
[Also Covered By]: MSN Money
What do you make of this story ? Do you believe that this has more substance than hype ??
(Score: 5, Insightful) by Unixnut on Tuesday August 27 2024, @12:48PM (3 children)
Under what circumstances? LLM's can only regurgitate what has been sanitised and fed from other sources. That means that their AI assistance for the code upgrade to Java 17 was only possible because the LLM was able to draw upon the work of other human developers whose work in upgrading to Java 17 it ingested.
The day I can hand an "AI Assistant" a copy of the porting notes from the company and the old codebase and have it correctly upgrade the code is when I will really sit up and take notice.
That does not mean it is useless. I remember from the time porting python2 to python3 that a lot of the code changes are repetitive. Either simple search/replace (so simple a normal script could be used to do it), to variations on a simple rule (which needed human intelligence to apply correctly but was otherwise really mundane work).
If you can use the LLM to correctly infer the correct variation needed, then you may well find that upgrading a codebase will progress much faster with the AI assistant doing a lot of the drudgery and the developers mostly reviewing and correcting the code.
If their code review was robust then perhaps it is a great improvement, but we will only find out if they come back in 6-12 months time and report on what bugs appeared as a result of using the AI assistant (that may have been too subtle to notice with a cursory dev review) and whether the man hours they saved will be offset by the man-hours needed for support/debugging/bugfixing afterwards. To me it sounds like it is too early to tell.
(Score: 3, Touché) by bmimatt on Tuesday August 27 2024, @06:05PM
Whatever the LLMs produce needs to be reviewed in detail by someone competent and that, in my experience, can take as long as writing whatever code the LLMs can produce by myself.
That said, there's also this [wikipedia.org]
(Score: 2) by pkrasimirov on Tuesday August 27 2024, @10:21PM
"We asked LLM and after a few hours it said it's fine. Our QA team says the same but it take them few days to do so."
(Score: 0) by Anonymous Coward on Wednesday August 28 2024, @01:34AM
Perhaps the proof will be in the pudding? You know how Amazon shopping pages offer:
> Customers who bought this item also bought
Just watch to see how crazy or off the wall the next rounds of recommendations are. My guess is the AI will get these kinds of things all crossed up. And the few human coders left won't notice, or care.
(Score: 2) by srobert on Tuesday August 27 2024, @12:53PM (1 child)
When you report how much time you saved in "developer-years" instead of "man-hours" it seems to obfuscate the fact that the time is money and the developers are wage earners who will not be paid.
(Score: 2) by zocalo on Tuesday August 27 2024, @01:45PM
Translate that into a third party looking to use Amazon's platform, and yeah, maybe you're going to avoid paying as much in developer wages to get the job done in less time, but how much are you going to have to spend on Amazon services, other staff costs like planning & managing the process, procurement & commercial aspects of using Amazon's services, legal (you're letting Amazon re-write code for you), and so on? Or, putting it another way, how much of your savings are going end up going to Jeff Bezos, Andy Jassy, etc.?
UNIX? They're not even circumcised! Savages!
(Score: 5, Insightful) by bart on Tuesday August 27 2024, @12:59PM (6 children)
I worked on Kotlin at the time, and the AI suggestions were sometimes useful, sometimes great, and often mediocre, and not even compiling.
I quit using it after a month, went back to regular predictable code completion. I found that interpreting and changing or fixing the suggested AI code was more time-consuming than just typing the code myself.
I'm a very experienced developer (40+ years with dozens of languages), and generally am very fast at creating correct and understandable code. My experience with the AI assistant might have been different 20 years ago.
(Score: 2) by turgid on Tuesday August 27 2024, @01:16PM (1 child)
I'm not as experienced as you, but from what I've seen, those AIs don't do even as much as the scripts and macros I've developed to help me write code faster. What's amusing is the people using the AI output despite not understanding what it has produced.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 3, Touché) by ikanreed on Tuesday August 27 2024, @04:20PM
You know how an entire generation of developers got their jobs to turn half-working spreadsheets that the whole business depended on into something actually useful?
The next generation of coders gets to fix unmaintainable mass of barely functional happy-path-only AI-written scripts.
(Score: 4, Informative) by bzipitidoo on Tuesday August 27 2024, @01:16PM (3 children)
> not even compiling.
That's the part that I found shockingly unintelligent. In one case, the snippet of code it gave me cut off in mid line. You'd think it would know the very basic rule that in curly brace languages, lines must be terminated with a semicolon or a closing brace, but evidently it doesn't have any sense of that.
I messaged the AI that it ought to have tried compiling the code itself, to be sure it would compile, and it responded that it wasn't allowed to do that. Not allowed?! I suspect the AI was programmed to respond with a b. s. answer to questions of that sort. Maybe it meant (insofar as its output can mean anything) that it wasn't capable, or didn't understand what I said.
(Score: 5, Insightful) by turgid on Tuesday August 27 2024, @01:32PM
If the AIs are trained on random git repos all over the Internet they are going to ingest all sorts of broken and unfinished code.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2, Insightful) by shrewdsheep on Tuesday August 27 2024, @01:35PM
These are LLMs which precisely do not know the rules of the language. If you give a grammer to an LLM and ask whether a given text can be parsed with that grammer, it will fail but for the simplest grammers (haven't tried myself, but I am willing to wager a bet).
Given how much boilerplate Java is, I can imagine that it might have saved some time in the migration Amazon mentions, but I was shocked to learn that the LLM would review the code. Probably journalistic mishap, but that would be the last thing I would recommend an LLM for.
(Score: 1, Troll) by https on Tuesday August 27 2024, @02:03PM
Tell me you've never learned perl without saying you never learned perl.
Offended and laughing about it.
(Score: 4, Insightful) by looorg on Tuesday August 27 2024, @02:36PM (2 children)
You estimate? Estimate in this regard is just another, fancier, word for guessing. We guess something equivalent to, so the ballpark is pretty large at this point. Also how long is a developer-year? Is that the same as an actual year or is it about 40hours per week for however many weeks said Amazonian work in a year? Are they more akin to dog-years or just how long is an Amazon Developer Year in real terms? Just wondering. The crazy part is that they couldn't even use some already agreed upon measurement of time but had to invent their own. They couldn't even be very specific about it so it's a guess of an estimate to something that is about equivalent to something else that is equally vague. Great. So you saved some time you say ...
It's interesting that it is apparently passing shipping standards and quality and such, says nothing about how or what those are. But I guess if it compiles or JIT runs or whatever that is what matters. How secure is the code? How readable is it? Does it contain any useful comments? For the poor human developer when he, or she, might eventually have to get in the coding loop cause something broke down, or the AI fucked up whichever comes first.
Also as noted it's a horrible idea for the future. If the AI is now doing all the main coding then who will feed it new code snippets in the future when all the humans have been replaced. Gone on to do other work. Is it going to eat its own slop-code? How is that a learning experience? It probably already is. But where will they find that human that can come in and code anymore. I'm getting something-something COBOL flashbacks. I have not written things now in a long time, but I still get asked every now and then when things somehow breaks or they try making changes or it's supposed to do some new whizz-bang thing that wasn't a thing back then but it absolutely have to be incorporated now!
Sounds great. Pretty accurate isn't that inspiring tho. I guess if your company is just large enough then the little errors just doesn't matter much. That they shipped 79% of code without changes says nothing important. Perhaps it was just the boring code that wasn't important. The trivial stuff. But I'm sure it's impressive in some regard. Or it just says more about their code and Java 17 specifically. Or how bad their army of coders was before, if they could be replaced by AI.
Pretty accurate is only good for shotguns and artillery. For everything else you kind want better then pretty accurate.
(Score: 2) by ikanreed on Tuesday August 27 2024, @04:24PM (1 child)
Let's be even less generous. The "50 developer days" that the first few migrations of java took weren't spent recoding things. They were spent figuring out company level dependency issues that once resolved and documented, weren't going to take long again.
They're letting their shitty LLM take credit for what was probably painstaking and meticulous work of resolving serious and complex questions, by pretending that syntax changes were the wall.
(Score: 2) by looorg on Tuesday August 27 2024, @05:46PM
That is a thing to, we know nothing about how much work went into prepping the LLM for creating this or what it's creating. But it's hopefully a one time investment. But probably not. Tweaks and updates are required eventually.
https://www.amazon.jobs/en/landing_pages/pto-overview-us [amazon.jobs]
But I'm still wondering what a Amazonian Dev Year is. Lets just assume, since we don't know any better, that it's 40h a week. If you have been with Amazon for 6+ years then you get a month of vacation. So 48 work weeks to go per year then. But how much of that is really spent deving? There are all the meetings, I suspect there are a lot of those. Some sick days, either personal or for your kids, a few national holidays, more meetings. So how much efficient coding time do you get per week these days? A few hours per day?
I find it ludicrous that your holiday is set after how long you have been working at Amazon. Nothing about your actual age or experience. Years at the company as far as I can tell is all that matters. But I guess this is perhaps a Euro vs USA thing.
(Score: 0) by Anonymous Coward on Tuesday August 27 2024, @02:53PM
If you use "AI" tools (really, any kind of automatic transformations) in situations where the results can be automatically checked to see if it is any good, then you will probably get decent results.
I'm not a Java expert but porting an existing program from one version of the language to the next version probably fits into this category. Run the "AI" porting tool, check if the results pass tests, if it does, done. If it doesn't, you probably only need to look at a couple places that still have issues. Formal equivalence checking tools would also be useful -- such checks are commonly used in digital logic design to prove that some logic transformation does not introduce bugs but they are rare in the software space.
This "we can't be bothered to make sure version N+1 of something is compatibile with things designed around version N so instead we'll make extra work for everyone else forcing them to constantly change their shit in order to keep it working in exactly the same state it was last month" is one of the most boring and stupid self-inflicted problems in software design but there is too much money to be made solving such self-inflicted problems so there is no incentive to stop doing it.
(Score: 5, Informative) by ElizabethGreene on Tuesday August 27 2024, @02:57PM (5 children)
At my current skill level, I can't do math on how much time Copilot saves me. It's a #div/0 error. There are development tasks I would not even attempt without copilot. As an example, I work in Autodesk's CAD package, Fusion 360, and I'm making a drill index, a drill bit holder. For this part, I want the drill bits spaced a certain distance apart and the closest point of each hole to the edge to be a few mm. The work isn't hard here. Draw a rectangle, create a construction line that defines how far they should be from the edge. Draw a circle, make it tangent to the construction line, dimension the size, and dimension it center-to-center to its neighbor. This drill index has 118 bits in it, and doing this manually is ... a job for someone else.
Copilot will NOT replace developers. Copilot is completely incapable of looking at a problem and identifying a way to make the problem easier for humans. It has no capability to ideate "This would be easy in a spreadsheet if Fusion could eat a spreadsheet". That said, once you have thought of the way to make the problem easier and broken it into actionable steps, Copilot is fucking fantastic at turning actionable steps into code.
Fusion 360 supports add-ins, with Python being the most common. I read a book on Python a few years ago, but am not competent at it. Competent is defined as the ability to make a for loop or foreach over an array or collection without googling it.
I opened a sample add-in that did something completely unrelated (saving STLs), and with copilot and a half-dozen google searches I was able to write an add-in to consume a csv and create the holes of the specific diameter and spacing in a sketch. It took under an hour, with most of that being me putzing around in Bing trying to figure out how to find a specific sketch in a multi-component file. It really helps if you don't have a typo in the name of the sketch.
This morning, I wanted to make an SVG file of labels I can print to add to the index. I typed literally these three lines ...
# CreateLabels.py
# This script consumes a CSV file of holediameter, spacing, and label
# and creates a SVG file with the labels placed at the correct locations
It progressively generated imports for svgwrite, csv, and sys, made a read_csv function, a create_svg function, and bless it's little mechanical heart, fixed it to accept the csvfile and svgfile as command line arguments instead of just hard coding them in there like I would have done. I searched the docs to make sure the svgwrite library actually _existed_, was commonly used, and pip installed it. Then, with no other modifications, I tested it, and it made an SVG. Opening it, copilot had misunderstood the way I wanted the text placed so I fixed that by hardcoding the y position and added code to change the size of the drawing to where it wouldn't be truncated.
Everyone reading this is, unquestionably, a super-developer with a lifetime of python experience. I'm not. I read a book a while ago and have played with some scripts occasionally. The only language I truly know by heart, my lingua franca, is BASIC. That I was able to do this in 20 minutes leaves me absolutely gobsmacked.
(Score: 1, Insightful) by Anonymous Coward on Tuesday August 27 2024, @06:09PM
I think that is a great example of what it can be used for. Perhaps you can even improve your script with importing the csv from the spreadsheet. Excel does Python now as far as I have been told. So you might be able to automate a step in the process. Still I find it a great example of what can be done. Perhaps in that regard CoPilot is a replacement for some kind of low-code/no-code. But slightly more since you have to know or understand somethings.
The only problem I have with it is if it replaces actual coders. After all do you know exactly what your program does? Does it do more then you think but as long as it did what you wanted you don't care how it did it or if it left things hanging. In that regard I think that is my main issue with the story here, how they replace coders with AI in some regard. I wonder what it means for the future. Perhaps it will be great. Or it will be a large dumpster fire. Who knows.
I'm mainly good at obsolete coding and languages nobody cares about or wants anymore. I found modern development soulcrushing and moved on. To many meetings, style guides, "teamwork" and new languages that keep reinventing the wheel for some reason. I'm turning more and more into that guy that Wally met in the Dilbert cartoon -- beard, suspenders and telling people to have a nickle to buy a better computer, the UNIX oldtimer.
I recall there being some kind of joke back in the day that Python was just BASIC but without the line numbers. Perhaps there is something to it. The life time of python experience isn't very long, it didn't become popular until fairly recently or in the last decade or so but it has been around since the 90's. I don't recall it being well received when it was new a few decades ago. Sort of like JAVA. Horrible back then and I guess it tainted my views of it, or both of them.
(Score: 2) by VLM on Tuesday August 27 2024, @06:44PM (3 children)
A job for OpenSCAD "The Programmers Solid 3D CAD Modeller"
https://openscad.org/ [openscad.org]
I imagine that would be a very small script to make a STL to import into your Fusion 360 although I don't use F360 I assume its easy to import stuff in 2024.
Thats the problem with AI, a human will tell you to use another better matched tool (at least as I very briefly understand your problem) but an AI will "obey". "Hey AI write me a real time servo controller in LISP" and it'll merrily provide something that runs (... sometimes), but I wouldn't ship it.
Thinking back over my career some of my most important programming decisions were managing my manager and telling them no. For example, IIRC around the turn of the century I had a boss who wanted to assign IPv4 addresses in the range of 256-999, after all we're wasting a lot of space by only using 0-254 and it should be easy for me to modify my software to permit users to input IP addresses like 987.654.321.999
(Score: 2) by ElizabethGreene on Wednesday August 28 2024, @03:58PM
I've used Openscad a fair amount and it would be another option. It would be a bunch of circle and translate statements, then exporting the 2-d to a .dxf I could import into Fusion as a sketch.
My recollection is that it does circles as a series of line segments instead of as circles, so you'd want to test it to make sure it didn't cause a perf issue, but it would probably work.
(Score: 2) by ElizabethGreene on Wednesday August 28 2024, @06:51PM (1 child)
I agree with you 100% that the human in the loop is important for understanding the meta around what should or shouldn't be programmed. Another example would be if you're asked to port a single-threaded application to run on GPUs while maintaining single-threadiness. It's almost certainly possible, but it's pointless if you can't take advantage of the parallelism that GPUs provide.
An AI "can" write code to do a great many things, but it can't tell you if you *should* (yet).
My daughter is studying CompSci right now, and they're vehemently discouraging the use of AI for development. I think that's a good thing? My gut instinct is that code generation tools pooping out code you don't understand makes it exponentially more difficult to fix it when-not-if it breaks. That said, gcc poops out compiled binaries I don't really understand and digging off into them with Ida is a quick lesson in humility. Has my lack of understanding on how compilers do the needful made my job significantly harder? I don't know.
(Score: 2) by VLM on Thursday August 29 2024, @08:05PM
Those two lines are uniquely linked in that a large part of education is figuring out what questions to ask is often more important than figuring out the answer. And AI as commonly used only provides the answer.
I find it emits inhuman code. Weird orders and weird priorities and just alien sometimes. Yes it compiles but its hard to maintain because its design process is weird and not suited to beginners and sometimes mixes concepts together in a technically operational yet wrong manner.
AI generated code reminds me much of "obfuscated C code" contests from last century. Well yes it compiles and works but you shouldn't be shipping or committing code like that to a repo without an extremely good excuse.
(Score: 4, Insightful) by Mojibake Tengu on Tuesday August 27 2024, @03:12PM (2 children)
Amazon is selling Q as a load on their AWS cloud services. Sounds like pure marketing exaggeration.
Do we have any independent evaluation of its true productivity?
4500 devyears looks bureaucrazy but is not unusual for corps. Most people I know could flick out such a version migration in a person alone.
Though, maybe I am biased since I just know some strange people...
Go find some real programmers if you do real business.
Rust programming language offends both my Intelligence and my Spirit.
(Score: 3, Insightful) by krishnoid on Tuesday August 27 2024, @05:05PM (1 child)
"Independent"? After 2016, how about anything other than some C-suite's say-so? Like a few examples, a demo ... anything? I mean, you can't even trust engineers' time estimtates [joelonsoftware.com] on how long it will take to complete something, much less how much time was "saved".
(Score: 3, Informative) by PiMuNu on Wednesday August 28 2024, @12:43PM
> you can't even trust engineers' time estimates
Your link says that you can trust the time estimates, if they are done right. It fits with my experience as well. Occasionally I get a gnarly bug that takes an unexpected amount of time to unravel (why the **** is it throwing a segv there??) but usually my time predictions are pretty good, when I bother to do them in detail.
(Score: 3, Touché) by Rosco P. Coltrane on Tuesday August 27 2024, @03:23PM
4,500 freshly laid, freshly converted off neo-Luddites.
(Score: 3, Interesting) by Freeman on Tuesday August 27 2024, @08:01PM (1 child)
It sounds like they were just working on migrating support for one version of Java to a new version of Java. Which may actually be a nice use case and something that programmers generally like. Not needing to do X monotonous task a few dozen times in a row for many applications seems like something we like.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 5, Insightful) by MrGuy on Tuesday August 27 2024, @09:00PM
An AI with no deep understanding of what the code actually does can probably get the code to run in the new version.
But the problem is understanding why the code needs to change at all. If it doesn’t work, it’s because something about the language changed. Why did it change? What pattern or approach to writing code changed? Should your code change with it? Is there something complicated in your current code that’s now easy? Is there an approach you used to work around a language limitation that now works properly? Is there an approach you used that’s now deprecated and should be rethought?
If your goal is to get old code to just run, I’m sure an AI is a great quick fix. If your goal is to keep up with evolving languages, then eventually quick fixes will catch up with you.
(Score: 4, Interesting) by Thexalon on Wednesday August 28 2024, @01:32AM
As someone who currently codes for Amazon, I could tell you all more about what this tool is and isn't capable of. But I suspect given the number of developers that they have, this number isn't completely implausible.
"Think of how stupid the average person is. Then realize half of 'em are stupider than that." - George Carlin
(Score: 3, Insightful) by jb on Friday August 30 2024, @06:30AM
Everyone seems captured by the "AI" aspect. The real story here is the load itself:
If it takes 4,500 man years to "upgrade an application to Java 17", then either:
The fact that the CEO is turning that into a story about AI suggests that the CTO probably lacks much of a fundamental understanding of software engineering.