Arthur T Knackerbracket has processed the following story:
The United Kingdom's Government Digital Service (GDS) has found that giving civil service employees access to Microsoft 365 Copilot saved them an average 26 minutes per day on office tasks.
Microsoft 365 Copilot provides generative AI assistance with various Microsoft Office applications like Word, Excel, PowerPoint, Outlook, and Teams. It allows workers to accomplish some tasks through a natural language chat interface instead of mouse movements and menu clicks.
UK Technology Secretary Peter Kyle discussed the results of the study in a presentation at SWSX London.
"Whether it’s helping draft documents, preparing lesson plans, or cutting down on routine admin, AI tools are saving civil servants time every day. That means we can focus more on delivering faster, more personalised support where it really counts," said Kyle in a statement
The GDS ran a trial of Microsoft M365 Copilot with 20,000 government employees from September 30, 2024, through December 31, 2024.
Based on self-reported data, the resulting study [PDF] showed fairly consistent time savings across professions and organizational ranks, though precise tool use and benefits varied.
"Over 70 percent of users agreed that M365 Copilot reduced time spent searching for information, performing mundane tasks, and increased time spent on more strategic activities," the report says.
"Perceived concerns with security and the handling of sensitive data led to reduced benefits in a minority of cases. Limitations were observed when dealing with complex, nuanced, or data-heavy aspects of work."
The report claims if the reported time savings were replicated across a full working year, "users could save 13 days."
[...] The study didn't investigate whether the workers used this extra time to do more work, take extra time for lunch, or head off to the pub early. "Due to experimental constraints it was not possible to identify how time saved was spent," the report says.
(Score: 4, Informative) by looorg on Sunday June 08, @11:43AM (4 children)
Isn't the keyword here "AVERAGE", so there were some that perhaps "saved" even more then 26 minutes. But there will also then be people that probably didn't save a single minute per day but instead wasted minutes on Copilot (and similar). But those are not mentioned.
Also since they can't say what the saved minutes was used for does it really matter? There is no verification of the potential saving. Did it really even happen? Did they save 26 minutes, on average, so that they could go to the pub earlier then they did previously so does the saving really matter? Getting to the pub early tho could be a great benefit, for them personally.
It's a bit odd that they just merge all the none and potentially negative numbers into one group. You can literally not report that you lost time per day using the tool. This is a great way of hiding potential losses. Still it's interesting that more then half the people claim they saved at least 11 minutes per day on this. 11 minutes. Whooo! That isn't even time to go to the bathroom, take a tea-break or run down to the pub for a pint.
If you read the Register article I guess they don't believe the claim of 13 saved days per year, believing it to be more like 4-5 days. Still that is basically a work week. Not to shabby. But it then puts into question if Copilot can't do maths? In large it seems these suggested productivity gains are largely inflated and based on some kind of wishful thinking. They so desperately want this to work and be true that they will fix or adjust the studies so that it is "true".
Having a look at the PDF report the time period for the study was three months. It's mentioned in the summary here to. Which include the Christmas and the new years holiday. I guess nobody, or a lot less civil servants, worked then. So the number of actual working days would be even smaller for large amounts of the included 20000 government workers. Potentially skewing the average. But lets not even dig to deep into their methodology. Self reporting. Feedback surveys. Their boss asking them if they think they benefited from this tool ...
Also buried in the end they note that a lot of the participants couldn't really commit to this in general since they had their actual full time job to perform and doing this thing would steal time from that so it was probably at most done half-hearted or to keep the boss-man or Copilot-Clippy from nagging them.
How much time was spent verifying the results that Copilot gave or produced? None? Was that included or would that totally devastate the average gain? It's not mentioned anywhere as far as I can tell so I can only assume that they just trusted the results and gave about zero thought to that aspect.
That is a very positive spin on it all. We still don't know if or how much time they wasted. Also as noted these are self-reported numbers. If they claim they saved more then an hour per day, were not their job before then very un-optimized or badly managed time wise? If Copilot can save you an hour a day then something is really wrong with your job. It must contain a lot of summarizing bollocks or tasks that perhaps just wasn't needed in the first place.
By profession (page 13) it seems that "project delivery" was the timesaver. Unclear what that is but I assume here that Copilot wrote their report and summarized the latest Teams meeting for them. Instead of googling for answers they copiloted for answers. This also seems to be correct if one checks the user feedback where most of the selection blurbs seem to indicate that they use it to search for information and have the "AI" write or summarize text for them.
Beyond all this -- how much did Microsoft pay? Cause this all sounds like one of those sponsored white papers, or more like a brown paper since it appears to be utter bollocks.
(Score: 5, Insightful) by VLM on Sunday June 08, @11:53AM (2 children)
I would agree with and extend your remarks from mere labor hours to every other measurable metric. If each month of AI loses you 10% of your customers from "occasionally messing things up"... Or AI mistakes take longer to fix than human mistakes so how is increased labor in some other department reported, or how are worsening workplace metrics reported, etc?
A company can go bankrupt pretty fast by "saving" 10% on labor while simultaneously losing 10% on top-line revenue, for example.
(Score: 5, Insightful) by looorg on Sunday June 08, @02:09PM (1 child)
Perhaps that is why they did this and do similar studies on government entities. In a lot of countries the government can't go bankrupt. They can always just print more money, claim more taxes ... Their customers /citizens/ can't really complain or do to much about it either. It's not like they can go to the competition for better service or more favorable outcomes. They can elect someone new next time. But the change is really minor.
(Score: 2) by Unixnut on Sunday June 08, @05:21PM
All that and government employees are not exactly the paragon of efficiency, nor are most of their jobs particularly complex. I swear I've seen some government positions which could have been automated away by a Perl script, let alone "AI". A lot of these jobs are "make work" type jobs to reward loyalty to the ruling party.
In that context having AI save 26 minutes a day in government entities seems quite a poor result (especially considering the energy and other resources on the back end to create the AI in the first place), but it does give a nice bite-size metric that can be issued in government reports when it comes to lobbying for the next budget allocation, so no doubt MS is angling for government contracts here.
(Score: 3, Interesting) by Gaaark on Sunday June 08, @04:38PM
But because they 'reported' this, they probably get a reduction in MS TCO somehow:
Just like when MS funded a study that found that, what, a 10 year old unpatched linux was less secure than a brand new not seen in the wild Windows, fully patched for what issues testers found: it was quickly picked apart and laughed at by IT people, but MS salesmen were able to take it to the clueless buyers and say "Look at THIS!"
Somehow i see MS marketing behind this...
--- Please remind me if I haven't been civil to you: I'm channeling MDC. I have always been here. ---Gaaark 2.0 --
(Score: 3, Interesting) by VLM on Sunday June 08, @11:49AM (1 child)
Here's the usual innumeracy report:
I checked google and it seems list price for business-class copilot is $30/month.
26 minutes/day * about 20 days/month / 60 minutes/hr is about 8 hours/month
So, very handwavy, if your employees are paid more than $3.50/hr then buying copilot would appear to pay off.
There is the alternative explanation that people are aware of the risks of AI and only do "the stuff that doesn't matter" using AI. In which case the company would make vastly more money by simply not doing those make-work tasks rather than having AI do it.
Two other math related interesting concepts are distribution among workers and over time. Across workers it would be very interesting to see who's "phoning it in" by outsourcing their job to AI vs the coworkers doing their job. Someone who quiet quits and has AI do all their work gets credit I suppose for 8/hrs/day and an "average" of 1/16th of a work day might imply 1 in 16 workers have quiet quit. In some industries that might be very low. The other distribution is over time. Businesses are famous for not using automation tools and insisting on manual processes so I can totally see someone replacing a single task of manual data entry with using the AI to extract-transform-load another platform. Might be possible to replace the entire AI with an extra checkbox on some existing app. Or maybe these people are using it as a very fancy spam filter on literally every email (summarize this email and give me a list of my personal action items, often the result is "blank space")
From the top down companies are very automation focused but from the bottom up the world is full of people hand typing instead of cut and paste, using a hand calculator to type numbers into a word processor instead of using a spreadsheet, hand typing letters/emails instead of using mail merge, spreadsheet-as-a-database, etc, all of which would seem to benefit from "AI". Companies always stand in the way of progress (at least from bottom up) so they've successfully forced employees to do the above as business processes and the employees can VERY temporarily work around that block by implementing AI but the companies will surely find a way to stop them and go back to manual processes.
(Score: 5, Insightful) by Anonymous Coward on Sunday June 08, @03:49PM
I once walked past the desk of a woman I was moderately friendly with while she was working in excel. I was curious as to what she was doing, as she was manually adjusting columns of figures with occasional digits from the next column.
She told me it was a report she had to do each day. It was originally formatted to print out, and somebody had set up a process for her to print it to file. Then she had to import it to excel, using fixed width fields.
Trouble was some of the numbers were big enough to push digits into the next field, so she had manually cut characters from the start of the next field and paste them back onto the end of the number they came from. Then she had to remove the rows with the redundant column headers and page numbers. It took her about an hour a day to fix it. Every day.
I went back in to IT and spent 5 minutes adding an option to the report to print it as tab separated values (some of the text fields had commas), one set of headings at the top, and showed her how to import it. I don't know what she did with the extra hour, as I left not long after.
The moral of this story is that 26 minutes is nothing. Bullshit jobs abound, and all pretend AI is going to do is make more of them.
(Score: 4, Insightful) by YeaWhatevs on Sunday June 08, @11:10PM (2 children)
Problems abound.
* Self reported stats are always grossly exaggerated.
* This study says nothing about the quality of output. The writer would like us to assume that "all else is equal", but we've all seen how terrible copilot is.
* 26 minutes is actually very little improvement, especially when we begin comparing it to other options, such as actually being good at your job.
(Score: 0, Insightful) by Anonymous Coward on Sunday June 08, @11:23PM (1 child)
You should add... this study says absolutely nothing about output. If per-capita output was measured, since these are government jobs after all, we should see 3 days of extra output, and we should see that the amount of output is statistically significant compared to previous years around the same time. After all, if you don't get an improvement on the bottom line, what are you getting out of this, except possibly a kickback.
(Score: 3, Insightful) by YeaWhatevs on Monday June 09, @06:23AM
How is this trolling? I would actually like to see if all of those self reported savings add up to more output when measured. That's the theory right? Time saved means more time put into production, but in the end the proof of the pudding is eating it.
(Score: 2) by wirelessduck on Wednesday June 11, @05:18AM
I'm calling bullshit on this study's claims.
https://pivot-to-ai.com/2025/05/30/the-uk-will-totally-replace-two-thirds-of-junior-civil-servants-with-ai-chatbots-says-the-chatbot/ [pivot-to-ai.com]