SoylentNews is people
SoylentNews
Can European regulation rein in ill-behaving algorithms?
Until recently, it wasn't possible to say that AI had a hand in forcing a government to resign. But that's precisely what happened in the Netherlands in January 2021, when the incumbent cabinet resigned over the so-called kinderopvangtoeslagaffaire: the childcare benefits affair.
When a family in the Netherlands sought to claim their government childcare allowance, they needed to file a claim with the Dutch tax authority. Those claims passed through the gauntlet of a self-learning algorithm, initially deployed in 2013. In the tax authority's workflow, the algorithm would first vet claims for signs of fraud, and humans would scrutinize those claims it flagged as high risk.
In reality, the algorithm developed a pattern of falsely labeling claims as fraudulent, and harried civil servants rubber-stamped the fraud labels. So, for years, the tax authority baselessly ordered thousands of families to pay back their claims, pushing many into onerous debt and destroying lives in the process.
[...] Postmortems of the affair showed evidence of bias. Many of the victims had lower incomes, and a disproportionate number had ethnic minority or immigrant backgrounds. The model saw not being a Dutch citizen as a risk factor.
[...] As the dust settles, it's clear that the affair will do little to halt the spread of AI in governments—60 countries already have national AI initiatives. Private-sector companies no doubt see opportunity in helping the public sector. For all of them, the tale of the Dutch algorithm—deployed in an E.U. country with strong regulations, rule of law, and relatively accountable institutions—serves as a warning.
The hope is the European Parliament's AI Act, which puts public-sector AI under tighter scrutiny, will ban some applications (like law enforcement's use of facial recognition) and flag something like the Dutch Tax's algorithm as high-risk. Nathalie Smuha, a technology legal scholar at KU Leuven, in Belgium, summed it up:
"It's not just about making sure the AI system is ethical, legal, and robust; it's also about making sure that the public service in which the AI system [operates] is organized in a way that allows for critical reflection."
Originally spotted on The Eponymous Pickle.
(Score: 1) by pTamok on Thursday May 12 2022, @08:30AM (3 children)
What you say is how things should work.
Unfortunately, as far as benefits are concerned, the government saves more money by cutting off the benefits of the 1000 and waiting for complaints. Of the 900 wrongfully accused of fraud, a significant proportion will not have the resources to challenge the assessment. This is how the UK Disability Benefits system works - more than half the assessments that are appealed are overturned: BBC: Half of disability benefits appeals won in tribunal court [bbc.com], and many are not appealed. In certain areas it is worse: BBC: Disability benefits court appeals won four out of five times [bbc.com].
In addition, vast amounts of support goes unclaimed, for various reasons: £16 billion remains unclaimed in means tested benefits each year [entitledto.co.uk] Our annual review suggests about £15 billion of benefits remain unclaimed each year [entitledto.co.uk]
But yes, requiring AI decisions to be reviewed properly by humans would be great, so long as a mechanism is in place, perhaps even the one you describe, to ensure a proper and substantive review takes place, and it is not simply a box-ticking exercise.
(Score: 2) by Immerman on Thursday May 12 2022, @01:12PM (2 children)
Knowing that even 80% of the appealed assessments are overturned isn't very informative without also knowing what percentage of assessments are appealed. It could mean that the system is working correctly in 99.999% of cases, and 20% the people who appeal are doubling down on trying to game the system. (I doubt it, but it would be consistent with such limited data)
The government saving money may be motive not to fix the problem, but probably isn't the cause of the problem - after all saving the government money typically only benefits the politicians who get to spend it on something else, not the the people making the individual decisions, or even their organization (unless there's also a very perverse incentive structure in place)
(Score: 0) by Anonymous Coward on Thursday May 12 2022, @05:07PM (1 child)
"The government saving money may be motive not to fix the problem, but probably isn't the cause of the problem - after all saving the government money typically only benefits the politicians who get to spend it on something else, not the the people making the individual decisions, or even their organization (unless there's also a very perverse incentive structure in place)"
Guess you haven't noticed the trend of conservative politicians to use austerity as a club to destroy social services. The wealthy are happy to let the average person suffer and die than pay more taxes. The old line, Republicans say the government doesn't work then get elected and deliberatelt make things worse. DeJoy should be in jail for the destruction of the USPS.
(Score: 3, Informative) by Immerman on Thursday May 12 2022, @06:29PM
You're not really disagreeing - Republicans are among the politicians that get to spend the money on something else (in their case usually either tax cuts for their rich corporate sponsors, or domestic surveillance programs under the guise of kicking out illegal immigrants)
The person who is actually getting paid to decide if this specific person qualifies for benefits under whatever rules the current politicians have in place though? They rarely have any direct skin in the game, nor does their immediate boss. And probably not their boss either. They're all just middle-men handing out government money that they have no way to touch themselves.