SoylentNews is people
SoylentNews
A group of researchers at the Chinese web services company Baidu have been barred from participating in an international competition for artificial intelligence technology after organizers discovered that the Baidu scientists broke the contest's rules.
The competition, which is known as the "Large Scale Visual Recognition Challenge", is organized annually by computer scientists at Stanford University, the University of North Carolina at Chapel Hill and the University of Michigan.
It requires that computer systems created by the teams classify the objects in a set of digital images into 1,000 different categories. The rules of the contest permit each team to run test versions of their programs twice weekly ahead of a final submission as they train their programs to "learn" what they are seeing.
However, on Tuesday, the contest organizers posted a public statement noting that between November and May 30, different accounts had been used by the Baidu team to submit more than 200 times to the contest server, "far exceeding the specified limit of two submissions per week."
Jitendra Malik, a University of California computer scientist who is a pioneer in the field of computer vision, compared the accusations against Baidu to drug use in the Olympics. "If you run a 9.5-second 100-meter sprint, but you are on steroids, then how can your result be trusted?" Mr. Malik said.
The episode has raised concern within the computer science community, in part because the field of artificial intelligence has historically been plagued by claims that run far ahead of actual science.
[Related Paper]: Deep Image: Scaling up Image Recognition
Original Submission
(Score: 0) by Anonymous Coward on Saturday June 06 2015, @08:33PM
"If you run a 9.5-second 100-meter sprint, but you are on steroids, then how can your result be trusted?"
The result is that you ran a 9.5 second 100m sprint, you can trust it because you measured it yourself. How does another’s steroid use affect your ability to use a stopwatch?
(Score: 0) by Anonymous Coward on Saturday June 06 2015, @09:15PM
And how does getting advice from a wireless device affect a chess player's ability to move the pieces during a tournament?
(Score: 1, Insightful) by Anonymous Coward on Saturday June 06 2015, @09:26PM
It improves it.
If the goal is to play good moves then all players ought to be computer assisted.
The only reason to forbid it is because the goal isn't to play good games, it's to demonstrate virtuosity.
(Score: 3, Interesting) by physicsmajor on Saturday June 06 2015, @10:03PM
And that's why timed sports analogies don't work for machine learning.
This is a competition for accuracy. The teams competing are generating incredibly complex models to attempt to understand a test set of data. They do not have this test set, they only have a public training set. The only way to know how well their current parameters work is by having them evaluated against the test set. If you allow them to do this a bunch of times, it is possible to empirically "train" their model against the hidden test set.
That's cheating.
If you absolutely must have a sports analogy, this is like if everyone else got two time trials at the Indy 500 track while your team gets the track to yourselves for half a year, able to run as many trials as you like in as favorable conditions as you like and choose the best. Sound fair?
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @01:41AM
First: If you allow them to do this a bunch of times, it is possible to empirically "train" their model against the hidden test set.
I was under the impression that they uploaded their code, it was run, and they saw some results. There's no way that 200 runs should reveal enough information to train for that set. Pass/Fail for each test gives you n bits per run, and they need to extract 10^n bits.
Second:
1: - It's not a matter of it being fair, I made no such claims. I said that the result is real.
2: - Nope, they just get the final run.
If the goal is to produce good runners then certainly they ought to train for a half year on the course.
Sports analogies don't work because they're completely different. Sports is a demonstration of virtuosity, and entertainment. Competitions like this are 1) to drive up research interest and 2) to publicise results. The goals are completely different.
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @01:57AM
I'm tired and just reread that, 10^n is clearly wrong but I can't be bothered fixing it.