Chinese search giant Baidu has been disqualified from an artificial intelligence test in which it appeared to beat Google and Microsoft. The researchers from Baidu were barred after organizers discovered that the Baidu scientists broke the contest’s rules. The test, known as the “Large Scale Visual Recognition Challenge,” looked at how well computers recognise images. […]
Chinese search giant Baidu has been disqualified from an artificial intelligence test in which it appeared to beat Google and Microsoft.
The researchers from Baidu were barred after organizers discovered that the Baidu scientists broke the contest’s rules.
The test, known as the “Large Scale Visual Recognition Challenge,” looked at how well computers recognise images.
The event is organised annually by computer scientists at Stanford University, the University of North Carolina at Chapel Hill and the University of Michigan.
It requires that computer systems created by the teams classify the objects in a set of digital images into 1,000 different categories.
The rules of the contest permit each team to run test versions of their programs twice weekly ahead of a final submission as they train their programs to “learn” what they are seeing.
Baidu claimed to have scored an impressive 4.58% error rate on a test which asked contestants to accurately recognise a series of images from a large picture database.
Although Google and Microsoft were not far behind – with 4.94% and 4.8% error rates respectively – it was still regarded as a significant victory especially as Baidu is a relative newcomer to AI.
However, on Tuesday, the contest organizers posted a public statement noting that between November and May 30, different accounts had been used by the Baidu team to submit more than 200 times to the contest server, “far exceeding the specified limit of two submissions per week.”
By submitting to the test sever more frequently than allowed the team could have chosen similar solutions based on test performance for an advantage or chosen methods to research and develop based on test data rather than relying on training and validation of data.
The firm has apologised for “this mistake”. Its win has now been cancelled and it has been banned from similar challenges for a year.
It included some periods of very high usage – 40 submissions in five days during March.
In a statement Baidu said that it had been informed that “we exceeded the allowable number of weekly submissions to the ImageNet servers”.
“We apologise for this mistake and are continuing to review the results,” it added.
Read the statement here
