SoylentNews is people
SoylentNews
Huawei caught cheating performance test for new phones
UL, the company behind the tablet and phone performance benchmark app 3DMark, has delisted new Huawei phones from its "Best Smartphone" leaderboard after AnandTech discovered the phone maker was boosting its performance to ace the app's test. The phones delisted were the P20, P20 Pro, Nova 3 and the Honor Play. "After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings," the company said in a statement.
For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.
In other words, the phones aren't so smart after all.
Now I feel better about forgetting to submit this story.
Also at The Verge and Android Police.
(Score: 2) by Booga1 on Friday September 07 2018, @04:36PM (5 children)
You're barely scratching the surface of what Huawei tried to pull off. This line stands out to me:
So, when their cheating was caught, they tried to blame the "AI" feature of the phone. If that sticks we'll be seeing all sorts of stuff handwaved away as "it must have been the AI!"
(Score: 2) by DannyB on Friday September 07 2018, @05:29PM
"it must have been the AI"
Do you have any idea how many different kinds of marketing departments could use that line!
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 4, Interesting) by requerdanos on Friday September 07 2018, @05:43PM (2 children)
Well, the "AI" feature of an Android device that controls how fast the processor runs in response to a load is merely the CPU governor [androidforums.com] baked into any Linux kernel. Hardly a Huwaei exclusive.
On my workstation, I have a process running that looks every second or two at what processes are running and at the CPU temperature. If it sees a high temperature, it saves the current governor and changes to the "powersave" governor (runs at minimum speed regardless of load) until the temp comes down; if it sees any one of a list of processes (dpkg,apt,apt-get,aptitude,gcc,ld,ghb,gzip,gunzip,etc.) it saves the current governor and changes to the "performance" governor (run at max clock speed regardless of load) until the process(es) disappear(s).
Sounds like Huwaei is up to similar shenanigans.
(Score: 1, Informative) by Anonymous Coward on Friday September 07 2018, @08:14PM (1 child)
No, they just hardcoded to detect the name of the benchmark process. When they put in the same benchmark with a different name, they noticed the processor behaved differently. It was simply a case of engineering and coding to pass a test but not for real use.
(Score: 3, Touché) by aristarchus on Friday September 07 2018, @10:45PM
So, was it a diesel processor?
(Score: 2) by takyon on Friday September 07 2018, @05:51PM
AMD and Intel processors adjust frequency in real time in response to application demand, temperature, etc. AMD calls their implementation SenseMI [amd.com].
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]