What times we live in. Even neural networks cheat. Some have learnt to recognize a horse by spotting the tag www.pferdefotoarchiv.de, an archive of horse pictures; so if you paste this tag onto a Ferrari picture, the network will classify it as a horse. Some, when playing Atari Pinball game, have learnt to make high scores by repeatedly nudging the table, taking advantage of the high threshold of the tilting mechanism. Some have learnt to recognize airplanes just from the smooth blue landscape surrounding them in most pictures. These examples are among the results of a work carried out by a group of researchers from Fraunhofer Heinrich Hertz Institute and Technische Universität Berlin (Germany).
Researchers used the layer-wise relevance propagation (LRP) method with the help of a semiautomatic tool called SpRAy (spectral relevance analysis), thus producing “heatmaps” where colors represent how important a certain pixels was for the neural network to make its decision (ranging from green, low relevance, to red, high relevance). As researchers observe, “The above cases exemplify our point, that even though test set error may be very low (or game scores very high), the reason for it may be due to what humans would consider as cheating rather than valid problem-solving behavior. It may not correspond to true performance when the latter is measured in a real-world environment, or when other criteria (e.g. social norms which penalize such behavior) are incorporated into the evaluation metric.”
