An Anonymous Coward writes:
The Electronic Frontiers Foundation (EFF) have
published [eff.org] a breakdown of AI problems such as machine vision and game playing, with associated metrics and datapoints that compare against human-level performance. The code lives
here [github.com] and contributions are welcome. In addition to the usual suspects, they also have open problems in AI safety and security (e.g. resistance to adversarial examples) though with no associated metrics yet.
Original Submission