​​A solution to the single-question crowd wisdom problem
​​D. Prelec, H. Sebastian Seung, and J. McCoy, Nature (2017)
​​presented by K. Sasahara

​​Democratic methods of opinion aggregation

  • ​​Majority vote
  • ​​Weighted by confidence
  • ​​Max confidence

​​Known issues

  • ​​Weighting people equally
  • ​​Weighting by confidence
  • ​​These methods are biased for lowest common denominator information, at the expense of novel or specialized knowledge that is not widely shared.

​​US state capital question (see Fig. 1)

​​Question (P)

  • ​​Is Philadelphia the capital of Pennsylvania?
  • ​​No (i.e., Harrisburg)
  • ​​Respondents voting yes believe that almost every one will agree with them, whereas respondents voting no expect to be in the minority.
  • ​​Shared belief: Philadelphia is a big city. 

​​Question (C)

  • ​​Is Columbia the capital of South Carolina?
  • ​​Yes

​​A Solution

  • ​​To utilize the knowledge of a well-informed subgroup in the cloud.

​​Surprisingly popular algorithm (see Fig. 2)

  • ​​For a given question, people are asked two things:
  • ​​What they think the right answer is (brief)
  • ​​What they think popular opinion will be (reasoning)
  • ​​Select the answer that is more popular than people predict
  • ​​The biased coin model
  • ​​Two world and a biased coin
  • ​​Majority opinion favors yes in both worlds
  • ​​People know the coins biases but don’t know which world is actual.
  • ​​If predicted yes vote > actual yes vote, no is surprisingly popular.
  • ​​In (P), actual vote 2/3 (0.67), prediction 5/7~6/7 (0.71~0.86); yes is less popular than predicted
  • ​​In (Q), actual vote 2/3 (0.67), prediction 1/4~1/2 (0.25~0.5); yes is more popular than predicted
  • ​​Identical distributions of confidence can arise in different settings.

​​Experiments (see Fig. 4, 5)

  • ​​SP is the best performer across all studies
  • ​​Study (2) by MTerk
  • ​​In the art domain, the majority opinion is too conservative.
  • ​​The majority voting is strongly biased against the high category
  • ​​People have a belief that expensive pieces (> 30K) are rare.

​​Summary

  • ​​Not only people’s actual beliefs but also their reasoning about the beliefs can be exploited to recover truth even when traditional voting methods fail.
​​