I’ve been away from this blog for a week because I had the opportunity to participate in the Steve Fossett search in Nevada. I’ve been a member of the Civil Air Patrol for 5 years, and this has been the largest search I’ve been part of. I was impressed by the professionalism and skill exhibited by the CAP members participating in the search, and it made me proud to be a member.
What prompted me to go off topic, besides making an excuse for being absent from the site for way too long, was something I noted while checking out Amazon’s “Mechanical Turk” (http://www.mturk.com), and the effort to locate Mr. Fossett. This effort lets anyone participate by looking at satellite photographs of sections of the search and saying whether or not they think the section they’re looking at contains something that warrants further investigation. Each section is shown to many people, and any image that is flagged consistently is reviewed by experts and possibly passed on to search teams.
I think the idea of pursuing the search with many eyes is fantastic, and I hope it turns up useful leads. However, I was surprised to see this activity compared with the techniques outlined in James Surowiecki’s “The Wisdom of Crowds.” According to Surowiecki, as quoted in Wikipedia (The Wisdom of Crowds), for a crowd to be wise, it needs to have the following characteristics:
- Diversity of opinion: Each person should have private information even if it’s just an eccentric interpretation of the known facts.
- Independence: People’s opinions aren’t determined by the opinions of those around them.
- Decentralization: People are able to specialize and draw on local knowledge.
- Aggregation: Some mechanism exists for turning private judgments into a collective decision.
The MTurk process is independent and diverse, but it doesn’t provide a way to turn a private judgment into a collective decision, and it isn’t really decentralized, at least in the sense that localized knowledge can be applied by individuals. It really is asking for people’s observations, not their wisdom. In fairness, I don’t see any claim by Amazon that this is the technique in use, but references pop up repeatedly in other sources.
I’m not saying that what Amazon is doing is flawed; on the contrary, it is valid and important. However, I think it could be supplemented by using the ideas in the Wisdom of Crowds.
Suppose everyone was given a description of the situation (i.e., basic facts like the capability of the plane, maps of the area, confirmed and possible sightings of the plane in flight, etc.), and asked to make his or her best independent estimate of where the plane will be found. Then, take that information and aggregate it to identify a potential search area. This would be similar to the search for a Navy submarine that is described in Surowiecki’s book. In that case, even though some guesses were way off, the aggregate was very close to the actual location.
I suspect the toughest thing about using this method would be to keep participants from being influenced by the potential hits that have been reported by people participating in Amazon’s visual effort. That said, such an activity might yield useful results that certainly wouldn’t be any worse than the leads coming from people who claim to be psychics.