Using Amazon’s Mechanical Turk to cheat Digg
Ever since I decided to host DOWNFLY on Amazon’s Elastic Compute Cloud, I’ve been drinking the Amazon kool-aid.
Everything they are doing is revolutionary and also helps out start-ups like me. Their simple storage makes data redundant not just between machines, but between data centers before it gives a ‘successfully uploaded’ message (drool). And their EC2 allows me to boot up virtual machines on demand.
So I naturally thought the Mechanical Turk, a service by Amazon that allows you to outline a task and then have people perform it (Artificial Artificial Intelligence), was equally amazing and revolutionary.
Imagine all the great things you can do with it. You can have people tag images, classify articles, and create heaps of rich meta data. You can do artistic things like the 10,000 sheep market. But will this turn into another invention that was created out of good intention only to be misused by people?
It doesn’t take too much brainstorming to figure out devious things to do with the Mechanical Turk. I can start a request to have 1,000 people sign up for Digg (or use their Digg account) and digg this article (which I assure you I did not). I can say I’ll pay them each 10 cents for their efforts. I now paid $100 for a front page Digg link. Basically, I can start cheating and gaming systems that computers find difficult to game, but people find easy.
What about finding a vendor that pays you 50 cents a survey and you pay out on the turk 10 cents to take it? This one is already a reality.
How about this one? I’m a restaurant owner in New York City. I know people read reviews off of citysearch.com and menupages.com and make decisions on where to eat based off of them. So I post a job for 100 people to write convincing reviews and all give high ratings.
Sure you could do that yourself – but they’d all come from the same IP address and that can be detected. These types of AAI attacks would be extremely difficult to detect.
4 Comments, Comment or Ping
Ryan Mahoski
Very insightful, Billy. You appear to be someone who thinks a couple of steps ahead. I do think your Turk ideas introduce a couple of serious problems, however.
I suspect Amazon would quickly learn of your scheme (all three scenarios violate AWS policy) and probably close your AWS account. Your hypotheticals are clever and may well expose a loophole but I doubt Amazon would risk corporate goodwill to requester whim so carelessly. It seems to me that Amazon has probably already considered how their service might be used for evil. Any wealth Amazon gained ($100 * 10% = $10, in the case of the digg sham) would pale in comparison to the P.R. trouble your proposed HIT group would invite. No doubt, Amazon has a few mechanisms in place to abort deviant behavior.
Consider the HIT browser interface, where workers have two primary choices: Accept HIT or Report this HIT as Inappropriate (Why?). The latter button is much larger in the hopes good samaritans will rat out the requesters who appear to be playing dirty. The parenthetical (Why?) is a popup link:
—————————————————-
When should I report a HIT as inappropriate?
If you see a HIT that may violate the Amazon Mechanical Turk Participation Agreement, or otherwise goes against the spirit of Amazon Mechanical Turk, you can report the HIT as inappropriate. Reported HITs will be reviewed by Amazon Mechanical Turk staff and considered for removal from the web site.
For example, a HIT that solicits users to sign up for accounts or offers on other web sites is inappropriate because it violates the Participation Agreement.
If you have any questions about what is considered appropriate material, or if you have any questions about the Participation Agreement, feel free to contact us. Thank you!
—————————————————-
In this way, workers help police the space and it’s probable Amazon has set up additional triggers in order to nab scoundrels. I think that requesters who might, despite the above, still be considering the gambit would be wise to heed the warnings on the official CreateHIT web form:
—————————————————-
You may not ask Workers to:
1) Provide personally identifiable information
2) Violate the terms of service of another site
3) Click on affiliate links or ads
These are violations of the Mechanical Turk Participation Agreement and can result in the termination of your account.
—————————————————-
I think your proposed cheats risk not just a requester’s Mechanical Turk privileges but in fact one’s entire AWS account not to mention reputation/karma. You may well be onto a shrewd hack but my advice is: drop the racket ideas and focus your energy on the extraordinary legitimate possibilities of Mechanical Turk. I think the naysayers’ arguments are weak; this tool should have been going supernova by now and instead developers are ignoring it almost entirely.
Still want an angle? With AWS, but especially with Mechanical Turk, developers ought to be paying better attention to security.
Imagine if someone stole another person’s AWS identity by, say, hacking into the requester’s web server. Assuming the requester hadn’t taken additional server-side security measures (and it’s safe to say most do not), the hacker would find both access keys in plain view, unencrypted. The successful intruder can now masquerade in Tor as the requester. This is a serious problem. A GetAccountBalance call identifies his available resources for whatever HITs his conscience and creativity allow him to create. If the scheme doesn’t work (e.g., workers report questionable HITs), the hacker casually moves on leaving the real requester to deal with the authorities.
Another AWS security tip: if you enable global write access on an S3 bucket, obfuscate the bucket so only your application knows its name. Otherwise, pranksters or business competitors might think you’re soliciting terabytes of random gifts.
……….
Your eye for the aesthetic and imagination for color is remarkable. Self Portrait and Invasion were especially haunting. May I suggest a donation link?
May 20th, 2007
billy
Hi Ryan,
Thanks for your comment.
The problem is that a “Good Samaritan” may look the other way when they are being paid to do something.
Furthermore, as the Turk gets more popular, Amazon staff moderating will get tougher and tougher (lots of jobs to sift through)
And finally, if the Turk is big enough and the job is small enough, I can probably get it done under the radar and before they cancel my account. You can’t undo what’s already been done (and accounts are disposable commodities)
The problem is that spammers and con-artists will always find a way around our methods. It’s just human ingenuity (especially when it comes to making a buck)
I do agree with you though, that at least Amazon has installed some amount of precautionary measures.
Thanks for also taking a look at my artwork. I got a nice sculpture in the works — coming out soon
May 21st, 2007
Ryan Mahoski
Great response. One corollary: As relationships between requesters and workers mature, requesters may come to understand the ethical boundaries of individual workers. It is conceivable that a requester could use this information to qualify the unethical workers for rogue HITs, thus circumventing Turk’s dragnet.
May 21st, 2007
james
http://www.AWSurveys.com/HomeMain.cfm?RefID=hpacura
very easy!!
Jan 18th, 2008
Reply to “Using Amazon’s Mechanical Turk to cheat Digg”