Comments on: Reprioritizing human intelligence tasks for low latency and high throughput on Mechanical Turk http://www.deaneckles.com/blog/19_reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk/?utm_source=rss&utm_medium=rss&utm_campaign=reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk Dean Eckles blogs on people and technology Fri, 11 Dec 2009 23:55:08 -0700 http://wordpress.org/?v=2.8.6 hourly 1 By: Brendan O'Connor http://www.deaneckles.com/blog/19_reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk/comment-page-1/#comment-162 Brendan O'Connor Sat, 26 Jul 2008 07:20:29 +0000 http://www.deaneckles.com/blog/19_reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk/#comment-162 Yeah, there's definitely room to improve the decisionmaking of which questions to present when. Lots of the experiments on our blog use straight up MTurk with static EQ's, but we're moving to having all question htmls dynamically generated, so we can follow a strict priority ordering of questions, or weighted random sample them, whatever. The worst is if you have a batch of HITs and they start stalling out at 50% done, sometimes if you kill the job then resubmit those 50%, with the exact same payments and everything, suddenly you can get much higher throughput. I think what's happening is the HIT group gets buried in the Turkers' HIT browsing interface because it's not new. Totally lame. Yeah, there’s definitely room to improve the decisionmaking of which questions to present when. Lots of the experiments on our blog use straight up MTurk with static EQ’s, but we’re moving to having all question htmls dynamically generated, so we can follow a strict priority ordering of questions, or weighted random sample them, whatever.

The worst is if you have a batch of HITs and they start stalling out at 50% done, sometimes if you kill the job then resubmit those 50%, with the exact same payments and everything, suddenly you can get much higher throughput. I think what’s happening is the HIT group gets buried in the Turkers’ HIT browsing interface because it’s not new. Totally lame.

]]> By: Mike Krieger http://www.deaneckles.com/blog/19_reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk/comment-page-1/#comment-160 Mike Krieger Fri, 25 Jul 2008 17:46:51 +0000 http://www.deaneckles.com/blog/19_reprioritizing-human-intelligence-tasks-for-low-latency-and-high-throughput-on-mechanical-turk/#comment-160 Hi Dean, Great post! I think you've really nailed down the issues that affect throughput on mTurk. I was just finishing up a similar post when I read yours this morning, so I rewrote it to include your findings and elaborate on one additional one (task visibility) that I've come across as a throughput factor. Let me know what you think, Mike Hi Dean,

Great post! I think you’ve really nailed down the issues that affect throughput on mTurk. I was just finishing up a similar post when I read yours this morning, so I rewrote it to include your findings and elaborate on one additional one (task visibility) that I’ve come across as a throughput factor. Let me know what you think,

Mike

]]>