Definitely! Also, there are a couple of wonderful followup projects that think about how to make crowdsourcing affordable (each task still costs money): http://www.cs.berkeley.edu/~kuangc/
This is very interesting. Proofreading as a service could be used by so many content companies out there.
I guess for something long like a book it would make more sense to have the same person proofread the entire thing, but for blog posts and other text, parallelism could be very powerful.
I'm not sure how the pricing will work out though:
$0.004/w = $1.4/page = $350 for my book
this is 1/6 of what I paid to my copy editor. Who will work for such a low wage?
Last but not least, does it all have to be through-the-web. I find I catch a lot more mistakes on paper.
Either way, I wish you good luck with your business and I will be following the developments closely.
Right, looking at crowdsourcing (the main proofist site) was born out of my frustration that most copyediting comments are for simple mistakes that anyone could have caught. I want to make ours cheap enough that you can get 10-20 pairs of eyes per piece of text and without the usual wait.
This game is an experiment to make it free, and if we get mass, can get the desired level of matching. (In just one hour, we're already a hit with educators in math/science!)
I've been working on my own mturk-based proofreading system for OCR scans, modeled after Ryan Tate's typingpool. This is very close to a web-based version.
Your illustrations suggest you chunk content to mturk at the paragraph level, is that correct? Did you try it at other levels of granularity, by the sentence, word, or N number of words? If so, what were the results? If not, why not?
How did you settle on 4¢/word?
Are you considering a mobile app/FB version of Proofville?
What's the back-end? Is it your own system, or is it an existing crowdsourcing infrastructure like PyBossa?
How many reviewers compare a passage? 1? 2? Best of N until there's consistency?
A housemate (linked below) runs http://captricity.com/ -- crowdsourcing OCR is a really good idea.
The collaborator for the main site used to work @ cap, and we've been evolving a REST API to simplify calling into mturk.
For quality, we don't use arbitrary mturkers. Getting good results on mturk takes a lot of experimentation: I now think about it is as a combination of HR agency, traffic conductor, and street vendor. The challenges of building a useful & sustainable app on it are a bit different from what's addressed by crowdsourcing research papers, though that's not so surprising in retrospect.
Sounds like this deserves a blogpost. It's been a fun summer project. Basically, nodejs with roll-our-own crowdsourcing algorithms, APIs, and UI widgets. A lot more to do, but it's finally good enough to address my proofreading needs, and I'd like to help others who also need to get some writing done asap :)
Btw, we're $0.004 / word, not $0.04 ;-) That means we already cost 60% less than shops with comparable quality and slower turnarounds.
Yes, please do! I can only begin imagine some of the issues, having fed a couple thousand minutes through typingpool, but there isn't a lot of writing about using it at scale. I'll look forward to it.
There is for the main site. I like the idea of different matchmaking modes for the game. A player suggested allowing more than two reviewers working together in a session, which may be one fun way to do it!
I absolutely love this. This is about way more than spell and grammar check. It is not just proofreading. Now the crowd can be a test audience to help make content more readable and digestible. I hope this works out!
But they can't deeply check style, clarity, or other such qualitative aspects. Word won't tell you when your sentence is difficult to read or confusing.
http://projects.csail.mit.edu/soylent/