It's actually not that hard -- invite them to work with you on a small project. Pay them reasonably. See how they work in RL. Then decide, and -- by all means -- give them honest feedback even when not hiring.
We did that one time. On the one had it worked very well. The person wasn't a good fit on the team so we declined. On the other hand it was emotionally draining because the guy was a really nice guy. It's really, really hard to work with someone, get to know them a bit and then at the end say, "Sorry, you're just not for us". I think it was even harder for the candidate who didn't clue in that they weren't doing very well. They met the team, got to know people a bit, got to feel the atmosphere and started to imagine what it would be like to work with us.
In the end it was such an upsetting experience for everybody that we agreed never to do it again. I think there are probably ways to make it work, but you need to be really careful to maintain some distance... But then, will you get the result you are looking for if you do?
In Germany this is called "Probearbeiten" and is fairly common for software or design jobs: when you have passed the (mostly theoretical) interview, you're invited to work a day or up to one week to evaluate your practical skills.
Generally you tend to be given a few tasks that are representative for the work you will be doing if you get hired. This is similar to spec work (e.g. the result will typically be discarded unless it is exceptionally good and solves real-world problems) but typically you will be reimbursed if you are not hired.
For a programming job you may be given a task that doesn't require intimate knowledge of any of the company's codebases but should give some insights into how you work, followed by a short review.
It's important to note that this doesn't scale well. It works best if the candidate works on site and can work alongside future team mates, which may impact the team's productivity for the duration. This approach works best for small to medium scale companies with a small pool of viable candidates. It's beneficial for the company to only send a candidate through this process if they're very likely to hire them.
I actually prefer this approach. By the time you're invited for "Probearbeiten" both sides are fairly confident you're going to be hired and it gives both sides a chance to determine whether it's a good fit.
It should also be noted that even in companies that don't do "Probearbeiten" there's a trial period ("Probezeit") after you're hired of up to six months where you're pretty much employed "at will" and can be fired on the spot.
I don't think that was the question, though. Small firms can afford to do this, but how does this scale to companies with who want to interview a few hundred candidates this month? Where do the hundreds of real-world projects come from?
This sounds like a huge time-sink on both sides. The interviewee will need to clear up a lot of time to work on the small project (which is not always possible).
This does not scale - the number of people that can be interviewed per unit time will go down drastically.
This wastes interviewers time, they could be spending the time working on real work (unless the project is real work).
If the project is a real work for hiring organisation, it likely would reveal to outsider things the company probably wants to keep secret (infrastructure, technology, processes, etc).
