This is absolutely a problem we thought about, but we never found a good solution. Try getting the general public to use two-factor auth. Just try and see how that works out.