Some people say it to mean that. But another perspective is that there can be racist structures which do not sit in any one individual. So you can be personally non-racist but somewhat trapped in such structures. If you go against the grain, you can risk life-and-limb (hey, Godwin!) or in a less extreme setting, socially ostracized.