Chrome caches in Incognito mode. It just doesn't let the cache leak out of the session. So this might explain the first tweet you load being slow, but if you load a second it should be just as fast as in normal mode.
Yeah, that is exactly the pattern. I don't use twitter, and neither do most of my friends, but occasionally I'll get linked to it.
I just chalked up the painfully slow load times to "well, it's twitter..."
(And sorry, twitter, it sounds like this has changed, but my experience with twitter involved a TON of fail whales, which is partially why I never got hooked on it, I'm sure)