Most of the time that is likely due to bundler/rubygems stuffing your load path full and causing thousands of unnecessary stat calls.
The actual time spent loading/parsing files is in most cases a tiny fraction of the startup time of any large project using rubygems and bundler.
I counted several hundred thousand unnecessary stat calls on the biggest app I have, and ended up with a ugly hack where we trimmed the load path around each set of require's to only paths needed by that specific gem.