Actually I found out that there was an error with my server setup. The reason Jetpack seemed to be the cause is because it causes more disk IO and server load than any other plugin.
I still want to disable everything but the stats, but the interface is pretty crappy and I'm not sure if that's even possible.
Unfortunate, because I really like the wordpress.com stats engine. Too bad they tacked it to Jetpack.