turn off any modules in jetpack settings that you dont need running. Very few sites would find a use for all the modules. Even though jetpack is designed to make use of wordpress.com features, it does make alot of http calls. Also adds a lot of unnecessary mysql queries to the database.
So go to jetpack > click the "Learn More" button on each module you dont need running > deactivate.
When running smoothly with your site, modules like photon should actually increase your site performance substantially.