You can get gains to WordPress performance by modifying your .htaccess as well.
I don't know where you're hosted, but my sites have all been running under a 4 yr old GoDaddy deluxe hosting account. I had been having trouble w/ page load times (4-8 secs). My blogs had 3 simple plugins (Akismet, Contact Form 7, and Thesis Openhook) and less than 100 pages each with very little media. So it should have been screaming IMO.
After many calls to support over months and months, I was finally told that Godaddy used to put your site on a single server (i.e. no load balancer)... If others on that server were eating up CPU, I/O, or bandwidth then your site's performance suffered. Now they offer the same Deluxe account on their "grid computing" (load balanced) platform for the exact same price. I moved over to their grid hosting and my pages are all loading in 1 sec or so.
So it's not ALWAYS WP or the Theme. There are a lot of external factors that can affect your page load times... things like hosting, DNS lookups, etc.