There are several questions that need to be answered before anyone can offer any really useful advice.
1. What type of backlink building have you been doing? If most of your link building is through the tens of thousands of crappy directories, the links could have been discounted or the site could have been filtered out by Penguin.
2. Is all of the content on the site absolutely unique? Is there any spun content? Does it all pass the Copyscape test? Panda is still snagging sites that it missed earlier.
3. Have you done a backlink audit to see if someone is engaging in a negative SEO attack against the site? Negative SEO does work and Penguin enabled it to work better. If you see lots of backlinks from unrelated sites, a competitor could be engaging in an attack.
4. Have any of the sites that you link to seen their home page PageRank drop to n/a? Your site inherits penalties applied to the site that you link to, so if there is any question about the quality of a site, use the nofollow attribute in your links.
5. Have other proxy lists seen similar problems? Google periodically goes after specific industries. They have decimated the rankings for most directories, affiliate marketing sites, blogs with crappy content, article marketing sites, etc.
"It's inexcusable for scientists to torture animals; let them make their experiments on journalists and politicians." -Henrik Ibsen