I can verify that large numbers of irrelevant backlinks can sabotage a site's rankings. One of my sites has been getting hit almost daily for the past year from links in postings from a large network of blogs. Each of the contextual links has irrelevant text in the hyperlink. My site does have original content and was not hit by Panda, but over the past year the rankings declined and the traffic dropped off to a trickle. I have not had a chance to deal with this issue.
I have another site that is under assault with hundreds of links from unrelated sites in Russia. Its rankings and traffic were also dropping. I'm battling this one by setting up lots of relevant backlinks. Thus far that is working.
Anyone who believes that "a link is a link and every link has value" is not keeping up to speed with current SEO issues. The quality of the links does matter. Keep this in mind when you are building backlinks.
"Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin