Honestly, I believe search engines only want to provide relevant results to its users. Google uses its PR system to account for a sites reputation. Reputation, which can also be said as credibility depends on the search engines algorithm. Google's algorithm is quite complex because it has about 200 factors. When retrieving results from its index, search engines will most likely consider the quality of the links pointing to and from a particular site. It will also try to match "keywords". I guess that's when relevance comes in. Google does not use the keywords meta tags, but other search engines might.
Say you have a site about "baby products" and a reputable site on "mortgage/finance" is linking to you. You clearly do not have the same keywords as the mortgage site, therefore it isn't relevant to your site.
I believe google uses the description and content from a site. I believe it is important to only link to relevant sites because, if a user comes to your baby product site and finds links to mortgage sites, they're going to be like WTF? lol..So, it helps in terms of rankings and real-time visitors.
PS: Matt Cutts recently stated that a website's speed is more important when compared to relevancy, in terms of ranking.