There is no doubt that the anchor text of links is one of the major metrics used by Google, but it is far from being the only one. Search engines have the ability to look much more deeply into the context surrounding a page. Other signals regarding the page relevance are taken into consideration.
Are the links on the page located closest to your own high quality and niche related sites? If so, this can be taken as a positive signal. Your site may gain trust by association to other quality related sites. This is why you may see links to such site as Wikipedia in articles that have obviously been created to boost search engine rankings. On the flip side of things, if the closest links to your own are for other niches such as , Viagra, or adult material, this may appear as a low quality link. It looks like the link is coming from a link farm, and that is a bad signal.
Where on the page does your link appear? Is your link located in the body of the page content, in a side bar, or buried at the bottom of the page? Better page placement can indicate higher quality links. This is referred to as “link prominence” and also can be applied to on page SEO in terms of keyword placement.
Does your link’s anchor text fit within the context of the writing surrounding it? Does that content logically relate to the content of the page your link is pointed at? This is another positive signal that your page is of high quality. This is often referred to as “proximity”.
Modern search engines are highly intelligent and look at more than just the raw textual content. They can actually look deeper into the structure of a page and the section on which your link is located. In doing so they can determine the section’s topic by way of header tags, or the nearest bold highlighted text, which often act as titles or headers to the nearby content. As you can imagine, a close relation between your page, and these headers, are another positive signal.
So the nearby content, links, and headers are all related to your page’s content. This is excellent, but to take it a step further, it’s even better if the overall topic of the page your inbound link is located on is closely related to your own page’s topic.
You should be able to see where I’m going with things at this point. You want relevant inbound links, and if the general topic covered by the entire site that is linking you is of similar nature to your own, this is another very positive signal. It tells the search engines this site is dedicated to the topic, and in linking to your related page, they consider your content of value.