What is over optimization?

Over Optimization happens when your website is considered "too good" by Google - either in terms of a sudden volume of backlinks, or because of heavy on-page optimization. In other words, if Google considers that your website optimization is beyond acceptable limits , your website will be red-flagged and automatically restricted or penalized.
There is a fine line between over optimization and spamming, and it is on this line that Google can appear to err. However, this is not a mistake by the search engine - in fact, Google calculates rankings by considering thousands and thousands of different factors - and a lot of importance is attached to average "trends" within the niche / keyword range that a website is optimizing for.
The bottom line is that over optimization is non-spamming search engine optimization that is misread by Google as being beyond acceptable limits, thus leading to a penalty in search engine rankings.

What criteria does Google use?

To understand why Google can consider certain websites over-optimized, it is important to factor in the criteria that Google uses to rank websites.
When fully indexing a website, Google does not just look at the optimization of the target website; it also compares the website with all the other websites that belong to the same niche / category / keyword range. Through this comparison, Google can then figure out the following:

• Is this website "way more" optimized than the current top ranking websites?
• In the past, have over-optimized websites been discovered as spam websites?
• What are the trends / acceptable limits for well-optimized websites in this niche/keyword range?

Since Google is automated, it cannot do what we do - look at the webpage and determine if the purpose is spam or delivering truly useful information. Instead, the search engine uses historical trends to predict what the acceptable limits of over-optimization are, and how likely over-optimized websites are to be found out as spam