The People Behind Google's Search Results

Think Google’s algorithm changes are determined only by a mathematical equation run by a gigantic computer?

Think again!

Since at least 2004, Google has utilized the services of third party companies to hire search quality evaluators- actual humans who help rate the quality of results and test potential algorithm configurations to see if they work as planned to improve results. I recently learned about Search Quality Evaluators from a Search Engine Land interview with one.

Evaluators are asked to test algorithms or potential algorithm changes and rate results for their relevance and usefulness. Search quality evaluators rate organic results, paid results, news results, maps, and other searches. They judge results based on given tasks and their ability to fulfill a searcher’s intent.

The Google Blog said, “We sometimes show evaluators whole result sets by themselves or side by side with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.”

Quality evaluators are tasked with the job of rating how well Google is doing at matching searches with the best websites and information sources. This includes side-by-side tasks, which compare 2 potential sets of results, perhaps one with the existing results and one that displays results as a new algorithm would.

Other tasks involve rating a query and a URL, and asking them to rate the URL compared to the query and decide if the URL provided would be relevant to what the searcher wants. The possible ratings are Useful, Relevant, Slightly Relevant or Off-topic. Additionally, raters are able to leave notes about the site or mark it as spam. Read more about the definition of each of the possible ratings evaluators use http://searchengineland.com/the-google-quality-raters-handbook-13575. Note that quality raters are separate from Google’s spam team, which are employed by Google, and have their own set of guidelines.

How Does One Become a Quality Rater?

Well, it sounds like it’s not easy. The quality of search results are paramount to the usefulness of a search engine, so Google trusts companies to hire people whose judgment aligns with their own in rating results. According to the interview with a rater, there are 2 difficult tests one must pass before being trusted as an evaluator.

Once you pass an initial test, candidates must take a practical exam which involves rating actual queries, and answering theory questions. To be hired, one must pass this exam with 90% accuracy, meaning the ratings they choose must match desired ratings.  Candidates for this position do not necessarily need internet marketing experience, and it is best if they are “average users”.

Evaluators work part time and have a set number of hours they must maintain, and have high expectations about the number of evaluations done in an hour. The fast rate they are expected to meet when rating sites with queries could mean they don’t even visit a site, but judge it based on meta data and the instant preview of a website.

Do Evaluations Directly Affect My Site’s Rankings?

The short answer is no. The evaluations are judging the algorithm, not the specific rankings. The long answer is yes- if a site is perceived as not relevant, then the algorithm which displays the site is decided to be unfavorable. One evaluator’s rating of a site’s relevance to a query does not directly affect your rankings. Each task is rated by more than one evaluator, and if 2 disagree about the result, a discussion is held and moderated to reach a decision. If many evaluators agree that a site is irrelevant or spammy, feedback is used to assess the algorithm that provides better results, thus potentially bumping down your site in the SERPs.

Takeaways

• Evaluators can rate your URL without even looking at it.
• Meta description and tags should be as accurate as possible to describe what’s on your site. Sometimes they will actually view the page and rate quality, other times they might just look at the results and the query. They may just be judging your site on your URL and meta data, so be sure it’s accurate and free from errors.
• First impressions really count.Search evaluators will likely make a snap judgment about your site based on its appearance.  Does the website’s design look credible and up to date? Is it packed with ads above the fold, and impossible to find content? If a search rater quickly visits a site and notices that it looks spammy and outdated, it will likely be rated that less favorably than a site with a quality design and layout.