You are on page 1of 12

Can crawlers find all your important content?

• Sometimes a search engine will be able to find parts of your site by


crawling, but other pages or sections might be obscured for one
reason or another. It's important to make sure that search engines
are able to discover all the content you want indexed, and not just
your homepage.

• Can the bot crawl through your website, and not just to it?
Can crawlers find all your important content?
• Is your content hidden behind login forms?
If you require users to log in, fill out forms, or answer surveys before
accessing certain content, search engines won't see those protected
pages. A crawler is definitely not going to log in.

• Are you relying on search forms?


Robots cannot use search forms. Some individuals believe that if they
place a search box on their site, search engines will be able to find
everything that their visitors search for.
Can crawlers find all your important content?
• Is text hidden within non-text content?
Non-text media forms (images, video, GIFs, etc.) should not be used to display text
that you wish to be indexed. While search engines are getting better at recognizing
images, there's no guarantee they will be able to read and understand it just yet. It's
always best to add text within the <HTML> markup of your webpage.

• Can search engines follow your site navigation?


Just as a crawler needs to discover your site via links from other sites, it needs a path of
links on your own site to guide it from page to page. If you’ve got a page you want
search engines to find but it isn’t linked to from any other pages, it’s as good as invisible.
Many sites make the critical mistake of structuring their navigation in ways that are
inaccessible to search engines, hindering their ability to get listed in search results.
Can crawlers find all your important content?
Can crawlers find all your important content?
Common navigation mistakes that can keep crawlers from seeing all of your site:
• Having a mobile navigation that shows different results than your desktop navigation.
• Any type of navigation where the menu items are not in the HTML, such as
JavaScript-enabled navigations.
• Personalization, or showing unique navigation to a specific type of visitor versus
others, could appear to be cloaking to a search engine crawler
• Forgetting to link to a primary page on your website through your navigation —
remember, links are the paths crawlers follow to new pages!

This is why it's essential that your website has a clear navigation and helpful URL folder
structures.
Providing answers
• Search engines typically assume that the more popular a site, page, or
document, the more valuable the information it contains must be.
This assumption has proven fairly successful in terms of user
satisfaction with search results.
Providing answers
• How do search engines determine relevance and popularity?
• Popularity and relevance aren’t determined manually. Instead, the
engines employ mathematical equations (algorithms) to sort the
wheat from the chaff (relevance), and then to rank the wheat in
order of quality (popularity).
• These algorithms often comprise hundreds of variables. In the search
marketing field, we refer to them as “ranking factors.”
• Example: : https://moz.com/search-ranking-factors
Providing answers

You might also like