Crawling and website audits
If you work on or manage a website there is a useful procedure you can carry out to audit your site, especially with regards to how well the website might perform in search engines. It is called ‘Crawling’, an established technique used in Search Engine Optimisation (SEO).
Website crawling is the process where software, usually called a ‘spider’, ‘bot’ or ‘crawler’, travels through a website, scanning and collecting information. These ‘crawlers’ follow links and discover pages, images, content and metadata etc. Major search engines, like Google and Bing, will constantly be crawling the web to keep their indexes up-to-date, looking for new websites and changes to existing sites, as well as for signals that help decide if a webpage should be returned in their search results.
SEO site audits
Crawling can also be used to carry out site audits. There are several SEO website services, usually subscription based, that have tools for carrying out site audits. For example, Moz.com and Ahrefs.com both have user friendly services that can be easily setup. Alternatively, specialist software can be installed on your computer and used to carry out a site audit. This approach usually requires more technical skills but can provide a greater level of detail than the website-based audits. Two examples would be ScreamingFrog and Visual SEO Studio.
Carrying out this type of audit can reveal a number of issues. For example, which links are broken, and which pages are slow to load. It can highlight if the site has duplicates of the same content (bad for SEO) or has page titles and meta descriptions that are too long. More technical issues may be discovered, around the use of meta tags that may be sending the wrong signals to search engines or the use of the wrong type of redirect, which can adversely affect a web page’s ranking in search engines.
If you want to crawl your website but have limited knowledge of the technical side or SEO, then a user-friendly web-based service would be a better approach. For those who are happy getting into the technical details and have a good understanding of SEO, try installing your own crawler software. Many of them are free to use for smaller websites.
With a larger site, if you have concerns about how well the site ranks in search engines, then it may be better to enlist the help of an expert to carry out a detailed site crawl. Fixing the types of issues found with this type of site audit is usually one of the most cost-effective ways of making your website get more visitors through search.