Technical SEO: Screaming Frog SEO Spider Website Crawler – Product Review
I was on Twitter a month ago and a digital marketer asked the audience “Which SEO tools do you find most valuable?”. The responses poured in. We heard familiar names like SEMRush, Ahrefs, SEO Surfer and Moz for on-page SEO. All of those are great tools. But almost universally for technical SEO responders praised Screaming Frog.
What in the heck is Screaming Frog? It is an automated app that will go through every page, image, and every other resources on your site to check for SEO problems.
What can Screaming Frog do?
- Find broken links, errors, and redirects.
- Audit redirects
- Analyze Page Titles, Headlines, Image sizes, Image tags, and Meta Data.
- Discover duplicate content.
- And a long list of other capabilities
You cannot rank highly in Google search results if your site has glaring technical problems.
This is a tremendously powerful tool for any digital marketer or web developer. If your site’s technical SEO is not properly setup, then the on-page SEO that you do has to struggle to get traction to get properly indexed by Google if those pages get indexed at all.
I ran this tool on some of my smaller websites and immediately I found broken links on my site that I simply missed due to human error. I also discovered that some of my external links were now linking to pages that no longer exist. Plus, I discovered some of my old links were still using HTTP rather than HTTPS.
Then I went deeper and examined the content on all of my pages. All of my pages had Page Titles, but some were simply too long and needed to be optimized. I discovered that some of my pages lacked H1 or H2 headlines, so I added them. Screaming Frog found them all.
I looked at my meta data and discovered that some pages did not have a short description (which shows in organic search results), or if it did it was too long. Some pages did not have meta descriptions at all. So, I went through all of my meta descriptions and optimized them for target keywords on every page.
I found images that were too large or lacked alt tags. So, I optimized those.
This tool is remarkable. It is simply not possible for a human to do this level of testing on their own for any reasonably sized website.
I have one client that has over 50,000 pages in their website due to the enormous size of their product catalog. Screaming Frog crawled their entire site and discovered numerous problems that needed to be corrected. It also discovered that many pages were not being indexed which is now leading to a deeper investigation of the site map and canonicals.
Screaming Frog offers a free version that will scan 500 URLs. The paid version is only £149.00 Per Year. That is about $211 per year as of May 18, 2021.
I highly recommend this product for any web developer or digital marketer.
For more information, please visit: https://www.screamingfrog.co.uk/