The Screaming Frog SEO Spider runs on your website and crawls website URLs to collect key elements to audit onsite technical SEO. Identify and evaluate an SEO crawl on your site using the Screaming Frog SEO Spider tool. The tool provides simple format reports for SEO evaluation of your site by locating ordinary issues such as duplicate content, broken elements on the page, etc. Check the response codes for your website by inspecting the Client Error (4XX) and Server Error (5XX) inside the response Code tab. It optimizes a website by analyzing on-site elements, such as page titles, meta descriptions, URL structure, images, etc. In the Directives tab, find information about the meta robots tag, canonical links, and rel=next/prev annotations. The SEO spider tool helps improve the performance of websites in search result pages. The tool finds broken links, generates XML sitemaps, crawls JavaScript websites, and a lot more.
Read more about Search Engine Optimization by clicking on SEO Fundamentals.
Screaming Frog Interface
The Screaming Frog Interface fig1 is made up of 1) Screaming Frog SEO Spider Menu 2) Crawl Bar 3) Tab Bar with Menu at top 4) Side Bar Scroll and menu at top 5) Main Window with Slider 6) Detailed Window with Menu at bottom 7) Graph
Crawl a Website
First, in the Filter drop-down select ‘HTML’ [refer fig 1]. Next, in the screaming frog SEO spider menu bar, go to ‘Mode’ and from the drop-down select ‘Spider’. Enter the URL in the crawl bar located at the top of the screen and click ‘Start’. All the crawled URLs get displayed in the Main Window. Click on any of the URL in the Main Window and go the Menu located at the bottom of the Detailed Window and click on ‘Inlinks’. All the details pertaining to that URL gets displayed in the Detailed Window. Find out which pages show a 404 error that needs to be addressed and these details can be exported in a CSV file.
To crawl a list of URLs go to Mode and select List. From the Upload drop-down located in the crawl bar, upload a list of URLs from a file or enter manually by pasting in the displayed box.
Screaming Frog Side Bar Tab
The Side Bar Scroll gives you a summary of URLs crawled. Details of all the SEO elements are available by scrolling down in the sidebar. Connect [fig2] to Google Analytics, Google Search Console, Majestic, Ahrefs & Mozscape for external link audits available only in the paid version.
Tabs: Overview> Site Structure> Response Times> API.
Under response time you can find how fast your pages are loading. The response times shown in the [fig3] is 0.25 seconds which is as per the standard norms.
Screaming Frog Tab Bar
Tab Bar: Internal> External> Protocol> Response Codes> URI> Page Titles> Meta Description> Meta Keywords> H1> H2> Images> Canonicals> Pagination> Directives> Hreflang> AJAX> AMP> Sitemaps> Custom> Analytics> Search Console> Link Metrics.
To be concise some of the main tabs, in the above is considered as per below.
Internal
Internal [fig4] refers to all that is on-page. Filter by All> HTML> JavaScript> CSS> Images> PDF>Flash > Others.
Scroll along the bar and check to optimize column wise.
Tabs: Address> Content> Status Code> Status [HTTP response code]> Indexability [URLs are indexable or not]> Indexability Status> Title 1> Title 1 Length> Title 1 Pixel Width [can be upto 50-60 characters or 512 pixels]> Meta Description 1> Meta Description 1 Length [under 158 characters]> Meta Description 1 Pixel Width> Meta Keyword 1> Meta Keywords 1 Length> H1-1> H1-1 length> H2-1> H2-1 length> Meta Robots 1> Meta Refresh 1> Canonical Link Element 1 (tells search engine which URL is the master copy]> rel=”next”1> rel=”prev”1> Size> Word Count [at least 350 words]> Text Ratio> Crawl Depth [upto 3 is acceptable but level 4 and 5 are considered bad]> Link Score> Inlinks [check each page lowest number/ highest number]> Unique Inlinks> % of Total> Outlinks [check each page lowest number/ highest number]> Unique Outlinks> Unique External Outlinks> Hash> Response time [0.25 second]> Last Modified> Redirect URI [301, 302 etc]> redirect Type> URL Encoded Address.
External
The external tab [fig5] gives information about external URIs.
Tabs: Address> Content> Status Code> Crawl Depth> Inlinks>
Screaming Frog Protocol
The protocol tab [fig6] gives data HTTP Vs HTTPS from internal and external URIs.
Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Canonical Link Element1> Meta Robots1> X-Robots-Tag 1.
Screaming Frog Response Codes
The response code tab [fig7] gives response information from internal and external URIs. To find redirects go to bulk exports and select response codes and click on redirection (3xx) in-links. The redirection_(3xx)_inlinks.csv file saved on your computer shows a table where the redirects are on that need to be changed to the actual redirects.
Tabs: Address> Status Code> Status> Indexability> Indexability Status> Inlinks> Response Times> Redirect URI> Redirect Type.
URI
The URI tab [fig8] includes data related to the URLs requested which is displayed in the main box.
Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Hash> Length> Canonical Link element 1> URL Encoded Address.
Page Titles
Go to page title tab. Export the report and get information about length and character/length and pixels of the title. Should the title be long, it will get truncated in Google search results. Check your titles to spot similarities so that you can merge such similar pages into one.
Tabs: Address> Occurrences> Title 1 Length> Title 1 Pixel Length> Title 1 Pixel Width> Indexability> Indexability Staus.
Meta Descriptions
Keep meta description between 120 to 158 characters. Google’s meta description length is up to 920 pixels.
Tabs: Address> Occurrences> Meta Description 1> Meta Description 1 Length> Meta Description 1 Pixel> Indexability> Indexability Status.
Meta Keywords
Ignore the Meta Keywords tag as Google doesn’t use the “keywords” meta tag in web search ranking.
H1
Each of your pages should have a single H1 heading. Check if anything is marked zero.
Tabs: Address> Occurrences> H1-1> H1-1 Length> Indexability> Indexability Status.
H2
Every page is not required to have an H2 heading. Check if any page that is not having an H2.
Tabs: Address> Occurrences> H2-1> H2-2> H2-2 length> Indexability> Indexability Status.
Images
The image tab (fig9) includes data pertaining to any images crawled. The Image has to be under 100 KB for it to load faster on web pages. Check for missing or long text alt tags.
Tabs: Address> Content> Size> IMG Inlinks> Indexability> Indexability Status.
Link Metrics
Tabs: Address> Status Code> Title 1.
Connect Screaming Frog with Ahrefs, Majestic, etc. These Link Profilers scan websites and databases to determine the authority of websites for these links from where they come. Backlinks are the Off-page SEO that is outside the scope of normal SEO Audits.
Detailed Window – Lower Tab
Detailed Window Box displays results for any single URL that is clicked in the Main Window Box for any selection of lower tab menu: URL Info > Inlinks> Outlinks> Image Info> Resources> SERP Snippet> Rendered Page> View Source.
The SERP Snippet tab [fig10] shows results for the URL: https://sapcanvas.com/yoast-seo/ providing information on the truncated ‘text’ for the Title and Description (if any) in Search Results. This can be rectified.
Screaming Frog Menu
Configuration
The default settings [fig11] are good enough.
Mode
The limit for List mode is up to 500 URLs for the free version.
Bulk Exports
The Bulk Exports tab allows the saving of any of the elements from the drop-down list [fig13] as a .csv file to your computer. For eg to obtain a list of 404 go to the menu – bulk exports and select response code and click on client error (4xx) and it will save this file for you.
Screaming Frog Reports
In the same way, all the reports refer [fig14] can be saved in a .csv file on your computer. For example, under reports select the redirect & canonical chains and the redirect_and_canonical_chains.csv file saved will show the link to a redirect and that redirect is redirecting to another page.
Screaming Frog Sitemaps
In the menu select site map fig15 and in the window that pops up leave the setting at default and click next. This will then allow you to save the sitemap.xml file wherever you want. Check this sitemap file errors for broken links, redirects, nasty URLs, etc. In a similar manner, you can save the images_sitemap.xml file and check errors for the missing alt text and image file sizes.
Visualizations
Graphical representation for each drop-down elements in the v
Crawl Analysis
Crawl Analysis fig17 configuration is only available for the paid version.
Other Options to Screaming Frog
- visual-seo.com
- wildshark.co.uk/spider-tool/
- beamusup.com
Conclusion
The Screaming frog SEO spider helps quickly audit your website. The tool finds the below issues so that you can address and fix them.
- Redirects 301, 302,
- Duplicate content,
- Broken links,
- Pages with thin content,
- Check- XML site maps, server response times, etc;
- Optimize- page titles, internal links, etc
- SERP snippets
The paid version allows unlimited crawling of URLs all for the price of £149.00 per Year. Besides scheduling it has options for crawl configuration, saved crawls & reloads, and AMP crawling & validation. Custom source code searches and extractions, with integrations for Google Analytics, Search Console, and Link metrics. It renders JavsScript, and store | view raw & rendered HTML Moreover set up custom robots.txt files with free technical support.