Screaming Frog

Spread the love
Image-Screaming Frog
Image-Screaming Frog

The Screaming Frog SEO Spider runs on your website and crawls website URLs to collect key elements to audit onsite technical SEO. Identify and evaluate an SEO crawl on your site using the Screaming Frog SEO Spider tool. The tool provides simple format reports for SEO evaluation of your site by locating ordinary issues such as duplicate content, broken elements on the page, etc. Check the response codes for your website by inspecting the Client Error (4XX) and Server Error (5XX) inside the response Code tab. It optimizes a website by analyzing on-site elements, such as page titles, meta descriptions, URL structure, images, etc. In the Directives tab, find information about the meta robots tag, canonical links, and rel=next/prev annotations. The SEO spider tool helps improve the performance of websites in search result pages. The tool finds broken links, generates XML sitemaps, crawls JavaScript websites, and a lot more.

Read more about Search Engine Optimization by clicking on SEO Fundamentals.

Screaming Frog Interface

fig1-Screaming Frog Interface
fig1-Screaming Frog Interface

The Screaming Frog Interface fig1 is made up of 1) Screaming Frog SEO Spider Menu 2) Crawl Bar 3) Tab Bar with Menu at top 4) Side Bar Scroll and menu at top 5) Main Window with Slider 6) Detailed Window with Menu at bottom 7) Graph

Crawl a Website

First, in the Filter drop-down select ‘HTML’ [refer fig 1]. Next, in the screaming frog SEO spider menu bar, go to ‘Mode’ and from the drop-down select ‘Spider’. Enter the URL in the crawl bar located at the top of the screen and click ‘Start’. All the crawled URLs get displayed in the Main Window. Click on any of the URL in the Main Window and go the Menu located at the bottom of the Detailed Window and click on ‘Inlinks’. All the details pertaining to that URL gets displayed in the Detailed Window. Find out which pages show a 404 error that needs to be addressed and these details can be exported in a CSV file.

To crawl a list of URLs go to Mode and select List. From the Upload drop-down located in the crawl bar, upload a list of URLs from a file or enter manually by pasting in the displayed box.

fig2-Side Bar~API
fig2-Side Bar~API

Screaming Frog Side Bar Tab

The Side Bar Scroll gives you a summary of URLs crawled. Details of all the SEO elements are available by scrolling down in the sidebar. Connect [fig2] to Google Analytics, Google Search Console, Majestic, Ahrefs & Mozscape for external link audits available only in the paid version.

 fig3-ScreamingFrog-SideBar-ResponseTimes
fig3-ScreamingFrog-SideBar-ResponseTimes

Tabs: Overview> Site Structure> Response Times> API.

Under response time you can find how fast your pages are loading. The response times shown in the [fig3] is 0.25 seconds which is as per the standard norms.

Screaming Frog Tab Bar

Tab Bar: Internal> External> Protocol> Response Codes> URI> Page Titles> Meta Description> Meta Keywords> H1> H2> Images> Canonicals> Pagination> Directives> Hreflang> AJAX> AMP> Sitemaps> Custom> Analytics> Search Console> Link Metrics.

To be concise some of the main tabs, in the above is considered as per below.

fig4-Screaming Frog~Internal
fig4-Screaming Frog~Internal

Internal

Internal [fig4] refers to all that is on-page. Filter by All> HTML> JavaScript> CSS> Images> PDF>Flash > Others.

Scroll along the bar and check to optimize column wise.

Tabs: Address> Content> Status Code> Status [HTTP response code]> Indexability [URLs are indexable or not]> Indexability Status> Title 1> Title 1 Length> Title 1 Pixel Width [can be upto 50-60 characters or 512 pixels]> Meta Description 1> Meta Description 1 Length [under 158 characters]> Meta Description 1 Pixel Width> Meta Keyword 1> Meta Keywords 1 Length> H1-1> H1-1 length> H2-1> H2-1 length> Meta Robots 1> Meta Refresh 1> Canonical Link Element 1 (tells search engine which URL is the master copy]> rel=”next”1> rel=”prev”1> Size> Word Count [at least 350 words]> Text Ratio> Crawl Depth [upto 3 is acceptable but level 4 and 5 are considered bad]> Link Score> Inlinks [check each page lowest number/ highest number]> Unique Inlinks> % of Total> Outlinks [check each page lowest number/ highest number]> Unique Outlinks> Unique External Outlinks> Hash> Response time [0.25 second]> Last Modified> Redirect URI [301, 302 etc]> redirect Type> URL Encoded Address.

fig5-Screaming Frog~External
fig5-Screaming Frog~External

External

The external tab [fig5] gives information about external URIs.

Tabs: Address> Content> Status Code> Crawl Depth> Inlinks>

fig6-Screaming Frog~Protocol display in side bar
fig6-Screaming Frog~Protocol display in side bar

Screaming Frog Protocol

The protocol tab [fig6] gives data HTTP Vs HTTPS from internal and external URIs.

Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Canonical Link Element1> Meta Robots1> X-Robots-Tag 1.

fig7-Screaming Frog~Response codes display in side bar
fig7-Screaming Frog~Response codes display in side bar

Screaming Frog Response Codes

The response code tab [fig7] gives response information from internal and external URIs. To find redirects go to bulk exports and select response codes and click on redirection (3xx) in-links. The redirection_(3xx)_inlinks.csv file saved on your computer shows a table where the redirects are on that need to be changed to the actual redirects.

Tabs: Address> Status Code> Status> Indexability> Indexability Status> Inlinks> Response Times> Redirect URI> Redirect Type.

fig8-Screaming Frog-URI

URI

The URI tab [fig8] includes data related to the URLs requested which is displayed in the main box.

Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Hash> Length> Canonical Link element 1> URL Encoded Address.

Page Titles

Go to page title tab. Export the report and get information about length and character/length and pixels of the title. Should the title be long, it will get truncated in Google search results. Check your titles to spot similarities so that you can merge such similar pages into one.

Tabs: Address> Occurrences> Title 1 Length> Title 1 Pixel Length> Title 1 Pixel Width> Indexability> Indexability Staus.

Meta Descriptions

Keep meta description between 120 to 158 characters. Google’s meta description length is up to 920 pixels.

Tabs: Address> Occurrences> Meta Description 1> Meta Description 1 Length> Meta Description 1 Pixel> Indexability> Indexability Status.

Meta Keywords

Ignore the Meta Keywords tag as Google doesn’t use the “keywords” meta tag in web search ranking.

H1

Each of your pages should have a single H1 heading. Check if anything is marked zero.

Tabs: Address> Occurrences> H1-1> H1-1 Length> Indexability> Indexability Status.

H2

Every page is not required to have an H2 heading. Check if any page that is not having an H2.

Tabs: Address> Occurrences> H2-1> H2-2> H2-2 length> Indexability> Indexability Status.

fig9-Screaming Frog~H1, H2, Images
fig9-Screaming Frog~H1, H2, Images

Images

The image tab (fig9) includes data pertaining to any images crawled. The Image has to be under 100 KB for it to load faster on web pages. Check for missing or long text alt tags.

Tabs: Address> Content> Size> IMG Inlinks> Indexability> Indexability Status.

Link Metrics

Tabs: Address> Status Code> Title 1.

Connect Screaming Frog with Ahrefs, Majestic, etc. These Link Profilers scan websites and databases to determine the authority of websites for these links from where they come. Backlinks are the Off-page SEO that is outside the scope of normal SEO Audits.

fig10-Screaming Frog~Detailed Window~Lower Tabs~SERP Snippet
fig10-Screaming Frog~Detailed Window~Lower Tabs~SERP Snippet

Detailed Window – Lower Tab

Detailed Window Box displays results for any single URL that is clicked in the Main Window Box for any selection of lower tab menu: URL Info > Inlinks> Outlinks> Image Info> Resources> SERP Snippet> Rendered Page> View Source.

The SERP Snippet tab [fig10] shows results for the URL: https://sapcanvas.com/yoast-seo/ providing information on the truncated ‘text’ for the Title and Description (if any) in Search Results. This can be rectified.

 fig11-Screaming Frog~Menu Bar
fig11-Screaming Frog~Menu Bar

Screaming Frog Menu

Configuration

The default settings [fig11] are good enough.

fig12-Mode
fig12-Mode

Mode

The limit for List mode is up to 500 URLs for the free version.

fig13-Bulk Exports
fig13-Bulk Exports

Bulk Exports

The Bulk Exports tab allows the saving of any of the elements from the drop-down list [fig13] as a .csv file to your computer. For eg to obtain a list of 404 go to the menu – bulk exports and select response code and click on client error (4xx) and it will save this file for you.

fig14-Screaming Frog~Reports

Screaming Frog Reports

In the same way, all the reports refer [fig14] can be saved in a .csv file on your computer. For example, under reports select the redirect & canonical chains and the redirect_and_canonical_chains.csv file saved will show the link to a redirect and that redirect is redirecting to another page.

fig15-Screaming Frog~Sitemaps
fig15-Screaming Frog~Sitemaps

Screaming Frog Sitemaps

In the menu select site map fig15 and in the window that pops up leave the setting at default and click next. This will then allow you to save the sitemap.xml file wherever you want. Check this sitemap file errors for broken links, redirects, nasty URLs, etc. In a similar manner, you can save the images_sitemap.xml file and check errors for the missing alt text and image file sizes.

fig16-Visualizations

Visualizations

Graphical representation for each drop-down elements in the visualization tab [fig16] gets displayed.

fig17-Crawl Analysis

Crawl Analysis

Crawl Analysis fig17 configuration is only available for the paid version.

Other Options to Screaming Frog

  • visual-seo.com
  • wildshark.co.uk/spider-tool/
  • beamusup.com

Conclusion

The Screaming frog SEO spider helps quickly audit your website. The tool finds the below issues so that you can address and fix them.

  • Redirects 301, 302,
  • Duplicate content,
  • Broken links,
  • Pages with thin content,
  • Check- XML site maps, server response times, etc;
  • Optimize- page titles, internal links, etc
  • SERP snippets

The paid version allows unlimited crawling of URLs all for the price of £149.00 per Year. Besides scheduling it has options for crawl configuration, saved crawls & reloads, and AMP crawling & validation. Custom source code searches and extractions, with integrations for Google Analytics, Search Console, and Link metrics. It renders JavsScript, and store | view raw & rendered HTML Moreover set up custom robots.txt files with free technical support.