A working guide to Screaming Frog SEO Spider: crawl setup, response codes, redirect chains, duplicate titles, canonicals, sitemap exports and a repeatable audit workflow.
What Screaming Frog SEO Spider Actually Does
Screaming Frog SEO Spider is a desktop crawler that visits your site the way a search engine does, following every link it finds and recording what it sees: response codes, page titles, meta descriptions, headings, canonical tags, images, directives and more. Where a manual review checks ten pages, a crawl checks ten thousand and returns them sorted, filtered and exportable to spreadsheets. It sees the site as machines see it, not as humans imagine it.
The free licence covers 500 URLs per crawl, which suits small sites and single sections; the paid licence removes the cap and adds scheduling, saved crawls and deeper configuration. Either version runs on Windows, Mac and Linux. Its value is not the tool itself but the audit discipline it enforces — every recurring SEO failure, from broken links to duplicate titles, is findable in minutes rather than discovered by accident.
Crawl Setup: Mode, Limits and Rendering
Start with the basics: enter the URL and choose Spider mode for a standard link-following crawl, or List mode when you want to check a specific set of URLs — old redirect targets, for instance. Set sensible limits in the configuration panel: maximum URL count, response-time thresholds and crawl depth, so a sprawling site returns results before your coffee goes cold. Both modes accept authentication, so member-only areas can be crawled on your own site.
The setting most audits miss is rendering. By default the crawler reads raw HTML; sites built on JavaScript frameworks need Configuration, then Rendering, set to JavaScript, so the crawler executes scripts and sees the page as a browser does. Get this wrong and you audit half a website. Respect robots.txt by default, and override it only when deliberately auditing blocked sections of your own site. Render-only pages are invisible to a plain crawl — and invisible to audits.
Response Codes: 4xx, 5xx and Redirect Chains
Open the Internal tab and filter by Response Codes. Client errors, the 4xx family, are broken links — pages deleted, URLs mistyped, internal links left rotting after a redesign. Each one wastes crawl budget and leaks authority; fix them with 301 redirects to the nearest relevant page or correct the source link. Server errors, the 5xx family, are more urgent: they signal that pages sometimes fail to load at all.
Redirects deserve their own pass. Filter for redirection — the 3xx codes — and hunt down chains: URLs that bounce through two or more hops before arriving, because chains dilute link equity and slow crawlers. Point every redirect at the final destination directly. The tool lists the chain per URL, so the cleanup becomes a spreadsheet exercise — source URL, final destination, one hop, done. One hop keeps authority flowing and crawlers fast; every extra hop taxes both.
Titles, Meta Descriptions and H1 Audits
The Page Titles tab sorts titles by missing, duplicate, over 60 characters, under 30 and multiple. Missing and duplicate titles are pure lost ranking opportunities — every important page needs a distinct, keyword-carrying title. Meta descriptions get the same treatment; they rarely move rankings directly, but they decide the click, and duplicates here mean Google writes your snippets for you. Titles are the strongest on-page signal you control — treat every duplicate as a decision you never made.
H1 audits follow the same rhythm: filter for missing, multiple and duplicate H1s. A missing H1 leaves Google guessing the page's subject; multiple H1s usually reveal template problems; duplicates across a blog often mean recycled headings where unique ones should differentiate. The fix list from these three tabs alone — titles, descriptions, H1s — typically justifies the entire audit on most sites. Fix them in priority order of traffic, not in alphabetical order of URL.
Images, Canonicals and Directives
The Images tab exposes two quiet performance killers: missing alt text, which costs accessibility and image-search visibility, and oversize images that bloat the load time of every page they sit on. Export the list, prioritise the pages that earn money, and fix alt text while compressing the heavy files. Both fixes are mechanical, cheap and compound across every page they touch. Minutes of work, returned as speed on every visit thereafter.
Canonicals and directives decide what Google indexes, so verify them deliberately. Check for missing canonicals, conflicting ones and pages that canonicalise to the wrong URL. Review the Directives tab for noindex tags — intentional ones are fine, accidental ones are costly — and confirm robots.txt is not blocking folders you actually want crawled. Half of all my-page-vanished mysteries resolve inside these two checks. Indexation errors stay silent until a traffic report explains them months later.
Sitemaps, Comparisons and API Integrations
Exports are where the crawler pays for itself: an XML sitemap built from your actual crawl reflects reality rather than guesswork, and the bulk exports hand clean spreadsheets to developers. The crawl comparison feature, invaluable since its arrival, diffs two crawls of the same site — before and after a migration, a redesign or a fix sprint — and shows precisely what changed: new 404s, lost titles, shifted canonicals.
Connect the integrations and the data multiplies: the Google Search Console API adds clicks and impressions per URL, Google Analytics layers behaviour data on top, the PageSpeed API attaches performance scores, and custom APIs extend it further. Connecting Search Console before an audit lets you prioritise by traffic — a duplicate title on a page earning visits outranks a pristine page nobody reaches. Prioritise by impact, never by URL order.
A Practical Audit Workflow That Repeats
A repeatable sequence: crawl with rendering on; clear 4xx and 5xx first, then flatten redirect chains; verify noindex and canonical behaviour on money pages; fix missing and duplicate titles, descriptions and H1s; clean image alt text and oversize files; export the sitemap; then rerun the crawl and use comparison mode to confirm every fix landed. One afternoon for a small site; a week for a large one.
Run the full audit quarterly and after every migration or redesign, and hand exports to developers with the priority order attached. Pair the crawl with Search Console data so the queue is ranked by commercial impact rather than alphabetically. This is the unglamorous core of technical SEO — and it is exactly how our team opens every engagement, from local Tricity businesses to sites serving clients across 7 countries.
Want This Done for Your Business?
Topseoservices implements everything in this article — and reports results in leads and revenue, not vanity metrics. Start with a free 30-minute strategy call, then a 7-day free trial with no advance payment.
