What can the free Screaming Frog licence crawl?

The free version can crawl up to 500 URLs and includes many core audit checks. Larger sites and advanced workflows normally require a paid licence.

Why can a crawler report a different result from a browser?

Rendering mode, cookies, authentication, robots rules, user agent and JavaScript can change what is returned. The crawl configuration should match the question being investigated.

Is every warning an SEO problem?

No. Crawler reports need context. A deliberate redirect, canonical or noindex rule may be correct, while a repeated warning across a key template may be urgent.

Screaming Frog

Technical SEO crawler for site audits and issue detection.

Category: SEO

Visit Website

Where Screaming Frog fits

Screaming Frog SEO Spider is a desktop crawler for inspecting how a website is connected and what each URL returns. It is particularly good at finding patterns that are difficult to see page by page: redirect chains, orphaned metadata, inconsistent canonicals, broken resources and template-wide heading issues.

A crawler for releases as well as audits

We use it before migrations to inventory the existing site and after deployment to confirm that important URLs, links and directives survived the change. Custom extraction can inspect elements that are specific to a project, while JavaScript rendering helps when the meaningful HTML is produced in the browser. Analytics and Search Console data can be joined to the crawl when prioritisation needs real traffic context.

Configuration matters

A crawl is not neutral. Limits, robots handling, user agent, rendering, authentication and URL parameters all affect the result. We save the configuration and record the crawl date so comparisons remain useful. Reports are reviewed against intent: a short title is not automatically poor, and a redirected legacy URL is not a defect if no sitemap, canonical or internal link still points to it.

Common use cases for Screaming Frog

We use the SEO Spider to crawl staging and production sites, compare releases, test redirect maps, review canonical and hreflang signals, find broken internal links, inspect metadata patterns, and extract template-level information for investigation.

F.A.Q.

: The free version can crawl up to 500 URLs and includes many core audit checks. Larger sites and advanced workflows normally require a paid licence.
: Rendering mode, cookies, authentication, robots rules, user agent and JavaScript can change what is returned. The crawl configuration should match the question being investigated.
: No. Crawler reports need context. A deliberate redirect, canonical or noindex rule may be correct, while a repeated warning across a key template may be urgent.

Related expertise