Discovering Hidden Pages and Sitemap Problems with KWT Spider and Screaming Frog
By kwt spider 10-10-2026 2
A website can contain valuable pages that are difficult to discover through normal navigation. Content like product pages, archived posts, landing pages, and new releases can easily become isolated with few or no internal links, despite remaining accessible via direct URLs. Similarly, an XML sitemap can contain obsolete URLs, omit critical pages, or include URLs that redirect or return errors.
These can complicate website maintenance and create a gap between the pages a business wants search engines to find and the URLs that are actually reachable through the site’s structure.
KWT Spider is a technical SEO crawler for professionals that allows you to crawl URLs, check response codes, analyze internal links, and generate reports for further analysis from your desktop. When combined with a structured sitemap review, it can help SEO specialists identify pages that deserve closer attention and improve their understanding of website crawl coverage.
Another tried and true tool for these tasks is Screaming Frog SEO Spider, which offers dedicated sitemap audits and orphan-page analysis workflows. Understanding how both tools fit into an audit can help website owners choose the right process for their technical SEO needs.
Why Hidden Pages and Sitemap Problems Matter
Search engines find URLs from many sources, including links on your site, XML sitemaps, links on other sites, and URLs they have already crawled. Visitors or crawlers can also find a page by using other sources . But if a page is not included in the internal navigation of a website , it can be more difficult to find .
This is particularly important for large websites where hundreds or thousands of URLs are created, updated, consolidated, or removed over time.
Typical examples are:
- Product pages that are still published after being removed from category navigation.
- Posts that no longer get internal links after a redesign.
- Landing pages only available via legacy campaigns.
- URLs included in a sitemap even though they redirect or return an error.
- Important pages that are internally linked but missing from the XML sitemap.
- Competing duplicate versions of URLs with preferred canonical URL.
- Sitemap entries that link to pages marked non-indexable.
It's not every weird URL that's an SEO problem. Some pages are deliberately omitted from navigation and indexing. The aim is to uncover unexpected gaps and determine for each URL if it should be discoverable, receive internal links, be included in the sitemap, or be removed.
How KWT Spider Finds Hidden Pages for You
KWT Spider is a desktop technical SEO spider. It can discover URLs by following links on a site, and gathering information about the pages it can reach. The crawl settings enable users to set URL limits, crawl depth, and request concurrency based on the site being audited.
This provides a useful basis for finding pages that are technically accessible but are not linked to the site in the expected way.
1. Crawl the Website to Map Its Internal Structure
You can begin by specifying the main URL of the site and setting the crawl options. KWT Spider, will follow any internal links it can find and gather information about the URLs it discovers.
The resulting dataset helps SEO professionals understand how pages link to each other and what sections may have technical issues.
For example, an ecommerce site could have hundreds of category and product pages. If some products are still available but they don’t appear to be linked from relevant categories anymore, a crawl can help identify gaps in the internal structure of the site when those pages are discoverable via other links or from other URL sources.
2. Check Crawl Depth and Internal Links
Crawl depth is the number of link steps away from the beginning of a crawl that a URL is. Pages that are just a few clicks away from the homepage may be worth a closer look, especially if they contain important content or are receiving organic traffic.
During an audit with KWT Spider, check internal linking information and crawl depth in conjunction with the importance of each page.
Search for:
- Very little internal linking to important pages.
- Pages deeply nested in the site structure.
- Pages with broken navigation trails.
- URLs that appear to have no useful route from the main website.
A deeper page is not automatically problematic. Most large websites require multiple levels of navigation. The question is whether users and crawlers can reach important content through clear, helpful links.
3. Crawl URLs that a normal crawler can't find
Normally a crawler finds its pages by following links from the starting URL. Which means if a page has no incoming links from the found website structure it might not be crawled.
This is where the second URL source comes in handy. Compare crawl results to an XML sitemap, a list of URLs exported from a content management system or other URL inventories that may be available.
KWT Spider enables users to import URL lists for auditing purposes, which can help teams review known URLs outside of the links found during a regular crawl.
For example, if a marketing team has a spreadsheet of published landing pages, it can use that list to check whether the pages respond correctly and contain the expected SEO elements. Comparing this list against discovered URLs helps flag pages needing deeper review.
This is an important distinction: Importing / auditing a list of known URLs is not the same as automatically discovering every orphan page on a site. To do a good orphan page analysis, you need to compare multiple URL sources.
KWT Spider to Discover XML Sitemap Problems
While XML sitemaps highlight key pages for search engines to discover, they should support—not replace—a strong internal linking structure.
Besides the crawling and reporting functions, KWT Spider also offers XML sitemap generation. You can use its crawl data to review the technical condition of discovered URLs and determine which pages need to be reviewed before a sitemap update.
1. Check If Important Pages Are Missing
Compare URLs discovered during a crawl with the URLs present in the site’s XML sitemap.
If it is important, canonical and meant to be indexed, you may need to add an internally linked page that is not in the sitemap. Pages intentionally omitted, however, should not be automatically considered errors.
A business may publish a new service page and link to it from its main services section but forget to update its sitemap. If the SEO team notices this discrepancy, they can then check on the page’s indexing settings and add it, if appropriate.
2. Find URLs that Redirect and Generate Errors
As a general rule, a sitemap should list the URLs that a site prefers to have crawled by search engines, not obsolete URLs, nor URLs that point to other locations.
Analyze response-code and redirection in KWT Spider for sitemap entries pointing to:
- Send replies somewhere other than where you intended.
- Broken pages with 4xx status codes.
- Server errors happening with 5xx status codes.
- Unexpected redirect chains.
- URLs that redirect to a different canonical page.
After you’ve confirmed the issues, update the sitemap to reflect the URLs you want to be crawled.
3. Audit Canonicalization & Indexability Signals
A URL can return a successful response and still not be suitable for a sitemap. For example, a page may have a no-index directive or a different canonical URL.
The technical audit features of KWT Spider can help you collect important page data such as canonical tags and other on-page indicators. Check these in combination with response codes to see which URLs should be included in the sitemap.
The goal is consistency in internal linking, canonical declarations, indexing directives, and sitemap entries.
4. Generate & Validate the Sitemap
Once the issues found have been fixed, KWT Spider can be used to create an XML sitemap from the crawl URLs that qualify. Check the output before submitting it to search engines.
Be sure that:
- Important, canonical URLs are provided
- URLs that cause errors or redirect are excluded.
- Proper handling of URLs that should not be included in search results.
- Sitemap includes correct preferred versions of URLs.
- The sitemap can be found in its expected place.
Even a generated sitemap must be checked against the website’s indexing strategy. The automated generation does not obviate the need to consider what pages are appropriate to include.
KWT Spider vs Screaming Frog for Sitemap Audits and Hidden Pages
Both tools can be helpful for technical SEO audits but the features and workflows available are important considerations when planning an audit.
KWT Spider - Practical Crawl Analysis
KWT Spider is a desktop crawler, which checks the response and analyzes the on-page content and then exports the report. It helps users to find common technical issues, check known URLs, and investigate the structure of a website.
It can be useful if a team has a list of pages from another source and wants to audit those URLs separately from a normal internal-link crawl; it has the ability to import a list of URLs. The sitemap generation feature can also assist with the task of creating an updated sitemap after corrections are made.
KWT Spider is a tool specifically for SEO professionals who want to work with exported crawl data and process SEO crawl data locally.
Screaming Frog to Find a Dedicated Sitemap and Orphan URLs
Screaming Frog has specific sitemap auditing workflows. Its documentation mentions comparing URLs discovered during a website crawl to those listed in XML sitemaps to find orphan URLs and URLs not in the sitemap . It can also pull in Google Analytics and Google Search Console data to find extra URLs that may not be found through internal links. <Cite refs={["turn806324search1","turn806324search2"]}/>
These features can be useful in an audit that requires combining multiple URL sources into a single analysis. Some workflows require additional configuration, analysis post crawl or a paid license depending on the feature used. <Cite refs={["turn806324search1","turn806324search2"]}/>
KWT Spider and Screaming Frog should then be considered against the task scope. A simple crawl and URL-list review might necessitate a different workflow than an audit that pulls together crawl data, sitemap entries, and analytics or Search Console data.
That is not to say one crawler is better in all cases. It’s about choosing a process that gives the team enough evidence to find missing pages and fix sitemap inconsistencies.
A Step-by-Step Workflow to Find Hidden Pages and Sitemap Gaps
To make the audit more comprehensive, establish a repeatable process that combines the following steps.
Step 1: Crawl a website. Get response codes, link information and relevant on-page data with KWT Spider to find internal URLs.
Step 2: Additional URL Sources. Get the latest XML sitemap and, if it's available, the URL lists from the CMS, analytics platform, or Google Search Console.
Step 3: List of URLs, comparison. Locate pages that exist in one source but are missing from another. These are possible lines of inquiry, not definitive errors.
Step 4. Check the remaining URLs. Import known URLs where supported and check their response codes, canonical signals, metadata and other technical elements.
Step 5: Categorize each finding. Separate important orphan pages, outdated sitemap entries, sitemap URLs that are missing, duplicate versions and pages that are intentionally excluded.
Step 6: Correct the root cause. Update the sitemap, fix redirects, or delete outdated entries to match the intended website structure. Add correct internal links.
Step 7: Re-crawl & verify. Repeat the relevant checks to verify that the changes are implemented correctly.
This workflow helps you avoid a common pitfall: treating every difference between a sitemap and a crawl as a problem. A useful audit separates deliberate exclusions from pages that have been missed.
Common Mistakes to Avoid When Auditing Your Sitemap
A capable crawler is important, but the quality of an audit is dependent on how the findings are interpreted.
If you think all orphan pages need to be deleted: An orphan page might contain valuable content that just needs better internal linking.
Including each found URL in the sitemap: Search filters, duplicate URL variations, redirects and non-indexable pages may not be sitemap material.
Ignoring crawl configuration: URL limits, crawl depth, robots.txt rules and rendering settings can affect the pages you find.
Indexability validation with a successful response: A page with HTTP 200 can still have restrictions or canonical signals that need attention.
The sitemap alone will not suffice: A sitemap is a useful list of the URLs you want to include but it does not tell you if those pages are well connected within the site.
Not validating fixes: Just updating a sitemap or adding internal links doesn’t mean it’s done. A later crawl can confirm that the desired changes have been made.
Last Thoughts
Recovering hidden pages and fixing XML sitemap issues is an important part of maintaining a technically sound website. A structured crawl can help SEO teams identify differences and determine what corrections are needed when critical URLs are removed from internal navigation or when sitemap entries become stale.
KWT Spider provides a handy desktop workflow for website crawling, URL analysis, response-code checking, URL-list auditing and sitemap generation. These capabilities make it worth a look for teams that want to investigate technical issues and organize their findings locally.
Screaming Frog provides dedicated sitemap comparison and orphan-page workflows, with the option to bring in additional URL sources. These capabilities could be especially useful for audits that combine crawl results with analytics and Search Console data.
Ultimately, a good crawl, an intelligent URL comparison and good SEO judgment is the best audit. Website owners can determine which pages need to be linked, which URLs need to be included in the sitemap, and which entries need to be corrected or removed, creating a clearer structure for users and search engines.
Frequently Asked Questions
1. What is a hidden page in technical SEO?
A hidden page is often a URL that remains accessible but is difficult to discover through a website's normal internal links. Such pages may be called orphan pages when no internal linking path is found during a crawl.
2. How can KWT Spider help find hidden pages?
KWT Spider crawls discoverable internal URLs and collects technical information about them. Comparing the crawl with an imported URL list or other known URL sources can help identify pages that need further investigation.
3. What are common XML sitemap problems?
Common problems include outdated URLs, redirecting entries, broken URLs, missing important pages, duplicate URL versions, and entries that conflict with canonical or indexing directives.
4. Can an XML sitemap contain URLs that are not linked internally?
Yes. A sitemap can list URLs that are not discoverable through the website's internal links. These URLs should be reviewed to determine whether they are important pages that need internal links or obsolete entries that should be removed.
5. How does KWT Spider compare with Screaming Frog for sitemap auditing?
KWT Spider supports crawling, technical analysis, URL-list auditing, and XML sitemap generation. Screaming Frog offers dedicated sitemap comparison and orphan-URL workflows, with additional URL-source integrations available for more extensive audits.
6. Should every page discovered by a crawler be added to the XML sitemap?
No. Include URLs that are important, canonical, and intended for indexing. Redirecting, duplicate, non-indexable, and intentionally excluded URLs generally require different treatment.
7. How often should a website's sitemap be audited?
Audit the sitemap after major website changes, migrations, URL restructuring, or substantial content updates. High-change websites may also benefit from regular automated checks and periodic manual reviews.
