Have you published a website or a new page, only to find that it does not appear on Google? Its absence from search results does not tell you what is wrong. Google might not have discovered the page, it might be excluded from indexing, or it might rank too low for the query you tried. Before changing settings, check each stage separately. This guide helps you identify possible obstacles without assuming that every published page will automatically be indexed.
1. Discovery, indexing and ranking are different stages
When people talk about Google indexing, they sometimes use the term for the entire process. It helps to distinguish three stages:
- Discovery: Google learns that a URL exists, perhaps through a link on another page or an XML sitemap.
- Indexing: after fetching and analysing the page, Google decides whether to store it in its index and which URL best represents the content.
- Ranking: for a particular search, Google decides whether and where to display the page in the results.
Crawling takes place between discovery and indexing. If access is blocked, the server returns errors or the content cannot be interpreted properly, the process can be affected.
An indexed page is not guaranteed a particular position or visibility for every phrase you want to target. Similarly, submitting a URL does not require Google to add it to the index. Your first step should therefore be to check the page’s status, rather than repeatedly searching for your preferred keywords.
2. Inspect the URL in Google Search Console
Google Search Console is the main starting point for these checks. Add your website property and verify that you are authorised to manage it using an available verification method. Make sure the property covers the address you are investigating, including HTTPS and the relevant subdomain where applicable.
- Copy the exact URL of the page you want to check.
- Enter it in the URL Inspection tool.
- Read its indexing status and the accompanying explanation.
- Review the crawling information, whether indexing is allowed and the canonical URL selected by Google.
- Use the live URL test to check current accessibility and certain technical conditions.
A successful live test does not confirm that the page is indexed. It checks the situation available at the time of testing, while the indexing information reflects data already known to Google. A positive test result does not guarantee future inclusion in the index.
Also review the page indexing report to identify issues affecting several URLs. A search using the site: operator can provide clues, but it is not a complete inventory of indexed pages and should not replace URL inspection.
3. Check that your XML sitemap lists the right pages
An XML sitemap helps Google discover the URLs you want to put forward for indexing. It is not an instruction to index them and does not replace internal links between pages.
Find the sitemap address in your application settings, SEO extension or website documentation. File names vary, so do not assume that every website uses the same location. Some sites publish a sitemap index that points to several individual sitemaps.
- Open the sitemap and check that it is publicly accessible without a login.
- Make sure it contains the final URLs, using the correct domain and protocol.
- Remove addresses that redirect, return errors or are intentionally excluded with noindex.
- Include the canonical pages you want to present in search, rather than duplicate versions.
- Submit the sitemap address in the relevant Google Search Console section and review any processing messages.
If you use WordPress, check for conflicting SEO settings across plugins. A single, consistent configuration is easier to investigate. For your website infrastructure, you can explore WordPress hosting, but choosing hosting does not replace indexing checks.
4. Read robots.txt rules in context
The robots.txt file, published at the root of the relevant domain or subdomain, sets crawling rules. Check it on the exact host serving the page you are investigating: rules for one subdomain do not automatically apply to another.
Look at the User-agent groups and their Disallow or Allow rules. A Disallow: / rule in the applicable group can block crawling across the whole site. A directory rule can affect every page within that section.
Blocking crawling is not the same as preventing indexing. Google can discover a URL through links and may display it without being able to fetch its content. If robots.txt blocks the page, the crawler also cannot read a noindex directive within it.
Do not delete every rule simply to make one article accessible. Identify the relevant rule, establish whether it intentionally restricts a section, and change it only when you understand the effect. Robots.txt is not a security mechanism: private content needs authentication and appropriate access controls.
5. Look for noindex directives and check the canonical URL
A noindex directive asks a search engine not to include a page in its results. It can appear in an HTML robots meta tag or an X-Robots-Tag HTTP header. The header is also relevant to resources that are not HTML pages, such as PDF documents.
- Check the site’s visibility settings and the page’s SEO options.
- Look for noindex in the HTML and inspect the HTTP response headers.
- Check whether a site-wide rule is accidentally affecting every page.
- If the page should be public and eligible for indexing, remove the directive from the configuration that generates it.
- Clear any relevant caches and check the public response again.
A common issue is leaving test-environment settings in place after launch. However, do not remove noindex from pages that should remain excluded. Remember that the directive does not protect confidential information from direct access.
Check the declared canonical URL as well. If the page identifies another URL as canonical, Google may index that version instead of the one you inspected. A canonical tag is a signal, not an absolute instruction, so compare the declared URL with Google’s selected canonical in Search Console.
6. Finish with technical checks and a justified indexing request
If you cannot find an obvious restriction, check that the page returns an appropriate HTTP response, does not require authentication and does not display an error message while returning a success status. Confirm that Google can access the main content, particularly when the site relies on JavaScript.
Add relevant internal links to the page and make sure it offers useful content rather than a near-identical version of another page. If you encounter server errors, investigate the application and your web hosting resources. For questions about the hosting service, use the contact page and provide specific URLs and error details without sharing passwords.
Your practical next step: inspect the URL, check the sitemap, robots.txt, noindex directives and canonical URL, then fix the issue you identify. After checking the changes, you can request indexing through Search Console if the option is available. Monitor the reports afterwards: a request does not guarantee indexing, a processing deadline or a position in search results.
