Automate SEO Indexing and Content Publishing for a New Website

The Problem

A startup just launched a documentation site with 200 pages. After two weeks, Google has only indexed 34 of them. The remaining 166 pages are invisible to search traffic, which means the company is missing out on organic visitors who are searching for exactly the topics those pages cover. Manually submitting URLs through Google Search Console one at a time is tedious, and the team has no way to verify which pages are actually indexed versus stuck in a crawl queue. Meanwhile, competitors with established domains rank for the same keywords because their content has been indexed for months.

The Solution

Using google-indexing to submit URLs to Google's Indexing API in bulk, web-scraper to audit the live site for crawlability issues, and markdown-new to scaffold SEO-optimized content pages with proper frontmatter and meta tags, the team gets 95% of their pages indexed within 72 hours.

Step-by-Step Walkthrough

1. Audit the site for crawlability issues

Crawl the live site to identify pages that are blocking indexing due to missing meta tags, broken links, or incorrect robots directives.

Use web-scraper to crawl https://docs.example.com and audit all 200 pages. For each page, extract: title tag, meta description, canonical URL, robots meta tag, internal links, response code, and page load time. Flag pages that have noindex tags, duplicate titles, missing meta descriptions, or broken internal links. Output the audit to /reports/crawl_audit.csv.

2. Fix issues and scaffold missing content pages

Generate properly structured markdown pages for any gaps found in the site's topic coverage.

Use markdown-new to create 15 new documentation pages for topics identified as gaps in our coverage. Each page should have YAML frontmatter with title, description (under 160 characters for meta), date, and category. Include proper heading hierarchy (single H1, H2 for sections, H3 for subsections), internal links to related existing pages, and a FAQ section targeting long-tail search queries. Save to /content/docs/.

3. Submit all URLs to Google for indexing

Use the Indexing API to notify Google about all new and updated pages in bulk.

Use google-indexing to submit all 215 page URLs (200 existing plus 15 new) to the Google Indexing API. Submit in batches of 50 with a 2-second delay between batches to respect rate limits. For pages that were updated after the crawl audit fixes, use the URL_UPDATED notification type. For new pages, use URL_UPDATED as well. Log each submission response to /reports/indexing_log.csv with timestamp, URL, and API response status.

The indexing submission log captures the API response for each URL, making it easy to identify failures:

google-indexing submit --urls /reports/all_urls.txt --batch-size 50 --delay 2s

Batch 1/5: 50 URLs submitted
  [200] https://docs.example.com/getting-started
  [200] https://docs.example.com/api/authentication
  [200] https://docs.example.com/guides/deployment
  ...
Batch 2/5: 50 URLs submitted
  [200] https://docs.example.com/api/webhooks
  [429] https://docs.example.com/api/rate-limits   (rate limited, queued for retry)
  ...
Batch 5/5: 15 URLs submitted (final batch)

Summary:
  Submitted:    215 URLs
  Accepted:     212 (200 OK)
  Rate limited:   3 (429, auto-retried after 60s, all succeeded)
  Errors:         0
  Log saved:    /reports/indexing_log.csv

The three rate-limited URLs were automatically retried after a 60-second backoff. All 215 URLs were accepted by the Indexing API on the first run.

4. Monitor indexing progress and resubmit failures

Track which pages get indexed and resubmit any that remain unindexed after 48 hours.

After 48 hours, use web-scraper to check Google's cache for each of the 215 URLs by querying "cache:url" and checking for a 200 response. Generate a report showing: pages confirmed indexed, pages still pending, and pages that returned errors during submission. Resubmit any unindexed pages. Compare organic search impressions this week versus last week from Search Console data.

Real-World Example

A developer tools startup launched their docs site and saw only 17% of pages indexed after two weeks. After running the crawl audit, they discovered 23 pages had accidental noindex tags from a staging environment config that carried over to production, and 41 pages had duplicate title tags making Google unsure which to index. They fixed those issues, scaffolded 15 new pages targeting high-volume keywords, and submitted all 215 URLs via the Indexing API. Within 72 hours, 204 of 215 pages were indexed. Organic search traffic went from 120 to 890 weekly visits within the first month.

Tips

• Run the crawl audit before submitting URLs. Submitting pages that have noindex tags or broken canonical URLs wastes your daily Indexing API quota and can delay indexing of valid pages.
• The Google Indexing API has a daily quota of 200 URL notifications per property. Plan your batch submissions accordingly and prioritize high-value pages if you exceed the limit.
• Schedule a weekly re-check of the indexing status for your full URL set. Google can drop pages from its index if they return errors or if crawl budget is reallocated.
• Meta descriptions under 160 characters perform best in search results. Longer descriptions get truncated, which reduces click-through rates.