Technical SEO for Non-Techies: Crawling, Indexing and Site Speed

Technical SEO sounds like the part you need a developer for. In practice, the issues that hold small business websites back are almost always simple, and about eighty per cent of them can be fixed by the person who edits the website.

This guide explains what is actually happening under the bonnet, in order, with the checks you can run today and a clear marker for the handful of jobs that genuinely need a developer.

How crawling, rendering and indexing actually work

Three distinct stages, and confusing them causes most of the panic we see in client emails.

1. Crawling. Googlebot requests your URLs, following links and reading your sitemap. If it cannot reach a page, nothing else can happen.
2. Rendering. Google runs the page much like a browser, including JavaScript, to see the finished content. If your content only appears after a script that fails, Google may see an empty page.
3. Indexing. Google decides whether the rendered page is worth storing and what it is about. Being crawled does not guarantee being indexed — that is a quality and duplication judgement.

The single most useful tool here is free: Search Console's URL Inspection. Paste any URL and it tells you whether the page is indexed, when it was last crawled, which canonical Google chose, and — under "View crawled page" — exactly what HTML it received. If you learn one technical tool, learn this one.

robots.txt, sitemaps and the canonical tag

robots.txt

A plain text file at yourdomain.com/robots.txt that tells crawlers where not to go. Visit yours right now. A safe, typical file looks like this:

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /wp-admin/

Sitemap: https://yourdomain.com/sitemap.xml

The dangerous line is Disallow: /, which blocks your entire site. It gets left behind after a site build more often than you would believe — we have found it on live sites three times in the last two years, each costing months of invisibility.

Remember: robots.txt controls crawling, not indexing. To keep a page out of the index you need a noindex meta tag, and the page must be crawlable for Google to see it.

XML sitemaps

A sitemap is a list of the URLs you want indexed. Most platforms generate one automatically — WordPress via Yoast or Rank Math, Shopify and Squarespace natively. Two rules: it should contain only canonical, indexable, 200-status URLs, and it should be submitted in Search Console.

Canonical tags

A canonical tag says "this is the definitive version of this page". Most pages should self-canonicalise. Problems arise when a template hard-codes the homepage as the canonical for every page — which tells Google to ignore everything else on your site. Check a few pages' source for rel="canonical" and confirm each points at itself.

Site structure, URLs and redirects

Structure is a technical topic that is really an information-architecture topic. The rule of thumb: every important page should be reachable from the homepage in three clicks or fewer, and its URL should read like a path.

Good:  /services/local-seo/
       /articles/keyword-research-for-beginners/

Avoid: /page-id=428?ref=nav
       /services/seo/local/uk/chester/map-pack/services/

Redirects, in plain English

Two rules that prevent most redirect damage: never chain more than one redirect (A→B→C loses value and slows the page), and always redirect to the closest equivalent page rather than dumping everything on the homepage. A mass redirect to the homepage is treated as a soft 404 and gains you nothing.

Core Web Vitals without the jargon

Three measurements, all about how the page feels to a real person on a real phone.

The five fixes that move the needle most

1. Compress and resize your images. Nine times out of ten this alone fixes LCP. A 3,000-pixel-wide photo displayed at 800 pixels is wasting most of its bytes.
2. Lazy-load everything below the fold — but never the hero image, which should load eagerly.
3. Audit your plugins and scripts. Every chat widget, popup tool, heatmap and social feed costs interactivity. Remove the ones nobody looks at the data for.
4. Set width and height on images. One attribute pair, most of your CLS problem solved.
5. Check your hosting. Shared hosting at £3 a month often has a server response time of over a second before anything else can start. This is the one fix that needs money rather than effort.

Measure with PageSpeed Insights, but trust the "Core Web Vitals assessment" from real users over the lab score. A 68 lab score with passing field data beats a 95 with failing field data.

Mobile, HTTPS and the basics people skip

Google indexes the mobile version of your site. If your mobile page hides content that appears on desktop, the hidden content effectively does not exist for ranking purposes.

• Open your own site on your own phone and complete the main action. It is astonishing how often nobody has.
• Check tap targets. Links crammed together in a footer are a genuine usability issue.
• Check that your cookie banner does not cover the answer and can actually be dismissed on a small screen.
• Confirm HTTPS everywhere. Every internal link, image and script should be https://. Mixed content warnings erode trust and sometimes break layout.
• Pick one canonical hostname. https://www. or https://, not both serving content. Redirect the other.

Structured data that is worth adding

Schema markup describes your page in a format machines parse reliably. It does not lift rankings by itself; it makes you eligible for richer presentation, which lifts clicks.

For a small business site, four types cover nearly everything worth having:

• Organization or LocalBusiness on the homepage and contact page — name, address, phone, opening hours, social profiles.
• Service or Product on the pages that sell something, including price where you publish it.
• Article on blog posts, with author and publication date.
• FAQPage where you genuinely have questions and answers visible on the page.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Your Business",
  "telephone": "+44 1244 555019",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "1 Steam Mill Street",
    "addressLocality": "Chester",
    "postalCode": "CH3 5AN",
    "addressCountry": "GB"
  },
  "openingHours": "Mo-Fr 09:00-17:00"
}
</script>

Validate with the Rich Results Test, and only mark up what is genuinely on the page. Invented reviews and invisible FAQs are exactly the kind of thing that earns a structured data manual action.

A 30-minute technical check you can run yourself

No crawler subscription needed. Work down the list in order and write down what you find.

1. Visit yourdomain.com/robots.txt. Is anything important disallowed?
2. Visit your sitemap. Does the URL count roughly match your real page count?
3. Search site:yourdomain.com. Are the results roughly what you expect, with no staging or duplicate URLs?
4. Search Console → Pages. Read the "Why pages aren't indexed" reasons and note the biggest group.
5. URL-inspect your three most important pages. Indexed? Correct canonical?
6. Run PageSpeed Insights on a service page and a blog post, on mobile.
7. Open both on your phone and try to complete the main action.
8. Click through your own main navigation and footer, looking for 404s.
9. Check that http:// and the non-preferred www variant both redirect to the canonical version.
10. Check one blog post's source for the rel="canonical" and confirm it is self-referencing.

If that produces a list you would rather hand over, our SEO Health Audit does the full 60-point version, and the SEO Starter Pack includes the checklist in a printable form.

Frequently asked questions

Do I need a developer for any of this?

Usually for three things: server response time (hosting), redirect rules at server level, and removing JavaScript that a theme depends on. Everything else in this guide is achievable in your CMS settings.

How long until technical fixes show results?

Indexing fixes can show within days. Speed improvements show in field data after roughly 28 days, because Core Web Vitals use a rolling window. Structure changes take a few weeks to settle.

Is a 100/100 PageSpeed score worth chasing?

No. Get into the green on real-user Core Web Vitals and spend the remaining effort on content and internal links, where the return is higher.

My site is on Wix/Squarespace — am I at a disadvantage?

Marginally, on fine control of speed and schema. Not enough to matter against typical small-business competitors, all of whom have the same constraint. Content and links decide those races.

Should I worry about crawl budget?

Not under roughly ten thousand URLs. Crawl budget is a genuine concern for large e-commerce sites and essentially never for a small business site.

Martin Huntbach

SEO strategist & co-founder

Martin has been doing SEO for small businesses since 2009. He runs the technical and strategy side of the studio and has a well-documented weakness for spreadsheets.

More about the studio