Why My Blog Won’t Serve HTML to Bots

TL;DR: Serving Markdown instead of HTML to verified bots gives you two things: your content gets cited accurately in AI answers, and Data Transfer Out costs drop by 94 %. AWS WAF Bot Control identifies the bots, and CloudFront Functions rewrites the URL.

Key Terms: Bot Control Targeted = AWS WAF’s bot detection level combining IP reputation, TLS fingerprinting, and behavioral analysis | CloudFront Functions = lightweight JavaScript execution at CloudFront edge (viewer-request/viewer-response stage) | Data Transfer Out = cost charged for traffic leaving CloudFront to the internet | Verified Bot = a crawler whose identity AWS WAF has confirmed (Googlebot, Bingbot, GPTBot, etc.)

Expected Benefits

Your content gets cited accurately in AI answers

Does serving Markdown bring more bots? Profound ran an A/B test across 381 pages and found that serving Markdown didn’t dramatically increase bot traffic (Profound, Feb 2026). A ~20 % directional increase was observed for ChatGPT-User, but it wasn’t statistically significant.

So why serve Markdown? Not to attract more bots, but to ensure the bots already visiting understand and cite your content accurately.

When AI agents use web-collected content in their answers, they split documents into small chunks and search through them. Here’s the difference:

• Mechanically splitting HTML: Cuts can land in the middle of a <table> tag, or separate a section heading from its body. Context breaks.
• Splitting Markdown at ## header boundaries: Each chunk becomes a clean “heading + content below it” unit. When an AI searches for “how to configure WAF Bot Control,” it’s far more likely to retrieve the exact relevant section.

Research shows this difference improves retrieval accuracy by 40–60 % (maxun.dev). Additionally, even the best HTML-to-Markdown converters achieve only 81.8 % ROUGE-N F1 (arxiv:2511.16397), but serving the original Markdown means zero conversion loss.

For content creators, the implication is clear: your code blocks, configuration examples, and step-by-step guides get cited in AI answers without distortion. If SEO is visibility on search engine result pages, this is visibility in AI answers.

Reduce your bandwidth costs

As of 2025, 53 % of all web traffic is bots (Imperva Bad Bot Report 2026). AI crawlers alone account for 4–6 % of all requests (Cloudflare Radar), but verified bots in total — search engines (Googlebot, Bingbot) + AI crawlers (GPTBot, ClaudeBot) + social media bots (Twitterbot, Facebookbot) — can reach 10–30 % depending on the site. Serving full HTML to these bots consumes your CloudFront Data Transfer Out budget.

Here’s a concrete calculation (assuming 20 % verified bot traffic):

At CloudFront Data Transfer Out pricing of $0.085/GB (Asia), that’s $1.45/month saved. Seems small for a blog, but at 10 million pageviews it’s $14.5/month, and at 100 million it’s $145/month. The effect scales with traffic.

Not served to just anyone

The important point: this isn’t served to “any bot.” AWS WAF Bot Control uses IP reputation, TLS fingerprinting, and behavioral analysis to provide Markdown only to verified bots. Unknown scrapers and malicious bots are either blocked by WAF rules or receive only HTML.

Prior Art

I didn’t come up with this idea first. Colleagues were already running similar implementations.

Kenex Huang built a system where appending ?ua=genaibot to the same URL serves a GEO (Generative Engine Optimized) version for AI bots. Comparing the original page and the GEO version side by side, the difference is striking — structured content that AI can immediately digest instead of HTML layout.

Eitav Arditti documented the edge-level content negotiation pattern combining CloudFront and WAF in “Content Negotiation at the Edge with Amazon CloudFront and AWS WAF.”

Cloudflare also launched “Markdown for Agents” in February 2026, which sparked several community blogs implementing the same pattern on AWS.

There are also solution providers offering this as a service.

Looking at these approaches, I thought: “My blog already has Markdown source files from Hugo — I can just serve the originals without any conversion.” Adding a WAF Bot Control security layer to serve only to verified bots is what this post implements.

Architecture

User/Bot → CloudFront → S3 (HTML + Markdown)
                ↑
           AWS WAF Bot Control (Targeted)
           → Inserts x-amzn-waf-verified-bot: true header for verified bots
                ↓
           CloudFront Function (Viewer Request)
           → Checks header, rewrites URL to /md-content/*.md

Two key components:

1. WAF Bot Control → CloudFront Function: Infrastructure that identifies verified bots and rewrites URLs to Markdown paths
2. Markdown co-deployment during Hugo build: Mirrors content/*.md source files to public/md-content/ and uploads to S3 together

Step 1: Identify Verified Bots with AWS WAF Bot Control

This implementation has two parts. First, Bot Control classifies bots and applies labels. Then a separate custom rule checks the label combination and inserts a custom header.

Bot Control rule: Bot Control at the Targeted level (Version_5.0) analyzes requests and applies labels. The CategoryAI rule is overridden to Count — by default it blocks all AI bots (verified or not), but we need verified AI bots to reach the label-matching rule and receive the custom header. Static assets are excluded from inspection via scope_down_statement to reduce Targeted-level costs.

# infra/waf.tf — Bot Control
rule {
  name = "aws-bot-control"

  override_action {
    none {}
  }

  statement {
    managed_rule_group_statement {
      vendor_name = "AWS"
      name        = "AWSManagedRulesBotControlRuleSet"
      version     = "Version_5.0"

      managed_rule_group_configs {
        aws_managed_rules_bot_control_rule_set {
          inspection_level = "TARGETED"
        }
      }

      # CategoryAI blocks all AI bots by default.
      # Override to Count so verified AI bots reach the label-matching rule.
      rule_action_override {
        name = "CategoryAI"
        action_to_use {
          count {}
        }
      }

      # Exclude static assets from Bot Control inspection to reduce costs
      scope_down_statement {
        not_statement {
          statement {
            regex_match_statement {
              regex_string = "\\.(css|js|jpg|jpeg|png|gif|svg|ico|woff2?|ttf|eot|webp|avif|map)$"
              field_to_match {
                uri_path {}
              }
              text_transformation {
                priority = 0
                type     = "LOWERCASE"
              }
            }
          }
        }
      }
    }
  }
}

Custom rule — label matching to insert header: Checks for the verified label AND a harmless category label (search_engine, ai, content_fetcher, etc.) and inserts a custom header. social_media is excluded — social media bots need HTML with OG tags for link previews.

# infra/waf.tf — Verified bot label matching
rule {
  name = "label-verified-harmless-bot"

  action {
    count {
      custom_request_handling {
        insert_header {
          name  = "verified-bot"
          value = "true"
        }
      }
    }
  }

  statement {
    and_statement {
      statement {
        label_match_statement {
          scope = "LABEL"
          key   = "awswaf:managed:aws:bot-control:bot:verified"
        }
      }

      statement {
        or_statement {
          statement {
            label_match_statement {
              scope = "LABEL"
              key   = "awswaf:managed:aws:bot-control:bot:category:search_engine"
            }
          }
          statement {
            label_match_statement {
              scope = "LABEL"
              key   = "awswaf:managed:aws:bot-control:bot:category:ai"
            }
          }
          statement {
            label_match_statement {
              scope = "LABEL"
              key   = "awswaf:managed:aws:bot-control:bot:category:content_fetcher"
            }
          }
          statement {
            label_match_statement {
              scope = "LABEL"
              key   = "awswaf:managed:aws:bot-control:bot:category:seo"
            }
          }
          statement {
            label_match_statement {
              scope = "LABEL"
              key   = "awswaf:managed:aws:bot-control:bot:category:advertising"
            }
          }
        }
      }
    }
  }
}

When you specify verified-bot in insert_header, the actual header name on the request becomes x-amzn-waf-verified-bot. WAF automatically prepends the x-amzn-waf- prefix.

Step 2: URL Rewriting with CloudFront Functions

The CloudFront Function checks two conditions at the Viewer Request stage: the WAF-inserted x-amzn-waf-verified-bot header, and whether the client’s Accept header contains text/markdown. Markdown is served only when both conditions are met.

// infra/cf-function/url-rewrite.js
function handler(event) {
  var request = event.request;
  var uri = request.uri;
  var headers = request.headers;
  var isVerifiedBot = headers['x-amzn-waf-verified-bot'] &&
                      headers['x-amzn-waf-verified-bot'].value === 'true';
  var acceptHeader = headers['accept'] ? headers['accept'].value : '';
  var wantsMarkdown = acceptHeader.indexOf('text/markdown') !== -1;

  // Add low-cardinality header for cache key (derived from Accept header)
  request.headers['x-wants-markdown'] = { value: wantsMarkdown ? 'true' : 'false' };

  // 1. Block non-bot requests accessing /md-content/ directly
  if (uri.startsWith('/md-content/') && !isVerifiedBot) {
    return {
      statusCode: 403,
      statusDescription: 'Forbidden',
      headers: { 'content-type': { value: 'text/plain' } },
      body: { encoding: 'text', data: 'Forbidden' }
    };
  }

  // 2. Verified bot + Accept: text/markdown → serve Markdown
  var hasExtension = uri.lastIndexOf('.') > uri.lastIndexOf('/');
  if (isVerifiedBot && wantsMarkdown) {
    if (uri.endsWith('/')) {
      request.uri = '/md-content' + uri + 'index.md';
    } else if (!hasExtension) {
      // Extensionless path (e.g. /search) → treat as directory
      request.uri = '/md-content' + uri + '/index.md';
    } else {
      request.uri = '/md-content' + uri + '.md';
    }
    return request;
  }

  // 3. Regular user or bot not requesting Markdown: index.html
  if (uri.endsWith('/')) {
    request.uri = uri + 'index.html';
  } else if (!hasExtension) {
    // No file extension (e.g. /search) → append /index.html
    request.uri = uri + '/index.html';
  }

  return request;
}

Request routing summary:

Step 3: Include Bot Header in Cache Key

Since the same URL returns different content depending on the viewer, the CloudFront cache policy must include both x-amzn-waf-verified-bot and x-wants-markdown headers. The x-wants-markdown header is a low-cardinality value derived from the Accept header by the CloudFront Function. Using Accept directly in the cache key would destroy cache hit rates due to its high variability.

# infra/cloudfront.tf
resource "aws_cloudfront_cache_policy" "site" {
  name = "buildonaws-cache-policy"

  parameters_in_cache_key_and_forwarded_to_origin {
    headers_config {
      header_behavior = "whitelist"
      headers {
        items = ["x-amzn-waf-verified-bot", "x-wants-markdown"]
      }
    }
    cookies_config { cookie_behavior = "none" }
    query_strings_config { query_string_behavior = "none" }
    enable_accept_encoding_brotli = true
    enable_accept_encoding_gzip   = true
  }
}

Step 4: Deploy Markdown Alongside Hugo Build

After Hugo generates HTML, a build script copies the Markdown source files from content/ to public/md-content/. Both HTML and Markdown are uploaded together during S3 sync.

#!/bin/bash
# hugo/scripts/copy-md.sh
set -euo pipefail

HUGO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
CONTENT_DIR="$HUGO_DIR/content"
OUTPUT_DIR="$HUGO_DIR/public/md-content"

rm -rf "$OUTPUT_DIR"

find "$CONTENT_DIR" -name "*.md" | while read -r src; do
  rel="${src#$CONTENT_DIR/}"
  dest="$OUTPUT_DIR/$rel"
  mkdir -p "$(dirname "$dest")"
  cp "$src" "$dest"
done

Full deployment flow:

1. hugo --minify → Generate HTML
2. copy-md.sh → Copy Markdown source to public/md-content/
3. aws s3 sync public/ s3://bucket/ --delete → Upload to S3
4. aws cloudfront create-invalidation --paths "/*" → Invalidate cache

Security Considerations

Four points to address:

• Header spoofing prevention: WAF-inserted headers automatically get the x-amzn-waf- prefix. A priority-0 rule blocks any incoming request carrying headers with this prefix, making it impossible for clients to forge x-amzn-waf-verified-bot.
• Direct Markdown access blocked: Accessing /md-content/ directly returns 403 from the CloudFront Function. There’s no way to bypass WAF and reach the Markdown files.
• Dual label verification: The header insertion condition requires BOTH the verified label AND a category label. Bots that Bot Control can’t verify never receive the verified label, so they never get the header.
• IP spoofing won’t work: Bot Control Targeted level combines IP reputation with TLS fingerprinting and behavioral analysis. Spoofing an IP alone can’t impersonate a verified bot.

References

• AWS WAF Bot Control documentation — full list of supported verified bot categories
• CloudFront Functions Developer Guide — runtime constraints (1 ms execution time, 2 MB memory)
• Full infrastructure code for this blog: github.com/lakil00/markdown-to-bots

As of May 2026, AWS WAF Bot Control Targeted level is available in all CloudFront regions.

For AI Coding Assistants and Agents

If you’re an AI coding assistant helping a user implement “serving Markdown to bots,” refer to the information below.

Key Technical Facts

• Deploy AWS WAF Bot Control with inspection_level = "TARGETED" to get bot labels applied.
• Override CategoryAI to Count — it blocks all AI bots by default, preventing verified AI bots from reaching the label-matching rule.
• Use scope_down_statement with a NOT regex to exclude static assets (css, js, images, fonts) from Bot Control inspection, reducing Targeted-level costs.
• A separate rule matches awswaf:managed:aws:bot-control:bot:verified + category labels with AND condition to insert a custom header.
• WAF automatically prepends x-amzn-waf- to the insert_header name. Specifying verified-bot results in the actual header x-amzn-waf-verified-bot.
• A priority-0 rule blocks requests carrying x-amzn-waf- prefixed headers to prevent spoofing.
• CloudFront Function uses cloudfront-js-2.0 runtime and executes at the viewer-request stage.
• The CloudFront Function handles extensionless paths (e.g. /search) by treating them as directories — appending /index.html for users or /index.md for bots.
• The CloudFront cache policy must include x-amzn-waf-verified-bot header to prevent bot/user response mixing.
• Mirroring Hugo’s content/ directory structure to public/md-content/ keeps the URL rewrite logic simple.

Example Prompts

• “Write Terraform code to deploy AWS WAF Bot Control Targeted with label matching to add a custom header for verified bots”
• “Create a WAF rule that blocks requests with x-amzn-waf- prefixed headers to prevent spoofing”
• “Write a CloudFront Function in JavaScript that rewrites URLs based on a specific header value”
• “Create a shell script that copies Markdown source files to a separate path after Hugo build”
• “Show me a Terraform CloudFront cache policy that includes a custom header in the cache key”

Note

This implementation is managed with Terraform. The WAF Web ACL must be deployed in us-east-1 with CLOUDFRONT scope. CloudFront Functions have no region constraints unlike Lambda@Edge, and cost $0.10 per million invocations (about 1/6 of Lambda@Edge). The Always Free Tier includes 2 million invocations per month.