> ## Documentation Index
> Fetch the complete documentation index at: https://connect-docs.supertab.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Bot Events on Fastly VCL

> Measure and classify bot traffic from a Fastly VCL service.

If you run a Fastly **VCL (Delivery)** service and don't want to stand up a Compute service, you can still see who is crawling your site. A VCL snippet builds one JSON line per request and a log-streaming endpoint ships it to Supertab, where it is classified the same way as events from the SDK.

The snippet runs in `vcl_deliver`, after the response has already gone to the visitor. It makes no network calls, adds no origin fetches, and does not affect caching.

<Note>
  This gives you analytics only — **no CAP enforcement**. Requests are observed and labelled, never blocked or challenged. Acting on a license needs the SDK in a Compute service; see [Connect on Fastly](/reference/fastly/connect-on-fastly). A VCL service can also serve your RSL license without the SDK — see [Serving the license on VCL](/reference/fastly/connect-on-fastly#serving-the-license-on-vcl).
</Note>

## What gets recorded

Per request: timestamp, a Fastly request id, client IP, user agent, path, method, `Accept-Language`, host, response status, country, network (ASN and operator), the `Sec-Fetch-*` headers, HTTP Message Signature headers when present, which standard headers were sent, whether a cookie was sent, and three measurements of the query string.

Deliberately **not** recorded — meaning never written to the log line Supertab receives. Your traffic is served exactly as before, so anything below still reaches your own origin as part of the normal request:

| Not recorded                    | What is kept instead                                                                                                                                                                                   |
| :------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cookie values**               | Whether a `Cookie` header was present — a true/false. The contents are never read.                                                                                                                     |
| **The query string**            | Its length, its parameter count, and whether it matches known exploit patterns. The snippet reads the query, derives those three numbers in your POP, and never writes the characters to the log line. |
| **The `Referer` value**         | Only that a referer was sent, as a name in the header list. A referer URL can carry another site's query string.                                                                                       |
| **Any other header value**      | Header *names* only. `Authorization`, for example, is recorded as present, never as its value.                                                                                                         |
| **Request and response bodies** | Nothing.                                                                                                                                                                                               |

## Before you begin

Authentication uses Fastly's IAM-role access method: your Fastly account writes directly into
a bucket provisioned for you, and Fastly proves which account it is writing for. **There are
no credentials** — nothing is sent to you that could leak, nothing to store in Fastly, nothing
to rotate.

That works because the bucket is bound to your Fastly account, so setup starts with one value
from you:

1. **Send Supertab your Fastly customer ID** — in the Fastly web interface under
   **Account** → **Account info**. If your organization has more than one Fastly account, it
   must be the account **this service is listed under** — an ID from a different account fails
   silently later (see troubleshooting).
2. Supertab provisions your bucket and sends back the two values below.

You then need permission to edit and activate the service, and three values:

| Value            | Where to get it                                                   |
| :--------------- | :---------------------------------------------------------------- |
| **Website URN**  | **View Website Details** in your dashboard, next to your API keys |
| **IAM role ARN** | Sent to you by Supertab                                           |
| **Bucket name**  | Sent to you by Supertab                                           |

## Set it up

<Steps>
  <Step title="Clone the active version">
    Open the service, click **Edit configuration**, and clone the active version. Everything below happens on that draft and goes live only when you activate it.
  </Step>

  <Step title="Create the VCL snippet">
    Go to **VCL Snippets** → **Create snippet**:

    | Setting        | Value                   |
    | :------------- | :---------------------- |
    | **Name**       | `bot_event`             |
    | **Type**       | `Regular`               |
    | **Subroutine** | `deliver (vcl_deliver)` |
    | **Priority**   | `100`                   |

    <Warning>
      The subroutine must be `deliver`. The snippet reads the response status, which does not exist in `recv` — the default — so choosing `recv` fails to compile.
    </Warning>
  </Step>

  <Step title="Paste the snippet">
    Paste the code below into the snippet's VCL field.

    <Warning>
      **Replace `YOUR_WEBSITE_URN` with your Website URN.** It appears once, marked by a `====` banner near the top of the snippet. Keep the `%22` quotes around it.

      Paste the **whole** value including the `urn:` prefix — the placeholder is the entire URN, not just the id:

      `urn:stc:merchant:system:3f9a1c72-8b04-4e15-9c7d-2a6f0e4b1d83`

      Getting this wrong raises no error. The snippet compiles, log files are written, and your dashboard stays empty.
    </Warning>

    ```vcl theme={null}
    # --- Supertab bot-event logging -----------------------------------------------
    # Builds one JSON line describing this request into a request header,
    # X-Bot-Event; the "bot_events" log endpoint writes out that header and nothing
    # else. It runs after the response has gone to the visitor, makes no network
    # calls, and does not affect caching.
    #
    # It must run in "deliver" — it reads the response status, which does not exist
    # earlier in the request lifecycle.
    #
    # VCL strings cannot hold a literal double quote, so every quote the JSON needs
    # is written as %22. Long strings ({"..."}) appear only where a % must pass
    # through untouched. One statement per line — VCL cannot continue one.
    #
    # If you use shielding, this VCL runs at both the edge and the shield POP. The
    # check below builds the line only at the edge, so requests are not counted
    # twice.

    # Never log a value that came from the client: the log endpoint runs on the
    # shield leg too, where this header still holds whatever the caller sent.
    unset req.http.X-Bot-Event;

    if (fastly.ff.visits_this_service == 0) {
      declare local var.line   STRING;
      declare local var.ip     STRING;
      declare local var.hdrs   STRING;
      declare local var.qs     STRING;
      declare local var.pcount INTEGER;

      # Record the client IP in one consistent form: IPv4 written as IPv6-mapped
      # (::ffff:1.2.3.4), IPv6 as-is, "::" when there is no address.
      set var.ip = client.ip;
      if (var.ip ~ {"^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$"}) {
        set var.ip = "::ffff:" var.ip;
      }
      if (var.ip == "") { set var.ip = "::"; }

      # Which standard headers the visitor sent — NAMES ONLY, never values. VCL
      # cannot list headers, so each is checked individually. The list is broad on
      # purpose: a header not checked here is never recorded. Which headers a client
      # sends is one of the strongest tells of a real browser versus software
      # imitating one.
      set var.hdrs = "";
      if (req.http.Accept)                    { set var.hdrs = var.hdrs "%22accept%22,"; }
      if (req.http.Accept-Encoding)           { set var.hdrs = var.hdrs "%22accept-encoding%22,"; }
      if (req.http.Accept-Language)           { set var.hdrs = var.hdrs "%22accept-language%22,"; }
      if (req.http.Authorization)             { set var.hdrs = var.hdrs "%22authorization%22,"; }
      if (req.http.Cache-Control)             { set var.hdrs = var.hdrs "%22cache-control%22,"; }
      if (req.http.Cookie)                    { set var.hdrs = var.hdrs "%22cookie%22,"; }
      if (req.http.DNT)                       { set var.hdrs = var.hdrs "%22dnt%22,"; }
      if (req.http.From)                      { set var.hdrs = var.hdrs "%22from%22,"; }
      if (req.http.If-Modified-Since)         { set var.hdrs = var.hdrs "%22if-modified-since%22,"; }
      if (req.http.If-None-Match)             { set var.hdrs = var.hdrs "%22if-none-match%22,"; }
      if (req.http.Origin)                    { set var.hdrs = var.hdrs "%22origin%22,"; }
      if (req.http.Pragma)                    { set var.hdrs = var.hdrs "%22pragma%22,"; }
      if (req.http.Priority)                  { set var.hdrs = var.hdrs "%22priority%22,"; }
      if (req.http.Range)                     { set var.hdrs = var.hdrs "%22range%22,"; }
      if (req.http.Referer)                   { set var.hdrs = var.hdrs "%22referer%22,"; }
      if (req.http.Sec-CH-UA)                 { set var.hdrs = var.hdrs "%22sec-ch-ua%22,"; }
      if (req.http.Sec-CH-UA-Mobile)          { set var.hdrs = var.hdrs "%22sec-ch-ua-mobile%22,"; }
      if (req.http.Sec-CH-UA-Platform)        { set var.hdrs = var.hdrs "%22sec-ch-ua-platform%22,"; }
      if (req.http.Sec-Fetch-Dest)            { set var.hdrs = var.hdrs "%22sec-fetch-dest%22,"; }
      if (req.http.Sec-Fetch-Mode)            { set var.hdrs = var.hdrs "%22sec-fetch-mode%22,"; }
      if (req.http.Sec-Fetch-Site)            { set var.hdrs = var.hdrs "%22sec-fetch-site%22,"; }
      if (req.http.Sec-Fetch-User)            { set var.hdrs = var.hdrs "%22sec-fetch-user%22,"; }
      if (req.http.Signature)                 { set var.hdrs = var.hdrs "%22signature%22,"; }
      if (req.http.Signature-Agent)           { set var.hdrs = var.hdrs "%22signature-agent%22,"; }
      if (req.http.Signature-Input)           { set var.hdrs = var.hdrs "%22signature-input%22,"; }
      if (req.http.TE)                        { set var.hdrs = var.hdrs "%22te%22,"; }
      if (req.http.Upgrade-Insecure-Requests) { set var.hdrs = var.hdrs "%22upgrade-insecure-requests%22,"; }
      if (req.http.User-Agent)                { set var.hdrs = var.hdrs "%22user-agent%22,"; }
      if (req.http.X-Requested-With)          { set var.hdrs = var.hdrs "%22x-requested-with%22,"; }

      # ===========================================================================
      #  YOUR_WEBSITE_URN below is the only edit this snippet needs. Replace it
      #  with your Website URN — the whole value, starting "urn:" — keeping the
      #  %22 quotes around it.
      # ===========================================================================
      set var.line = "{%22merchant_system_urn%22:%22YOUR_WEBSITE_URN%22";
      set var.line = var.line ",%22schema_version%22:3";
      set var.line = var.line ",%22source_cdn%22:%22fastly_vcl%22";
      set var.line = var.line ",%22status_source%22:%22observed%22";
      set var.line = var.line ",%22client_ip_source%22:%22cdn_declared%22";

      # REQUIRED — do not remove these four.
      # They describe a licence-enforcement decision. This configuration makes no
      # such decision (it only observes), so they are fixed values saying exactly
      # that: no token was checked, nothing was blocked, enforcement is off.
      #
      # They look optional and are not. A record that leaves any of them out is
      # rejected in full and discarded — you would see traffic being logged and no
      # data ever arriving.
      set var.line = var.line ",%22has_token%22:false";
      set var.line = var.line ",%22token_outcome%22:%22not_validated%22";
      set var.line = var.line ",%22final_action%22:%22allow%22";
      set var.line = var.line ",%22enforcement_mode%22:%22disabled%22";

      # The format is a long string so that % reaches strftime untouched.
      set var.line = var.line ",%22timestamp%22:%22" strftime({"%Y-%m-%dT%H:%M:%S"}, time.start);
      set var.line = var.line "." time.start.msec_frac "Z%22";

      # Text fields. These are always present, and empty when the header was absent.
      set var.line = var.line ",%22request_id%22:%22" req.xid "%22";
      set var.line = var.line ",%22client_ip%22:%22" var.ip "%22";
      set var.line = var.line ",%22user_agent%22:%22" json.escape(req.http.User-Agent) "%22";
      set var.line = var.line ",%22path%22:%22" json.escape(req.url.path) "%22";
      set var.line = var.line ",%22method%22:%22" req.method "%22";
      # The Referer value is deliberately not recorded — a referer URL can carry
      # another site's query string. Whether one was sent is still noted, as a name
      # in the header list above. The field itself must stay present and empty.
      set var.line = var.line ",%22referer%22:%22%22";
      set var.line = var.line ",%22accept_language%22:%22" json.escape(req.http.Accept-Language) "%22";
      set var.line = var.line ",%22host%22:%22" json.escape(req.http.Host) "%22";

      # The status actually served to the visitor. Available because this runs in
      # deliver, after the response exists.
      set var.line = var.line ",%22status_code%22:" resp.status;

      # --- Optional fields.
      # When a value is not available these MUST be null, never an empty string. An
      # empty string would be read as a real answer rather than "not known", which
      # silently changes how the request is classified.
      if (client.geo.country_code && client.geo.country_code != "--") {
        set var.line = var.line ",%22request_country%22:%22" client.geo.country_code "%22";
      } else {
        set var.line = var.line ",%22request_country%22:null";
      }

      if (client.as.number > 0) {
        set var.line = var.line ",%22request_asn%22:" client.as.number;
      } else {
        set var.line = var.line ",%22request_asn%22:null";
      }

      if (client.as.name && client.as.name != "unknown") {
        set var.line = var.line ",%22as_organization%22:%22" json.escape(client.as.name) "%22";
      } else {
        set var.line = var.line ",%22as_organization%22:null";
      }

      if (req.http.Sec-Fetch-Mode) {
        set var.line = var.line ",%22sec_fetch_mode%22:%22" json.escape(req.http.Sec-Fetch-Mode) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_mode%22:null";
      }

      if (req.http.Sec-Fetch-Site) {
        set var.line = var.line ",%22sec_fetch_site%22:%22" json.escape(req.http.Sec-Fetch-Site) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_site%22:null";
      }

      if (req.http.Sec-Fetch-Dest) {
        set var.line = var.line ",%22sec_fetch_dest%22:%22" json.escape(req.http.Sec-Fetch-Dest) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_dest%22:null";
      }

      if (req.http.Sec-Fetch-User) {
        set var.line = var.line ",%22sec_fetch_user%22:%22" json.escape(req.http.Sec-Fetch-User) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_user%22:null";
      }

      # HTTP Message Signature headers (RFC 9421), recorded exactly as sent. Only
      # well-behaved agents that cryptographically sign their requests send these;
      # they are how such an agent proves it is who it claims to be.
      if (req.http.Signature-Agent) {
        set var.line = var.line ",%22signature_agent%22:%22" json.escape(req.http.Signature-Agent) "%22";
      } else {
        set var.line = var.line ",%22signature_agent%22:null";
      }

      if (req.http.Signature-Input) {
        set var.line = var.line ",%22signature_input%22:%22" json.escape(req.http.Signature-Input) "%22";
      } else {
        set var.line = var.line ",%22signature_input%22:null";
      }

      if (req.http.Signature) {
        set var.line = var.line ",%22signature%22:%22" json.escape(req.http.Signature) "%22";
      } else {
        set var.line = var.line ",%22signature%22:null";
      }

      if (req.http.Cookie) {
        set var.line = var.line ",%22has_cookies%22:true";
      } else {
        set var.line = var.line ",%22has_cookies%22:false";
      }

      set var.line = var.line ",%22header_names%22:[" regsub(var.hdrs, ",$", "") "]";

      # Query string: measurements only. The query string itself is never recorded —
      # only how long it was, how many parameters it had, and whether it looks like
      # an attack. The characters are discarded here and never leave your POP.
      set var.qs = req.url.qs;
      if (var.qs == "") {
        set var.line = var.line ",%22query_length%22:0";
        set var.line = var.line ",%22query_param_count%22:0";
        set var.line = var.line ",%22query_suspicious%22:false";
      } else {
        # `+` is concatenation in VCL, not addition — arithmetic needs the `+=` form.
        set var.pcount = std.strlen(regsuball(var.qs, "[^&]", ""));
        set var.pcount += 1;
        set var.line = var.line ",%22query_length%22:" std.strlen(var.qs);
        set var.line = var.line ",%22query_param_count%22:" var.pcount;
        # Known exploit patterns, checked against the raw and URL-decoded query.
        if (var.qs ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"} || urldecode(var.qs) ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"}) {
          set var.line = var.line ",%22query_suspicious%22:true";
        } else {
          set var.line = var.line ",%22query_suspicious%22:false";
        }
      }

      set var.line = var.line "}";

      # The log endpoint reads this request header. It is set in deliver, after the
      # origin fetch has already happened, so it is never sent to your origin.
      set req.http.X-Bot-Event = var.line;

    }
    ```
  </Step>

  <Step title="Create the logging endpoint">
    Go to **Resources** → **Log streaming** → **Create endpoint** → **Amazon S3**:

    | Setting           | Value                           |
    | :---------------- | :------------------------------ |
    | **Name**          | `bot_events`                    |
    | **Placement**     | `Format Version Default`        |
    | **Log format**    | `%{req.http.X-Bot-Event}V`      |
    | **Access method** | `IAM role`                      |
    | **IAM role ARN**  | provided by Supertab            |
    | **Bucket name**   | provided by Supertab            |
    | **Path**          | `bot-events/`                   |
    | **Domain**        | `s3.eu-central-1.amazonaws.com` |
    | **Period**        | `60`                            |

    Under **Advanced options**, set **Log line format** to `Blank` and leave compression off.

    Four of these are easy to get wrong, and none of them raise an error:

    * **Log format** replaces the prefilled JSON template entirely — delete it. The snippet already builds the JSON; this field only emits it.
    * **Log line format** must be `Blank`. The `Classic` default prepends a syslog header and corrupts every line.
    * **Compression** must be off. Supertab reads the `*.log` objects as plain newline-delimited JSON.
    * **Path** must be exactly `bot-events/`, trailing slash included — not a subfolder, not your company name.

    Everything not listed here — **Timestamp format** included — stays at its default.

    <Note>
      `Period` `60` is the recommended steady-state value: one file per minute per cache node, and your dashboard stays about a minute behind real time. A longer period only delays your own data.
    </Note>
  </Step>

  <Step title="Attach a condition to the endpoint">
    Still on the `bot_events` endpoint, choose **Attach a condition** → **Create a new response condition**:

    | Setting      | Value                  |
    | :----------- | :--------------------- |
    | **Name**     | `bot-event-present`    |
    | **Apply if** | `req.http.X-Bot-Event` |

    This makes the endpoint log only when the snippet actually built an event. Without it, legs where the snippet correctly skips — most commonly the **shield leg on a shielded service** — log a literal `(null)` line, and one such line makes the entire file unparseable on our side: every valid event in it is silently lost, with no error anywhere.
  </Step>

  <Step title="Activate">
    Activate the version. The snippet, the condition, and the logging endpoint go live together.
  </Step>
</Steps>

## Confirm it's working

Send some traffic, then allow **15 minutes** before checking. Logging configuration takes a few minutes to reach every POP, Fastly writes one file per `Period`, and it buffers per cache node — so events arrive as a trickle rather than all at once. Requests served before the log endpoint reaches a POP are lost rather than delayed.

Then open your dashboard. Crawlers that hit your site in that window appear there, classified by operator. If it's still empty, work through the table below.

## Troubleshooting

Misconfigurations here fail quietly rather than erroring, so work down the list in order.

| Symptom                                                          | Check                                                                                                                                                                                                                                                                                                                                                                               |
| :--------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nothing after 15 minutes                                         | Was the version **activated**? A cloned draft serves nothing.                                                                                                                                                                                                                                                                                                                       |
| Still nothing                                                    | If your service uses custom VCL, does `main.vcl` still contain the `#FASTLY deliver` and `#FASTLY log` macros? Fastly inserts snippets and the log statement at those markers.                                                                                                                                                                                                      |
| Still nothing                                                    | **Log line format** `Blank`, compression off, **Path** exactly `bot-events/`, **Bucket** and **IAM role ARN** exactly as Supertab sent them. Fastly validates none of these.                                                                                                                                                                                                        |
| Still nothing                                                    | The service must be **listed under the Fastly account whose customer ID you gave Supertab**. A different account — even one you also own — is refused by AWS inside Fastly's infrastructure, so nothing is delivered and no error appears anywhere. Confirm which account owns the service, and if it isn't the one you sent, send the right ID and Supertab will re-bind the role. |
| Files are written but your dashboard is empty                    | The URN in the snippet — it must be your full Website URN, with no duplicated `urn:stc:merchant:system:` prefix.                                                                                                                                                                                                                                                                    |
| Part of the data is missing, and your service uses **shielding** | The **response condition** on the endpoint — see the setup step above. Without it, shield legs write `(null)` lines that make whole files unparseable. Adding it stops the loss immediately; tell us and we'll recover what the broken files still hold.                                                                                                                            |
| Data arrives but looks wrong                                     | Send us one example request and what you expected. Don't edit the snippet's field values to compensate — several look optional and are not, and removing one causes the whole record to be discarded.                                                                                                                                                                               |

<Note>
  Every request is logged, including cache hits. With shielding enabled the line is built only at the edge POP — that's what the snippet's `visits_this_service` guard is for — so requests are not counted twice, and the response condition keeps those shield legs from logging at all.
</Note>

***

## Related Docs

<CardGroup cols={2}>
  <Card title="Connect on Fastly" icon="bolt" href="/reference/fastly/connect-on-fastly">
    Serve your RSL license and enforce CAP on Fastly, with Compute or by chaining VCL to a validator.
  </Card>

  <Card title="Deploy at the Edge" icon="shield" href="/guides/deploy-cdn">
    CDN-agnostic guide covering RSL serving, CAP enforcement, and robots.txt.
  </Card>
</CardGroup>