NLWeb Explained: Natural Language Endpoints, MCP and How to Secure Them

NLWeb is an open project, announced by Microsoft on 19 May 2025, that turns a website's existing structured data (Schema.org markup, RSS feeds and similar lists of items) into a natural language endpoint called /ask, and makes every instance an MCP server at /mcp as well. To secure it, treat it as a public LLM application: scope what you index, authenticate /mcp, rate limit /ask, and patch its Python dependencies.

NLWeb is two things at once: a small protocol for asking a website a question and getting Schema.org JSON back, and a Python reference implementation that loads your content into a vector database and runs a pipeline of LLM calls over it. The protocol is simple. The implementation is a web server with API keys, a vector store, outbound calls to model providers, optional remote backends and an MCP endpoint, and each part carries a security decision.

This post goes one level deeper than our explainers on llms.txt, MCP messages, transports and authorization and A2A versus MCP on the wire. If you are new to MCP, start with what the Model Context Protocol is.

NLWeb facts come only from its official repository and docs, read on 24 October 2026 on the main branch of nlweb-ai/NLWeb (the original microsoft/NLWeb URL redirects there), plus Microsoft's announcement. MCP facts come from modelcontextprotocol.io. Defaults change between commits, so we name the file each detail comes from; check your own checkout.

Key takeaways

• A protocol plus a reference implementation. Ask a site a question in natural language, get Schema.org JSON back. The MIT licensed Python code indexes RSS, JSON lines and CSV into a vector store.
• Two endpoints, same arguments: /ask for web clients and /mcp for MCP clients, which offers the tools ask, list_sites and, when enabled, who, and reports MCP version 2024-11-05.
• Development defaults. The shipped config sets mode: development, binds to 0.0.0.0 and enables permissive CORS; in development mode the auth middleware accepts requests without a token.
• /ask is public in the shipped auth middleware, with a comment saying this "can be changed."
• One query can cost a lot. The project's docs say a single query "might involve over 50 LLM API calls," so rate limiting is a cost control too.
• Your content becomes the prompt. Indexed text can carry prompt injection, and anything indexed can be retrieved by anyone who reaches the endpoint.
• Patch it like any Python service. Several requirements.txt files, no published advisories, and release notes that record security fixes.

What is NLWeb, and who announced it?

Microsoft announced NLWeb in a company news feature dated 19 May 2025, "Introducing NLWeb: Bringing conversational interfaces directly to the web." It credits R.V. Guha, who had joined Microsoft as a corporate vice president and Technical Fellow, and describes him as the creator of RSS, RDF and Schema.org. It pitches NLWeb as the "fastest and easiest way to effectively turn your website into an AI app," and suggests it could play a role for the agentic web similar to the one HTML played for documents. The feature listed a dozen early publishing and ecosystem collaborators; we do not repeat their names.

The GitHub repository was created on 28 April 2025 and is MIT licensed. It now lives under the nlweb-ai organisation, and the old microsoft/NLWeb address redirects to it. The README still points support questions to a Microsoft address, and SECURITY.md is Microsoft's standard policy, which routes vulnerability reports to the Microsoft Security Response Center rather than public issues.

What problem does it try to solve?

The README starts from data sites already publish. It argues that Schema.org and semi-structured formats such as RSS, which it says over 100 million websites use, have become a semantic layer for the web, and that this layer is enough to build a conversational interface. A recipe site already marks up recipes; a podcast already has a feed; a shop already describes products. Load that into a vector store, put an LLM pipeline in front, and the site can answer "vegetarian mains under 30 minutes that use leftover rice" with real items from its own catalogue.

The second half of the pitch is agents. The README says the same natural language API serves humans and AI agents through native MCP support, and it compares NLWeb's relationship to MCP and A2A with the relationship between HTML and HTTP. The announcement says sites can opt in to make their content available to agents and other MCP participants.

Is it a standard or a product?

Neither, exactly. The README calls NLWeb a collection of open protocols and open source tools, and describes its code as proof of concept demonstrations of one possible approach rather than the definitive solution. It expects most production deployments to use their own interface, integrate NLWeb into their own application and connect it to live databases rather than duplicating content. It also notes that CI/CD pipelines are not yet included. Read that as a signal about how much hardening you own if you deploy the reference code as is.

How does NLWeb work?

At the highest level there are three stages: ingest, retrieve and rank, then respond. The project's "Life of a Chat Query" document describes the runtime part in detail, and the database load tool guide covers ingestion.

Ingest: from RSS and Schema.org to vectors

The loader is data_loading/db_load.py, run from the AskAgent/python folder. According to the database load tool guide, it accepts five input shapes:

1. Two tab separated columns per row: a URL and a JSON object.
2. One column of JSON per row, with the URL taken from the JSON.
3. A CSV file with headers.
4. An RSS or Atom feed.
5. A URL pointing to any of the above.

The JSON is expected to be Schema.org typed data, the same objects a site would embed in its pages as JSON-LD. For each item the loader does three things in order: load or scrape the data, compute embeddings with the provider set in config/config_embedding.yaml, and write the vectors into the database set as write_endpoint in config/config_retrieval.yaml. The guide notes that for a cloud vector service you need a key with write permission to the database. The command takes a source and a site name:

python -m data_loading.db_load <file-or-url> <site-name>

The site name matters later. It is the token that scopes a query to a subset of the store, and the token you use to delete a site's entries. The guide also documents options for loading a file of many feed URLs, changing the batch size (default 100) and writing to a different configured backend.

RSS is the easiest on-ramp because almost every publishing system emits it. The RSS 2.0 specification describes RSS as a web content syndication format and a dialect of XML, where each item carries elements such as title, link and description. That gives NLWeb a name, a URL and some text per item even when a site has no Schema.org markup.

Retrieve and rank: a pipeline of narrow LLM calls

When a query arrives, "Life of a Chat Query" describes this sequence:

1. The user submits the next query in the conversation.
2. Parallel LLM calls check relevance, decontextualise the query against history and decide whether anything should be remembered. A fast track skips ahead when nothing changes.
3. Based on a manifest in tools.xml, an LLM picks a tool (Search, Item Details, Ensemble, and Compare from the 23 June 2025 release) and extracts its parameters.
4. Results from the vector database are scored with further narrow LLM calls, which may also write a snippet. The top results above a threshold are kept with score, snippet and the stored object.
5. Optional post-processing summarises the list or tries to answer the question from it.

Two notes in that document matter for security and cost. First: "Processing a single query might involve over 50 LLM API calls." Second, because every listed result comes from the database, the document argues that listed results cannot be invented by the model, while noting that post-processing may weaken that guarantee. That helps against hallucination, and it is the same property that makes the index the thing to protect: whatever is in it can come out.

Respond: the /ask and /mcp endpoints

The REST API document says NLWeb supports two APIs, at /ask and /mcp, with the same arguments and most of the same behaviour. The /mcp endpoint returns answers in a format MCP clients can use and also supports the core MCP methods. The included implementation keeps no server side conversation state, so clients pass context back on each call.

The three modes map to three cost and risk levels. list returns the top matches. summarize adds an LLM written summary. generate is, in the doc's words, "much more like traditional RAG," with one or more LLM calls to answer the question. The response is JSON with a query_id and an array of results, each carrying url, name, site, score, an LLM generated description and schema_object, the stored item encoded as JSON.

What the MCP server actually offers

The MCP handler lives in AskAgent/python/webserver/mcp_wrapper.py and its routes in webserver/routes/mcp.py. In the code we read on 24 October 2026:

• POST and GET are both routed to /mcp and to /mcp/ followed by any path, with health checks at /mcp/health and /mcp/healthz.
• The handler answers initialize, notifications/initialized, tools/list and tools/call. A comment notes that the initialisation check was removed because "MCP clients might not send initialize first."
• tools/list returns ask (with query, an optional site array and generate_mode of list, generate or summarize), list_sites, and who when the who endpoint is enabled in config_nlweb.yaml (it is enabled in the shipped file).
• The server reports protocolVersion 2024-11-05 and the name nlweb-mcp-server.
• The route comment says MCP responses are regular JSON-RPC responses, not server-sent events.

The MCP specification describes servers offering resources, prompts and tools over JSON-RPC 2.0. NLWeb uses only tools, which keeps its surface small. The version gap matters: the 2025-11-25 Streamable HTTP transport requires an MCP-Protocol-Version header on subsequent requests, and the authorization specification has moved on since 2024, so a client that negotiates down to an older version may skip protections you expect.

The repository also contains A2A route and wrapper files, and the README says A2A support is coming. We did not test that path.

How does NLWeb relate to Schema.org, llms.txt, WebMCP, MCP and A2A?

These names get lumped together as making websites readable by AI. They answer different questions.

Schema.org is the vocabulary. Its about page describes a collaborative community activity "to create, maintain, and promote schemas for structured data on the Internet," founded by four search engine companies, with the W3C Schema.org Community Group as its main forum since April 2015. NLWeb uses Schema.org as the input it prefers to index and the output it returns. Good Recipe, Product, Event or PodcastEpisode markup leaves NLWeb much less to guess.

llms.txt is a file. The proposal by Jeremy Howard, first published on 3 September 2024 and modified on 10 August 2026 (now labelled v2), suggests a markdown file at /llms.txt or any path, with background, guidance and links to clean markdown versions of pages. It is static and cheap, and nothing runs on your side when an agent reads it.

WebMCP is a browser API proposal from the W3C Web Machine Learning Community Group. Its explainer describes pages exposing JavaScript functions, with natural language descriptions and schemas, as tools that browser agents can call. It contrasts this with "backend integrations" such as MCP, where an agent talks to a service's servers directly. WebMCP tools run in the page, in the user's session.

MCP is the protocol between AI applications and servers that provide context and tools. NLWeb is one such server, so every MCP security rule (authorization, Origin validation, token audience) applies to its /mcp endpoint.

A2A is a protocol for agents talking to other agents, which NLWeb says it will speak.

A compact way to hold it: Schema.org describes your items, RSS lists them, llms.txt points at your docs, NLWeb answers questions about your items, MCP is how an agent calls NLWeb, and WebMCP is how a page lends an agent its own buttons.

Comparison: NLWeb, llms.txt, WebMCP, a search API and a sitemap

The table is a decision aid, not a ranking. A docs site may need nothing beyond llms.txt and clean markdown. A catalogue site wanting conversational search for humans and a callable endpoint for agents is NLWeb's target. A web app whose value is in actions (book, pay, edit) is closer to WebMCP, because the action belongs in the user's session.

Which models, vector stores and hosting options does the project document?

All of this comes from the README, the hello world guide, the shipped config/ files and the release notes. Names here are the project's, and the list changes.

Operating systems. The README says Windows, macOS and Linux, and describes NLWeb as lightweight enough for data centre clusters or laptops.

LLM providers. The README lists OpenAI, DeepSeek, Gemini, Anthropic, Inception and Hugging Face. The shipped config_llm.yaml also has an Azure OpenAI endpoint, which is the preferred_endpoint by default, and the 29 July 2025 release notes add Ollama for local models. Each provider entry names a high and a low model; NLWeb uses the cheaper model for narrow calls. The model identifiers in the shipped config are examples that age quickly, so check each against your provider's current documentation before you deploy.

Embedding providers. Set in config_embedding.yaml. The hello world guide says the default embedding model for OpenAI is text-embedding-3-small.

Vector stores. The README lists Qdrant, Snowflake, Milvus, Azure AI Search, Postgres and two hosted search and RAG services; the 23 June 2025 release notes add OpenSearch and describe concurrent querying across several enabled backends with deduplication, with one write_endpoint. For a laptop, the hello world guide uses qdrant_local, a file based Qdrant store under ../data/db.

Hosting. The README documents running on a laptop and on Azure, with GCP and AWS marked coming soon. The repository includes Azure App Service config files and a deploy_azure_webapp.sh script, and the July 2025 notes mention Docker based installations.

Other modules. Besides AskAgent, the core query agent this post covers, the README lists AgentFinder, DataFinder, ModelRouter and NLWebScorer.

Two defaults in config_retrieval.yaml deserve attention before you run anything. Besides qdrant_local, the shipped file enables nlweb_west, an Azure AI Search backend that reads its endpoint and key from environment variables, and shopify, a backend described as "Shopify MCP Search" whose MCP endpoint is set dynamically per site. The hello world guide tells you to disable endpoints you have no credentials for. Do it deliberately: an enabled remote backend is a place your users' queries go.

The who tool is a similar case. config_nlweb.yaml enables the who endpoint by default, and config_webserver.yaml names a gateway host used to route queries to other sites. If you do not want your users' questions forwarded to a shared directory service, turn it off.

What are the risks of a natural language endpoint?

The OWASP Top 10 for LLM Applications 2025 is the clearest checklist for this, and NLWeb touches most of it. We map each risk to the NLWeb component where it lives.

Prompt injection carried in site content (LLM01)

NLWeb's ranking, snippet, summarise and generate steps all put your indexed items into LLM prompts. If any indexed text is written by someone other than you (product reviews, comments, user listings, forum posts, a third party RSS feed you aggregate), that person can write text aimed at the model rather than the reader. This is indirect prompt injection, and OWASP lists prompt injection as LLM01. The project's Responsible AI FAQ says the team tested "cross prompt injection attacks" with manual and semi-automated techniques, and also warns that the chosen LLM can still produce inappropriate content, so developers should add safety classifiers or filters suited to their context.

NLWeb's structural mitigation is mode=list: results are database items, and the LLM only scores and snippets them, though injection can still bias a score or snippet. In summarize and generate it can shape the answer itself. For /mcp, the bigger concern is downstream: an agent that reads NLWeb output and then acts with other tools can be steered by text from your index. Treat NLWeb output as untrusted input in every agent that consumes it.

Unpublished or private content leaking from the index (LLM02 and LLM08)

The index is a copy. It does not know about your CMS's draft state, embargo times, paywall rules or per-user permissions unless you build that in. If the feed or JSON you load includes drafts, member-only items, internal notes or unpublished prices, /ask will retrieve them for anyone. schema_object in every result is the stored item encoded as JSON, so fields you never render on a page (internal SKUs, cost prices, editor comments) are returned too if they are in the object. OWASP's LLM02 (sensitive information disclosure) and LLM08 (vector and embedding weaknesses) both describe this pattern.

The site parameter is a scoping token, not an access control. Anyone can pass any site name, and list_sites on the MCP side tells them what exists. If you need per-tenant separation, use separate indexes or separate instances, not site tokens on one shared store. Deletions need care as well: an item removed from your CMS stays in the vector store until you reload or run the loader's documented delete command for that site.

Cost abuse and the missing rate limit (LLM10)

OWASP's LLM10 is "unbounded consumption." The project's own note that one query can involve more than 50 LLM API calls is the number to plan around. A public /ask endpoint with generate mode, streaming and no per-client limit converts anonymous HTTP requests directly into model provider spend, and nothing in the request has to be malicious for that to hurt. The setup-oauth.md guide shows a rate limit setting under auth, aimed at brute force login attempts. The response headers guide shows an example rate_limits header string, but that header only tells clients about a limit; it does not enforce one. We did not find a documented, enforced per-client rate limit for /ask or /mcp in the docs we read, so plan to add one in front.

API key handling

A working instance holds LLM and embedding provider keys, vector store keys (with write permission where the loader runs), OAuth client secrets if you enable login, and a session signing secret. The .env.template file says it is "for demonstration purposes only and should not be used in production." Give the serving instance read-only vector store credentials, keep the write key where you run db_load, set spending caps on provider accounts, and keep .env out of images and repositories.

Unauthenticated MCP endpoints

This is where the shipped defaults matter most. In webserver/middleware/auth.py, /ask, /who, /sites and several static and health paths are listed as public, with a comment that public access to /ask is allowed "for now (can be changed)." /mcp is not on the public list. However, the same middleware lets requests with no token through when the server is in development mode, and the shipped config_webserver.yaml sets mode: development. A comment in that file adds that development mode lets query parameters override configuration, then says: "obviously, this cannot be allowed in production."

The shipped server config binds to 0.0.0.0 and enables CORS, and the CORS middleware we read returns a wildcard Access-Control-Allow-Origin with credentials allowed on every response. The MCP Streamable HTTP transport specification says servers "MUST validate the Origin header on all incoming connections," and adds that local servers should bind only to localhost and that servers should authenticate all connections. The MCP route handler we read does not check Origin itself. Put those together and an instance started with defaults on a laptop or a cloud VM is reachable from the network, open to any origin, and accepts MCP calls without a token.

The auth middleware also contains branches that recognise end to end and integration test token formats. That is normal in a reference implementation and wrong in production. Read that file in your version before you rely on it, and prefer putting your own authentication in front of the server.

The MCP authorization specification makes authorization optional, but an HTTP server that supports it acts as an OAuth 2.1 resource server, must check that tokens were issued for it as the audience, must answer invalid tokens with HTTP 401 and must not pass tokens through to downstream APIs. NLWeb's OAuth guide describes user login for its UI with session JWTs; the docs we read do not describe MCP authorization discovery. For authenticated agent access, put a gateway that implements MCP authorization in front, or use per-client keys on a private network.

Error detail and logging

When the MCP route we read hits an exception, it returns a JSON-RPC internal error whose data field holds the exception message, which can include paths, hostnames and config names. In production, log the detail and return a generic error. Logs go to ./logs/webserver.log by default; queries are user text and can contain personal data.

Keeping the project's own dependencies patched (LLM03)

NLWeb has a long dependency list across a root requirements.txt, an Azure specific file, AskAgent/python/requirements.txt and separate files for the other modules: a web framework, HTTP clients, JWT handling, provider SDKs, vector store clients and, in some modules, ML libraries. OWASP's LLM03 covers supply chain risk of exactly this shape.

The project's own history shows why this matters. A release note in docs/release_notes lists a fix for a "path traversal vulnerability in static file handler" (PR #231) and safer URL checking in streaming.js against domain spoofing (PR #236); the 29 July 2025 notes add a CodeQL workflow. When we queried the repository's GitHub security advisories on 24 October 2026, none were published. That does not mean no vulnerabilities; it means fixes arrive as commits and release notes, so track the commit you deployed and pull fixes yourself. The last push to main at the time of writing was 11 August 2026.

Risk summary

Step by step: run NLWeb locally, load an RSS feed, query it, connect MCP, then lock it down

These steps follow the project's hello world guide and REST API doc. The curl request shapes for /mcp follow the tool schema in mcp_wrapper.py. Run this on a development machine, not a server, until step 7.

Step 1: clone and install

The guide requires Python 3.10 or later. It still shows the old microsoft/NLWeb URL, which redirects.

git clone https://github.com/nlweb-ai/NLWeb
cd NLWeb
git rev-parse HEAD > DEPLOYED_COMMIT.txt   # record exactly what you run

python3 -m venv myenv
source myenv/bin/activate

cd AskAgent/python
pip install -r requirements.txt

Recording the commit is not in the guide. Add it anyway: without tags or published advisories, the commit hash is how you will know later whether a fix applies to you.

Step 2: configure providers and keys

cd ../../
cp .env.template .env
chmod 600 .env

Set only the keys you will use, for example OPENAI_API_KEY. Then edit three files in config/:

• config_llm.yaml: set preferred_endpoint to your provider (the default is azure_openai).
• config_embedding.yaml: set preferred_provider to match.
• config_retrieval.yaml: keep write_endpoint: qdrant_local, keep qdrant_local enabled, and set every other endpoint to enabled: false, including nlweb_west and shopify.

# config/config_retrieval.yaml (excerpt)
write_endpoint: qdrant_local
endpoints:
  nlweb_west:
    enabled: false
  qdrant_local:
    enabled: true
    database_path: "../data/db"
    index_name: nlweb_collection
    db_type: qdrant
  shopify:
    enabled: false

While you are in config/, set who_endpoint_enabled: false in config_nlweb.yaml unless you have a reason to use the shared who service, and set sites: to the site names you intend to serve.

Check connectivity with the project's script:

cd AskAgent/python
python testing/check_connectivity.py

Step 3: build a feed of published content only

Know what is in a feed before you load it. Build it from published items only, with only the fields you are happy to return. A quick look:

curl -s https://example.com/feed.xml -o feed.xml
grep -c "<item>" feed.xml
grep -o "<title>[^<]*</title>" feed.xml | head -20

If you export JSON instead, keep a field allow-list. With jq, for one Schema.org object per line:

jq -c '{"@type": .["@type"], "@id": .["@id"], url, name, description, datePublished}' items.jsonl > items.clean.jsonl

Step 4: load the feed

From AskAgent/python:

python -m data_loading.db_load https://example.com/feed.xml example-site

To remove that site's entries later, for example after you discover an item that should not be there:

python -m data_loading.db_load --only-delete delete-site example-site

Step 5: start the server and query /ask

python app-aiohttp.py

The hello world guide says the server listens on port 8000. Query it with streaming off so you get one JSON document:

curl -s "http://localhost:8000/ask?query=episodes%20about%20batteries&site=example-site&mode=list&streaming=false" | jq '.'

Look at the fields in each result, especially schema_object. This is exactly what any client on the network will see. Then try mode=summarize and compare the response time; that difference is the cost of the extra LLM calls.

Step 6: connect it as an MCP server

List the tools with a raw JSON-RPC call:

curl -s http://localhost:8000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq '.result.tools[].name'

Call ask:

curl -s http://localhost:8000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ask","arguments":{"query":"episodes about batteries","site":["example-site"],"generate_mode":"list"}}}' | jq '.'

If both calls succeed without any token, you have just confirmed the development mode behaviour described above. To use a graphical client, the MCP documentation describes the MCP Inspector, which ships as the @modelcontextprotocol/inspector package:

npx @modelcontextprotocol/inspector

In the Inspector, choose the HTTP transport and enter http://localhost:8000/mcp. For an AI assistant, add the URL as a remote MCP server following that client's own documentation. Remote connections generally need a reachable HTTPS URL, which is when the next steps stop being optional.

Step 7: switch to production mode and bind to localhost

Edit config/config_webserver.yaml:

port: 8000
mode: production
server:
  host: 127.0.0.1
  enable_cors: false
  max_connections: 100
  timeout: 30

Restart, and repeat the tools/list call from step 6 without a token. With mode: production the auth middleware should now answer /mcp with HTTP 401 and a WWW-Authenticate: Bearer header. /ask stays public because it is on the public list; if you want it closed, remove it from PUBLIC_ENDPOINTS in webserver/middleware/auth.py and carry that change as a patch you re-apply on every update.

Step 8: put authentication, an Origin check and rate limits in front

The cleanest pattern is a small gateway in front of NLWeb that does four things NLWeb does not do for you by default: authenticate each client, check Origin, enforce a per-client rate limit and cap the request size and mode. Any reverse proxy or API gateway you already run can do this. For illustration, here is a minimal gateway in Python using aiohttp, which NLWeb already depends on. It forwards to NLWeb on 127.0.0.1:8000 and listens on 127.0.0.1:8080, behind your TLS terminator.

# nlweb_gateway.py  (illustrative; run behind TLS, keep keys in a secret store)
import os, time, hmac
from aiohttp import web, ClientSession, ClientTimeout

UPSTREAM = "http://127.0.0.1:8000"
UPSTREAM_TOKEN = os.environ.get("NLWEB_UPSTREAM_TOKEN")  # NLWeb's own credential, not the client's
API_KEYS = {k.strip() for k in os.environ["NLWEB_GATEWAY_KEYS"].split(",") if k.strip()}
ALLOWED_ORIGINS = {"https://www.example.com"}
RATE = 20          # requests per window per key
WINDOW = 60        # seconds
MAX_BODY = 16 * 1024
ALLOWED_MODES = {"list"}  # keep generate off the public path

buckets = {}

def key_ok(presented: str) -> str | None:
    for k in API_KEYS:
        if hmac.compare_digest(presented, k):
            return k
    return None

def allow(key: str) -> bool:
    now = time.monotonic()
    start, count = buckets.get(key, (now, 0))
    if now - start > WINDOW:
        start, count = now, 0
    if count >= RATE:
        return False
    buckets[key] = (start, count + 1)
    return True

async def proxy(request: web.Request) -> web.StreamResponse:
    origin = request.headers.get("Origin")
    if origin is not None and origin not in ALLOWED_ORIGINS:
        return web.json_response({"error": "forbidden origin"}, status=403)

    auth = request.headers.get("Authorization", "")
    key = key_ok(auth[7:]) if auth.startswith("Bearer ") else None
    if key is None:
        return web.json_response({"error": "unauthorized"}, status=401,
                                 headers={"WWW-Authenticate": "Bearer"})
    if not allow(key):
        return web.json_response({"error": "rate limited"}, status=429,
                                 headers={"Retry-After": str(WINDOW)})

    if request.content_length and request.content_length > MAX_BODY:
        return web.json_response({"error": "too large"}, status=413)
    body = await request.read()
    if len(body) > MAX_BODY:
        return web.json_response({"error": "too large"}, status=413)

    mode = request.query.get("mode", "list")
    if mode not in ALLOWED_MODES:
        return web.json_response({"error": "mode not allowed"}, status=400)

    headers = {"Content-Type": request.headers.get("Content-Type", "application/json")}
    if UPSTREAM_TOKEN:
        headers["Authorization"] = f"Bearer {UPSTREAM_TOKEN}"
    async with ClientSession(timeout=ClientTimeout(total=30)) as s:
        async with s.request(request.method, UPSTREAM + request.path_qs,
                             data=body, headers=headers) as r:
            return web.Response(status=r.status, body=await r.read(),
                                content_type=r.content_type)

app = web.Application(client_max_size=MAX_BODY)
app.router.add_route("*", "/ask", proxy)
app.router.add_route("*", "/mcp", proxy)
web.run_app(app, host="127.0.0.1", port=8080)

Notes on the sketch:

• It never forwards the client's key. With NLWeb in production mode, /mcp needs a bearer token NLWeb itself accepts, so the gateway presents a separate service credential from NLWEB_UPSTREAM_TOKEN (a session token signed with the secret your NLWeb OAuth setup uses). That mirrors the MCP guidance against token passthrough.
• The in-memory bucket is per process. With several gateway instances, keep counters in a shared store.
• It only checks the mode query parameter. On /mcp the equivalent is generate_mode inside the JSON-RPC arguments; parse the body and apply the same rule there.
• For internet facing agent access, the MCP authorization specification (OAuth 2.1, protected resource metadata, audience-bound tokens) is the standard way to issue per-client tokens. A static key list suits a private network or a few known partners.

Finally, set spending limits on every LLM provider account the instance uses, and alert on spend per hour, not per month.

Step 9: verify

# No key: expect 401
curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:8080/ask?query=test"

# Wrong origin: expect 403
curl -s -o /dev/null -w '%{http_code}\n' -H 'Origin: https://attacker.invalid' \
  -H "Authorization: Bearer $KEY" "http://127.0.0.1:8080/ask?query=test"

# Burst past the limit: expect 429 after the 20th request
for i in $(seq 1 25); do
  curl -s -o /dev/null -w '%{http_code} ' -H "Authorization: Bearer $KEY" \
    "http://127.0.0.1:8080/ask?query=test&mode=list&streaming=false"
done; echo

# NLWeb itself must not answer from outside the host
curl -s -m 5 -o /dev/null -w '%{http_code}\n' "http://<server-public-ip>:8000/ask?query=test" || echo "not reachable (good)"

What should a site owner decide before exposing a natural language endpoint?

Most NLWeb risk is decided before anyone writes config. Write these decisions down.

Content scope. Which content types, fields and dates? Index only what is already public, from a feed or export built for this purpose, with a field allow-list. Exclude user generated content unless you accept a model reading it. Name who approves a new source.

Freshness and deletion. How fast must an unpublished or corrected item vanish from answers? The README recommends live databases over copies for freshness. If you load copies, schedule reloads and write a takedown procedure that uses the delete command.

Audience and authentication. Is /ask for anonymous visitors, logged-in users or partners? Is /mcp for any agent, known clients or internal tools? Each answer means a different setup: public with rate limits, OAuth login, or MCP authorization in a gateway.

Modes. list returns your items; generate writes prose. Many sites will offer list and summarize publicly and generate only to authenticated users, or not at all.

Logging and retention. Log query text, client key, model, token counts and cost per request; cost per request is what makes abuse visible. Queries can contain personal data, so set a retention period and rotation, and say so in your privacy notice. A data_retention response header tells clients about their side; it does not set yours.

Abuse handling. Decide thresholds for spend spikes, catalogue scraping through many small queries and looping agents, who gets paged, and whether the response is a block, a lower rate or a cheaper mode.

Third party flow. List which model providers see queries and content, which remote backends are enabled, and whether who is in use. Each is a data flow for your privacy review.

Patching. Name an owner for NLWeb updates. Without releases or advisories, review the commit log on a schedule and rebuild.

Timeline

Common mistakes

• Running with the shipped web server config on a reachable host. mode: development, host: 0.0.0.0 and permissive CORS are fine on a laptop behind a firewall and wrong anywhere else.
• Treating site as access control. It is a filter any caller can set. Separate audiences need separate indexes.
• Loading the CMS's full export. Drafts, internal fields and member-only items come straight back out in schema_object.
• Leaving remote backends and who enabled. Your users' queries go to services you did not choose.
• No spending cap, or the loader's write key on the serving instance. A looping agent becomes your invoice, and the server only needs read access.
• Assuming no advisories means no vulnerabilities. Fixes have shipped as commits and release notes; track the commit you run.
• Forgetting downstream agents. An agent that reads NLWeb output and then acts with other tools inherits any injection that reached your index.
• Forgetting deletes. Removing an item from your site does not remove it from the vector store.

FAQ

What is NLWeb in simple terms?

NLWeb is an open source way to let people and AI agents ask a website questions in plain language. It loads the site's structured data (Schema.org items, RSS feeds) into a vector database and answers through an /ask endpoint and an MCP endpoint, returning real items from the site as Schema.org JSON.

Who made NLWeb?

Microsoft announced it on 19 May 2025, crediting R.V. Guha, whom the announcement describes as the creator of RSS, RDF and Schema.org. The code is MIT licensed and now lives under the nlweb-ai GitHub organisation.

Is every NLWeb instance an MCP server?

Yes, according to the README and the announcement. In the code we read, the /mcp endpoint offers the tools ask, list_sites and, when enabled, who.

Does NLWeb replace llms.txt?

No. llms.txt is a static markdown file that helps agents read your documentation. NLWeb is a running service that answers questions over indexed items. A documentation site may only need llms.txt; a catalogue site wanting conversational search is NLWeb's use case. They can coexist.

Does NLWeb need authentication?

For anything reachable from the internet, yes, at least on /mcp, and rate limits on /ask. The shipped auth middleware makes /ask public and only enforces tokens on other paths in production mode, while the shipped config starts in development mode.

How much does an NLWeb query cost?

It depends on your models and mode. The project's documentation says one query can involve more than 50 LLM calls, most of them narrow calls suited to a cheaper model. Measure your own per-query cost with list and summarize before you open the endpoint.

Can NLWeb leak private content?

It can return anything you indexed, including fields you do not show on your pages, because each result includes the stored object. Index only published content with an explicit field allow-list.

Is NLWeb vulnerable to prompt injection?

Any system that puts third party text into LLM prompts can be. The project says it tested for cross prompt injection, and its list mode limits how much model output reaches users, but indexed text written by others can still influence scores, snippets and generated answers.

Where Safeguard fits

Safeguard does not implement NLWeb, does not sit in front of /ask, and does not detect prompt injection in your content. The overlap is narrower and practical. An NLWeb deployment is a Python service with several requirements.txt files and, usually, a container image, and that dependency tree is where most of its patchable risk lives. The Safeguard CLI can scan the checkout and the built container image, match components against vulnerability databases, apply reachability analysis so you see which vulnerable code paths the server actually calls, show which projects carry a given version, and gate builds with policies. Results land as JSON under ./output for your pipeline.

On the MCP side, Safeguard runs its own MCP server with OAuth and lets you choose which tools an assistant can use, which is the pattern this post recommends for /mcp. MCP Guardrail, which generates guardrail proxy URLs for MCP servers so agent tool traffic passes through a governed proxy, is in early access; ask before you plan around it. None of this replaces deciding what to index, putting authentication and rate limits in front, or setting provider spending caps.

Software supply chain hygiene still matters for NLWeb because the reference implementation ships without releases or published advisories, so security fixes reach you only when you pull and rebuild. Every rebuild also re-resolves a long list of provider SDKs, vector store clients and web libraries, and those are the parts the steps below cover.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos shows NLWeb. Each shows a Safeguard surface that uses a control this post recommends: an MCP connection with OAuth and chosen tools, scanning on the machine where the code lives, and approval before each command. You can download Safeguard and follow the CLI documentation to try them.

This demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. It is the opposite of an MCP endpoint left open in development mode: the client authenticates, and a person decides which tools the assistant gets. Use it as a reference for how you want agents to reach your own /mcp.

Click through it at your own pace, or open it full screen.

This demo sets up the local runner on your own machine, approves it with its device code, then scans from the terminal. For NLWeb, the same flow scans the checkout and its requirements.txt files on the machine where you build the service, before you push an image.

Click through it at your own pace, or open it full screen.

This is a real CLI session. Asked to fix a lodash CVE, the CLI reads the finding and advisory, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with approval, prints a summary and pushes nothing, then shows /permissions, /model and /scan .. The example is an npm project; the same propose, approve and test loop applies when a Python dependency in an NLWeb deployment needs to move to a fixed version.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• NLWeb repository README (nlweb-ai/NLWeb)
• Microsoft: Introducing NLWeb, 19 May 2025
• NLWeb Hello World guide
• NLWeb REST API
• NLWeb Database Load Tool Guide
• NLWeb: Life of a Chat Query
• NLWeb OAuth setup guide
• NLWeb response headers
• NLWeb Responsible AI FAQ
• NLWeb SECURITY.md
• NLWeb MCP wrapper source
• NLWeb MCP routes source
• NLWeb auth middleware source
• NLWeb CORS middleware source
• NLWeb web server config
• NLWeb retrieval config
• NLWeb LLM config
• NLWeb environment template
• NLWeb release notes, 23 June 2025
• NLWeb release notes (security fixes, PRs #231 and #236)
• NLWeb release notes, 29 July 2025
• Model Context Protocol specification 2025-11-25
• MCP specification: Transports
• MCP specification: Authorization
• MCP specification: Security Best Practices
• MCP Inspector documentation
• Schema.org: About
• RSS 2.0 Specification (RSS Advisory Board)
• The /llms.txt file proposal
• WebMCP explainer (W3C Web Machine Learning Community Group)
• OWASP Top 10 for LLM Applications 2025

Last reviewed: 24 October 2026.