Backend discovery modes

Virtual MCP Server (vMCP) supports two backend discovery modes, allowing you to optimize for either operational convenience (discovered mode with declarative backend management) or security (inline mode with minimal permissions).

Overview

The deployment mode is configured via the spec.outgoingAuth.source field in the VirtualMCPServer resource.

Mode	Backend Discovery	RBAC Requirements	K8s API Access
`discovered`	Runtime from K8s API	Namespace-scoped read + status updates	Yes
`inline`	Inline from configuration	Minimal (status updates only)	No (except status)

Design rationale

Backend discovery is configured via outgoingAuth.source because the authentication strategy and backend source are tightly coupled:

Discovered backends (source: discovered) are found at runtime by querying the Kubernetes API and can reference MCPExternalAuthConfig resources for their authentication
Inline backends (source: inline) are defined in the configuration and require their authentication to be explicitly configured in outgoingAuth.backends

This coupling ensures authentication configuration matches the backend discovery method, preventing misconfigurations.

When to use discovered mode

Choose discovered mode when:

Backends change frequently and you want declarative management via Kubernetes resources
Centralized authentication via MCPExternalAuthConfig is preferred
Namespace-scoped read permissions are acceptable

When to use inline mode

Choose inline mode when:

Security or compliance requires minimal permissions and attack surface
Backend configuration is stable and changes infrequently
Explicit control over all backend details is required (zero-trust, air-gapped environments)

Trade-offs comparison

Consideration	Discovered Mode	Inline Mode
Backend management	Declarative (K8s resources)	Explicit (in configuration)
Configuration changes	Add/remove resources without vMCP config changes	Update vMCP config and restart
RBAC permissions	Namespace read access	Minimal (status updates only)
Attack surface	Larger (K8s API access)	Smaller (no backend discovery API access)
Auth management	Centralized (MCPExternalAuthConfig)	Duplicated in YAML
Individual backend pod updates	Supported without vMCP changes	Requires vMCP awareness

Discovered mode

Discovered mode queries the Kubernetes API at runtime to find backend MCP servers. This is the default mode.

Discovered mode configuration

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: VirtualMCPServer
metadata:
  name: my-vmcp
  namespace: toolhive-system
spec:
  config:
    groupRef: my-group
  incomingAuth:
    type: anonymous
  outgoingAuth:
    source: discovered # Discover backends at runtime (default)

How it works

When vMCP starts in discovered mode:

Group verification: Verifies the referenced MCPGroup exists
Workload discovery: Queries all MCPServer and MCPRemoteProxy resources in the group
Backend conversion: For each workload:
- Extracts service URL and transport type
- Resolves authentication from externalAuthConfigRef if configured
- Adds metadata labels
Capability querying: Calls each backend's initialize method to discover available tools, resources, and prompts
Status updates: Reports backend health in the VirtualMCPServer status

Discovered mode RBAC

The operator automatically creates the required RBAC resources (ServiceAccount, Role, and RoleBinding) when you create a VirtualMCPServer resource.

For reference, the vMCP service account needs read access to:

configmaps, secrets: Read OIDC configs and auth secrets
mcpgroups: Verify group exists and list members
mcpservers, mcpremoteproxies: Discover backend workloads
mcpexternalauthconfigs: Resolve authentication configurations
mcptoolconfigs: Resolve tool filtering and renaming
virtualmcpservers/status: Update status with discovered backends

Runtime updates

When backend resources are added, modified, or removed in the group:

The change does NOT automatically trigger vMCP to rediscover backends
vMCP continues using the backend list from startup
To pick up changes, restart the vMCP pod:

kubectl rollout restart deployment vmcp-my-vmcp -n toolhive-system

Inline mode

Inline mode uses pre-configured backends defined in the VirtualMCPServer resource. This eliminates the need for Kubernetes API access (except status updates), reducing the attack surface.

Inline mode configuration

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: VirtualMCPServer
metadata:
  name: my-vmcp
  namespace: toolhive-system
spec:
  config:
    groupRef: my-group
    backends:
      - name: github-mcp
        url: http://github-mcp.toolhive-system.svc.cluster.local:8080
        transport: sse
      - name: fetch-mcp
        url: http://fetch-mcp.toolhive-system.svc.cluster.local:8080
        transport: streamable-http
  incomingAuth:
    type: anonymous
  outgoingAuth:
    source: inline # Use inline backend configuration
    backends:
      github-mcp:
        type: external_auth_config_ref
        externalAuthConfigRef:
          name: github-token-config

note

The groupRef field is required in both discovered and inline modes. In inline mode, the group reference is used for organizational purposes and status reporting, even though backends are defined inline rather than discovered from the group.

info

Backend authentication in outgoingAuth.backends uses references to MCPExternalAuthConfig resources, not inline configuration. Create the MCPExternalAuthConfig resource first, then reference it by name. See the Authentication guide for complete examples.

Backend configuration

Each backend in spec.config.backends requires:

Field	Description	Required
`name`	Backend identifier (must match auth config keys)	Yes
`url`	Backend MCP server URL (must be `http://` or `https://`)	Yes
`transport`	MCP transport protocol (`sse` or `streamable-http`)	Yes
`metadata`	Custom labels for the backend	No

Inline mode RBAC

Inline mode requires minimal permissions:

virtualmcpservers/status: Update status (only permission needed)

The operator still creates RBAC resources for status updates, but the vMCP pod does not query the Kubernetes API for backend discovery.

Verify backend status

Check VirtualMCPServer status

View discovered backends and their health:

kubectl get virtualmcpserver my-vmcp -n toolhive-system -o yaml

The status includes:

status:
  phase: Ready # Pending|Ready|Degraded|Failed
  backendCount: 2
  discoveredBackends:
    - name: github-mcp
      status: ready
      authType: token_exchange
      lastHealthCheck: '2025-02-02T15:30:00Z'
    - name: fetch-mcp
      status: ready
      authType: unauthenticated
      lastHealthCheck: '2025-02-02T15:30:00Z'

Query the status endpoint

vMCP exposes an unauthenticated /status HTTP endpoint for operational monitoring:

kubectl port-forward -n toolhive-system svc/vmcp-my-vmcp 4483:4483
curl http://localhost:4483/status

Response format:

{
  "backends": [
    {
      "name": "github-mcp",
      "health": "healthy",
      "transport": "sse",
      "auth_type": "token_exchange"
    },
    {
      "name": "fetch-mcp",
      "health": "healthy",
      "transport": "streamable-http",
      "auth_type": "unauthenticated"
    }
  ],
  "healthy": true,
  "version": "v1.2.3",
  "group_ref": "my-group"
}

Health values:

healthy: Backend is responding correctly
degraded: Backend responding but with errors
unhealthy: Backend not responding
unknown: Health check not yet performed

info

The /status endpoint is unauthenticated for operator consumption. It exposes operational metadata but does not include secrets, tokens, internal URLs, or request data.

Switch deployment modes

Switching between modes requires updating the VirtualMCPServer resource and restarting the vMCP pod.

From discovered to inline

List current backends to capture their configuration:

kubectl get virtualmcpserver my-vmcp -n toolhive-system \
  -o jsonpath='{.status.discoveredBackends}' | jq

Update the VirtualMCPServer to inline mode:

spec:
  config:
    groupRef: my-group
    backends:
      - name: github-mcp
        url: http://github-mcp.toolhive-system.svc.cluster.local:8080
        transport: sse
      # Add all backends from status.discoveredBackends
  outgoingAuth:
    source: inline

The operator automatically restarts the vMCP pod with the new configuration
Optionally reduce RBAC permissions by removing read access to MCPServer and MCPRemoteProxy resources (keep status update permissions)

From inline to discovered

Ensure backend MCPServer and MCPRemoteProxy resources exist in the group

Update the VirtualMCPServer to discovered mode:

spec:
  config:
    groupRef: my-group
    # Remove backends array
  outgoingAuth:
    source: discovered

Verify RBAC permissions are configured (operator creates them automatically)
The operator automatically restarts the vMCP pod with the new configuration

Check status to verify backends were discovered:

kubectl get virtualmcpserver my-vmcp -n toolhive-system \
  -o jsonpath='{.status.discoveredBackends}' | jq

Complete example

Here's a complete example showing all required resources for discovered mode with authentication:

---
# 1. Create the MCPGroup
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPGroup
metadata:
  name: engineering-tools
  namespace: toolhive-system
spec:
  description: Engineering team MCP servers

---
# 2. Create authentication config for GitHub backend
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPExternalAuthConfig
metadata:
  name: github-token-config
  namespace: toolhive-system
spec:
  type: tokenExchange
  tokenExchange:
    tokenUrl: https://oauth.example.com/token
    clientId: github-mcp-client
    clientSecretRef:
      name: github-oauth-secret
      key: client-secret
    audience: github-api

---
# 3. Create backend MCPServer
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: github-mcp
  namespace: toolhive-system
spec:
  groupRef: engineering-tools
  image: ghcr.io/example/github-mcp-server:v1.2.3
  transport: sse
  externalAuthConfigRef:
    name: github-token-config

---
# 4. Create VirtualMCPServer (discovered mode)
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: VirtualMCPServer
metadata:
  name: engineering-vmcp
  namespace: toolhive-system
spec:
  config:
    groupRef: engineering-tools
  incomingAuth:
    type: oidc
    oidc:
      issuer: https://auth.company.com
      audience: engineering-vmcp
  outgoingAuth:
    source: discovered # Discovers github-mcp and its auth config

Apply all resources:

kubectl apply -f vmcp-complete-example.yaml

Verify backends were discovered:

kubectl get virtualmcpserver engineering-vmcp -n toolhive-system \
  -o jsonpath='{.status.discoveredBackends}' | jq

Troubleshooting

Backends not appearing in status

Symptoms:

status.discoveredBackends is empty or missing backends
status.backendCount is 0 or lower than expected

Possible causes and solutions:

MCPGroup not in Ready state
```
kubectl get mcpgroup my-group -n toolhive-system
```
Wait for the group to reach Ready state before starting vMCP.
Backend resources not referencing the correct group
```
kubectl get mcpserver,mcpremoteproxy -n toolhive-system \
  -o custom-columns=NAME:.metadata.name,GROUP:.spec.groupRef
```
Ensure all backends have spec.groupRef matching the VirtualMCPServer's spec.config.groupRef.
vMCP pod not restarted after backend changes

Backend changes require a pod restart to be discovered:
```
kubectl rollout restart deployment vmcp-my-vmcp -n toolhive-system
```
RBAC permissions missing (discovered mode)

Check the vMCP service account has required permissions:
```
kubectl get role -n toolhive-system | grep vmcp
kubectl describe role vmcp-my-vmcp -n toolhive-system
```
The operator should create these automatically. If missing, delete and recreate the VirtualMCPServer resource.

Backends showing as unavailable

Symptoms:

status.discoveredBackends[].status is unavailable or unknown
/status endpoint shows health: unhealthy

Possible causes and solutions:

Backend pod not running

kubectl get pods -n toolhive-system -l app.kubernetes.io/name=my-backend

Check backend pod logs for errors:

kubectl logs -n toolhive-system deployment/my-backend

Backend service not accessible

Test connectivity from vMCP pod:

kubectl exec -n toolhive-system deployment/vmcp-my-vmcp -- \
  wget -O- http://my-backend:8080/health

Authentication failing

Check vMCP logs for auth errors:
```
kubectl logs -n toolhive-system deployment/vmcp-my-vmcp | grep ERROR
```
Common auth issues:
- Invalid OIDC configuration in MCPExternalAuthConfig
- Expired or invalid client secrets
- Token exchange endpoint unreachable
Backend returning errors on initialize

The backend may be misconfigured or failing to start properly. Check backend logs and ensure it responds correctly to MCP initialize requests.

RBAC permission errors

Symptoms:

vMCP logs show forbidden or unauthorized errors
Backends not being discovered in discovered mode

Error examples:

Failed to list MCPServers: mcpservers.toolhive.stacklok.dev is forbidden:
User "system:serviceaccount:toolhive-system:vmcp-my-vmcp" cannot list
resource "mcpservers"

Solutions:

Verify service account and role binding exist

kubectl get serviceaccount vmcp-my-vmcp -n toolhive-system
kubectl get role vmcp-my-vmcp -n toolhive-system
kubectl get rolebinding vmcp-my-vmcp -n toolhive-system

Check role permissions
```
kubectl describe role vmcp-my-vmcp -n toolhive-system
```
Required permissions for discovered mode:
- configmaps, secrets: get, list, watch
- mcpgroups, mcpservers, mcpremoteproxies: get, list, watch
- mcpexternalauthconfigs, mcptoolconfigs: get, list, watch
- virtualmcpservers/status: update, patch
Recreate RBAC resources

If RBAC resources are missing or incorrect, delete and recreate the VirtualMCPServer:
```
kubectl delete virtualmcpserver my-vmcp -n toolhive-system
kubectl apply -f my-vmcp.yaml
```
The operator will recreate all RBAC resources automatically.

Mode switching issues

Symptoms:

vMCP pod fails to start after switching modes
Configuration validation errors

Switching from discovered to inline:

Ensure you define spec.config.backends[] before changing source to inline:

spec:
  config:
    backends: [] # ❌ Empty array will fail validation

Switching from inline to discovered:

Remove the spec.config.backends[] array when switching to discovered mode:

spec:
  config:
    backends: [...] # ❌ Should be removed in discovered mode

Health check failures

Symptoms:

/status endpoint shows backends as degraded or unhealthy
Intermittent backend availability

Possible causes:

Backend service overloaded or slow

Health checks timeout after 5 seconds. If backends are slow to respond, they'll be marked unhealthy even if functional.
Network issues between vMCP and backends

Check network policies and service mesh configuration that might block or slow connections.
Backend requires authentication for initialize

Ensure externalAuthConfigRef is properly configured if the backend requires authentication.

Configuration validation errors

Missing groupRef:

Error: spec.config.groupRef is required

Fix: Add spec.config.groupRef referencing an existing MCPGroup.

Invalid backend URL in inline mode:

Error: spec.config.backends[0].url must start with http:// or https://

Fix: Ensure backend URLs use proper scheme:

backends:
  - name: my-backend
    url: http://my-backend.default.svc.cluster.local:8080 # Valid
    # url: my-backend:8080 # Invalid

Missing backends array in inline mode:

Error: spec.config.backends is required when outgoingAuth.source is "inline"

Fix: Define at least one backend in spec.config.backends when using inline mode.

Invalid transport protocol:

Error: spec.config.backends[0].transport must be "sse" or "streamable-http"

Fix: Use only supported transport protocols:

backends:
  - name: my-backend
    transport: sse # Valid
    # transport: stdio # Invalid for inline backends

Referenced MCPExternalAuthConfig not found:

Error: MCPExternalAuthConfig "github-token-config" not found in namespace "toolhive-system"

Fix: Create the MCPExternalAuthConfig resource before referencing it, or remove the auth reference.

Overview​

When to use discovered mode​

When to use inline mode​

Trade-offs comparison​

Discovered mode​

Discovered mode configuration​

How it works​

Discovered mode RBAC​

Runtime updates​

Inline mode​

Inline mode configuration​

Backend configuration​

Inline mode RBAC​

Verify backend status​

Check VirtualMCPServer status​

Query the status endpoint​

Switch deployment modes​

From discovered to inline​

From inline to discovered​

Complete example​

Troubleshooting​

Related information​

Overview

When to use discovered mode

When to use inline mode

Trade-offs comparison

Discovered mode

Discovered mode configuration

How it works

Discovered mode RBAC

Runtime updates

Inline mode

Inline mode configuration

Backend configuration

Inline mode RBAC

Verify backend status

Check VirtualMCPServer status

Query the status endpoint

Switch deployment modes

From discovered to inline

From inline to discovered

Complete example

Troubleshooting

Related information