← Back to posts

GCP ACE: Service Mesh Security — mTLS, Service Authentication, End-User Authentication, and Authorization

The RBAC and Workload Identity article earlier in this series covered who can call the Kubernetes API, and how a Pod authenticates to Google Cloud APIs. Inside the mesh, there’s a third and fourth question: can one service prove its identity to another service, and can an end user’s identity be verified and enforced as their request moves through multiple services. This article covers both, plus the encryption and authorization layers that make them enforceable.

Authentication and encryption: PeerAuthentication (mTLS)

Every meshed workload gets an identity (tied to its Kubernetes service account) and a certificate, issued automatically by the mesh’s control plane — no cert-management work on your part. PeerAuthentication controls whether workload-to-workload traffic is required to use mutual TLS.

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "mesh-wide"
  namespace: "istio-system"     # applying it here, with no selector, makes it mesh-wide
spec:
  mtls:
    mode: STRICT
ModeBehavior
PERMISSIVE (default)Accepts both plaintext and mTLS. Thanks to auto mTLS (on by default in Cloud Service Mesh), a client sidecar detects whether the server has a sidecar and uses mTLS whenever it does — this mode exists for gradual migration, so you can onboard services into the mesh one at a time without breaking traffic from ones that haven’t joined yet.
STRICTRejects plaintext entirely. A request from a Pod with no sidecar fails outright.

Scoping the policy to one namespace instead of the whole mesh is just a matter of putting it in that namespace’s own metadata.namespace with the same spec.

How the certificates actually work: the mesh’s certificate authority (in Cloud Service Mesh, the managed Cloud Service Mesh CA — formerly Mesh CA — or Certificate Authority Service) issues and rotates a workload certificate for every sidecar automatically, whether or not you’ve written a PeerAuthentication policy; the policy only decides whether mTLS is required. During the TLS handshake between two sidecars, each side presents its certificate, and the receiving side verifies the sender’s identity before forwarding the request to the application container. This is what “service authentication” means in practice — not a token the application checks, but a certificate the sidecar checks, transparently, before the application ever sees the request.

Service authentication: who is calling, as a workload

Once mTLS is active, every request inside the mesh carries a cryptographically verified source identity — the calling workload’s service account. In upstream Istio the default trust domain makes that cluster.local/ns/<namespace>/sa/<service-account>; in Cloud Service Mesh the trust domain is the fleet’s workload identity pool, so the identity is PROJECT_ID.svc.id.goog/ns/<namespace>/sa/<service-account>. This is what service-to-service authorization rules (below) are actually built on: not an IP address or a Service name, which can be spoofed or reused, but a certificate-backed identity tied to a Kubernetes service account.

End-user authentication: RequestAuthentication

PeerAuthentication verifies services. It says nothing about the human (or external client) whose request originally triggered the call chain. RequestAuthentication fills that gap — it validates a JWT attached to incoming requests, checking it against a trusted issuer and its public signing keys.

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: shop-jwt
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders-service
  jwtRules:
    - issuer: "https://login.shop.example"
      jwksUri: "https://login.shop.example/keys.json"
      audiences:
        - "orders.shop.example"

This only validates the token — checks the signature, issuer, and audience — and attaches the result (and the token’s claims) to the request for the next policy to use. On its own, it does not block unauthenticated requests; an AuthorizationPolicy has to do that explicitly.

Authorization: AuthorizationPolicy

AuthorizationPolicy is where an actual allow/deny decision gets made, using either the service identity (from mTLS) or the end-user identity (from RequestAuthentication), or both together.

Service-to-service authorization, based purely on the calling workload’s verified identity:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-checkout-to-orders
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "PROJECT_ID.svc.id.goog/ns/shop/sa/checkout-service"

Only requests whose mTLS-verified source identity is exactly checkout-service in the shop namespace are allowed to reach orders-service — every other caller is denied by default once any ALLOW policy exists for that selector.

End-user authorization, combining a validated JWT identity with a claim check:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-require-finance-role
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders-service
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals:
              - "https://login.shop.example/*"
      to:
        - operation:
            methods: ["GET"]
            paths: ["/api/invoices/*"]
      when:
        - key: request.auth.claims[roles]
          values: ["finance"]

This only allows GET /api/invoices/* when the request carries a JWT validated by the RequestAuthentication above (requestPrincipals), and that token’s roles claim includes finance. Claim names are case-sensitive and the issuer has to match exactly what’s configured in the RequestAuthentication — a common cause of unexpected 403s when these two objects are configured by different people.

DENY rules are evaluated before ALLOW rules. A DENY policy that matches takes effect regardless of any ALLOW policy that would otherwise permit the request — this ordering is a frequent exam detail. (Istio’s full order is CUSTOM, then DENY, then ALLOW; CUSTOM delegates the decision to an external authorizer.)

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-reporting-admin
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders-service
  action: DENY
  rules:
    - to:
        - operation:
            paths: ["/admin/*"]
      from:
        - source:
            principals:
              - "PROJECT_ID.svc.id.goog/ns/shop/sa/reporting-job"

How these four pieces fit together

A single request through the mesh might involve all four layers at once:

  1. PeerAuthentication (STRICT) ensures the request arrived over mTLS at all — rejecting it outright if not.
  2. Service authentication gives the receiving sidecar a verified identity for the calling service (checkout-service), derived from that mTLS handshake.
  3. RequestAuthentication separately validates a JWT in the request, proving who the end user is, without yet deciding anything.
  4. AuthorizationPolicy makes the actual decision, using either identity (or both), the request’s method/path, and claim values — DENY rules checked first.

Exam Takeaways

Sources

↗ LinkedIn