How Do You Integrate Emissary Ingress With OPA

API gateways play a vital role while exposing microservices. They are an additional hop in the network that the incoming request must go through in order to communicate with the services. An API gateway does routing, composition, protocol translation, and user policy enforcement after it receives a request from client and then reverse proxies it to the appropriate underlying API. As the API gateways are capable of doing the above-mentioned tasks, they can be also configured to send the incoming client requests to an external third-party authorization (authz) server. The fate of the incoming request then depends upon the response from this external authz server to the gateway. This is exactly where Open Policy Agent (OPA) comes into the picture.

There are many open source Kubernetes native API gateways out there like Contour, Kong Gateway, Traefik, Gloo, etc.. In this article, we will be exploring the Emissary Ingress.

Let’s dive deep and start understanding more bit about Emissary Ingress.

What’s Emissary Ingress?

Emissary Ingress was earlier known as Ambassador API gateway, it is an open source Kubernetes native API gateway and is currently a CNCF Incubation Project. Like many other Kubernetes gateways, Emissary has also been built to work with Envoy Proxy. It is deployed as complete stateless architecture and supports multiple plugins such as traditional SSO authentication protocols (e.g.: OAuth, OpenID Connect), rate limiting, logging, and tracing service. Emissary utilizes its ExtAuth protocol in AuthService resource to configure the authentication and authorization for incoming requests. ExtAuth supports two protocols: gRPC and plain HTTP. For gRPC interface, the external service must implement Envoy’s external_auth.proto.

OPA

Open Policy Agent is a well-known general-purpose policy engine and has emerged as a policy enforcer across the stacks be it API gateways, service meshes, Kubernetes, microservice, CICD, or IAC. OPA decouples decision making from policy enforcement such that whenever your software needs to make a decision regarding the incoming requests, it queries OPA. OPA-Envoy extends OPA with a gRPC server that implements the Envoy External Authorization API, thus making itself compatible to be as an external authz server to Emissary.

Integrating Emissary Ingress with OPA

The above figure shows highlevel architecture of Emissary and OPA integration. When incoming request from client reaches Emissary, it sends an authorization request to OPA which contains input JSON. OPA evaluates this JSON against the Rego policies provided to it and responds to Emissary, if this result JSON from OPA has allow as true then only the client request is further routed to API or else the request is denied by Emissary and never reaches the API. Now, we will be installing Emissary Ingress and integrate it with OPA for external authorization.

Getting Started

First, we will be needing to start a Minikube cluster. If you don’t have Minikube, you can install it from here.

minikube start

Install the Emissary Ingress to the minikube through Helm.

<span class="token preprocessor property"># Add the Repo:</span>
helm repo <span class="token keyword">add</span> <span class="token class-name">datawire</span> https<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>app<span class="token punctuation">.</span>getambassador<span class="token punctuation">.</span>io
helm repo update
 
<span class="token preprocessor property"># Create Namespace and Install:</span>
kubectl create <span class="token keyword">namespace</span> emissary <span class="token operator">&&</span> \
kubectl apply <span class="token operator">-</span><span class="token class-name">f</span> https<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>app<span class="token punctuation">.</span>getambassador<span class="token punctuation">.</span>io<span class="token operator">/</span>yaml<span class="token operator">/</span>emissary<span class="token operator">/</span><span class="token number">2.2</span><span class="token number">.2</span><span class="token operator">/</span>emissary<span class="token operator">-</span>crds<span class="token punctuation">.</span>yaml
kubectl wait <span class="token operator">--</span>timeout<span class="token operator">=</span>90s <span class="token operator">--</span><span class="token keyword">for</span><span class="token operator">=</span>condition<span class="token operator">=</span>available deployment emissary<span class="token operator">-</span>apiext <span class="token operator">-</span>n emissary<span class="token operator">-</span>system
helm install emissary<span class="token operator">-</span>ingress <span class="token operator">--</span><span class="token keyword">namespace</span> emissary datawire<span class="token operator">/</span>emissary<span class="token operator">-</span>ingress <span class="token operator">&&</span> \
kubectl <span class="token operator">-</span>n emissary wait <span class="token operator">--</span><span class="token keyword">for</span> condition<span class="token operator">=</span>available <span class="token operator">--</span>timeout<span class="token operator">=</span>90s deploy <span class="token operator">-</span>lapp<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>instance<span class="token operator">=</span>emissary<span class="token operator">-</span>ingress

Or go to Emissary Ingress Documentation to install it through Kubernetes YAMLs.

Configuring the routing for demo application

Different gateways have their own set of configurations for exposing a service. In Emissary, we need to configure the routing through Mappings and Listeners.

Mapping resource simply tells Emissary which service to redirect the incoming request to. It is highly configurable like Ingress. You can learn more about Mapping resource on Introduction to the Mapping resource page. We will create a simple Mapping resource which will redirect all the incoming requests to our demo application’s service that is demo-svc.

cat <span class="token operator"><<</span><span class="token constant">EOF</span> <span class="token operator">|</span> kubectl apply <span class="token operator">-</span>f <span class="token operator">-</span>
<span class="token literal-property property">apiVersion</span><span class="token operator">:</span> getambassador<span class="token punctuation">.</span>io<span class="token operator">/</span>v3alpha1
<span class="token literal-property property">kind</span><span class="token operator">:</span> Mapping
<span class="token literal-property property">metadata</span><span class="token operator">:</span>
  <span class="token literal-property property">name</span><span class="token operator">:</span> demo<span class="token operator">-</span>app<span class="token operator">-</span>mapping  
<span class="token literal-property property">spec</span><span class="token operator">:</span>
  <span class="token literal-property property">hostname</span><span class="token operator">:</span> <span class="token string">"*"</span>
  <span class="token literal-property property">prefix</span><span class="token operator">:</span> <span class="token operator">/</span>
  <span class="token literal-property property">service</span><span class="token operator">:</span> demo<span class="token operator">-</span>svc
<span class="token constant">EOF</span>

The Listener resource instructs Emissary where to listen on the network for the incoming request. Here we will create a listener to listen on port 8080 and HTTP protocol and associates with hosts in All namespace . For detailed info visit Listener Docs.

cat <span class="token operator"><<</span><span class="token constant">EOF</span> <span class="token operator">|</span> kubectl apply <span class="token operator">-</span>f <span class="token operator">-</span>
<span class="token literal-property property">apiVersion</span><span class="token operator">:</span> getambassador<span class="token punctuation">.</span>io<span class="token operator">/</span>v3alpha1
<span class="token literal-property property">kind</span><span class="token operator">:</span> Listener
<span class="token literal-property property">metadata</span><span class="token operator">:</span>
  <span class="token literal-property property">name</span><span class="token operator">:</span> demo<span class="token operator">-</span>app<span class="token operator">-</span>listener<span class="token operator">-</span><span class="token number">8080</span>
  <span class="token literal-property property">namespace</span><span class="token operator">:</span> emissary
<span class="token literal-property property">spec</span><span class="token operator">:</span>
  <span class="token literal-property property">port</span><span class="token operator">:</span> <span class="token number">8080</span>
  <span class="token literal-property property">protocol</span><span class="token operator">:</span> <span class="token constant">HTTP</span>
  <span class="token literal-property property">securityModel</span><span class="token operator">:</span> <span class="token constant">XFP</span>
  <span class="token literal-property property">hostBinding</span><span class="token operator">:</span>
    <span class="token literal-property property">namespace</span><span class="token operator">:</span>
      <span class="token literal-property property">from</span><span class="token operator">:</span> <span class="token constant">ALL</span>
<span class="token constant">EOF</span>

Install the Demo Application

Install a simple echo server as a demo application.

cat <span class="token operator"><<</span><span class="token constant">EOF</span> <span class="token operator">|</span> kubectl apply <span class="token operator">-</span>f <span class="token operator">-</span>
<span class="token literal-property property">apiVersion</span><span class="token operator">:</span> apps<span class="token operator">/</span>v1
<span class="token literal-property property">kind</span><span class="token operator">:</span> Deployment
<span class="token literal-property property">metadata</span><span class="token operator">:</span>
  <span class="token literal-property property">name</span><span class="token operator">:</span> demo<span class="token operator">-</span>app
<span class="token literal-property property">spec</span><span class="token operator">:</span>
  <span class="token literal-property property">replicas</span><span class="token operator">:</span> <span class="token number">1</span>
  <span class="token literal-property property">selector</span><span class="token operator">:</span>
    <span class="token literal-property property">matchLabels</span><span class="token operator">:</span>
      <span class="token literal-property property">app</span><span class="token operator">:</span> demo<span class="token operator">-</span>app
  <span class="token literal-property property">template</span><span class="token operator">:</span>
    <span class="token literal-property property">metadata</span><span class="token operator">:</span>
      <span class="token literal-property property">labels</span><span class="token operator">:</span>
        <span class="token literal-property property">app</span><span class="token operator">:</span> demo<span class="token operator">-</span>app
    <span class="token literal-property property">spec</span><span class="token operator">:</span>
      <span class="token literal-property property">containers</span><span class="token operator">:</span>
      <span class="token operator">-</span> name<span class="token operator">:</span> http<span class="token operator">-</span>svc
        <span class="token literal-property property">image</span><span class="token operator">:</span> gcr<span class="token punctuation">.</span>io<span class="token operator">/</span>google_containers<span class="token operator">/</span>echoserver<span class="token operator">:</span><span class="token number">1.8</span>
        <span class="token literal-property property">ports</span><span class="token operator">:</span>
        <span class="token operator">-</span> containerPort<span class="token operator">:</span> <span class="token number">8080</span>
        <span class="token literal-property property">env</span><span class="token operator">:</span>
        <span class="token operator">-</span> name<span class="token operator">:</span> <span class="token constant">NODE_NAME</span>
          <span class="token literal-property property">valueFrom</span><span class="token operator">:</span>
            <span class="token literal-property property">fieldRef</span><span class="token operator">:</span>
              <span class="token literal-property property">fieldPath</span><span class="token operator">:</span> spec<span class="token punctuation">.</span>nodeName
        <span class="token operator">-</span> name<span class="token operator">:</span> <span class="token constant">POD_NAME</span>
          <span class="token literal-property property">valueFrom</span><span class="token operator">:</span>
            <span class="token literal-property property">fieldRef</span><span class="token operator">:</span>
              <span class="token literal-property property">fieldPath</span><span class="token operator">:</span> metadata<span class="token punctuation">.</span>name
        <span class="token operator">-</span> name<span class="token operator">:</span> <span class="token constant">POD_NAMESPACE</span>
          <span class="token literal-property property">valueFrom</span><span class="token operator">:</span>
            <span class="token literal-property property">fieldRef</span><span class="token operator">:</span>
              <span class="token literal-property property">fieldPath</span><span class="token operator">:</span> metadata<span class="token punctuation">.</span>namespace
        <span class="token operator">-</span> name<span class="token operator">:</span> <span class="token constant">POD_IP</span>
          <span class="token literal-property property">valueFrom</span><span class="token operator">:</span>
            <span class="token literal-property property">fieldRef</span><span class="token operator">:</span>
              <span class="token literal-property property">fieldPath</span><span class="token operator">:</span> status<span class="token punctuation">.</span>podIP
<span class="token operator">--</span><span class="token operator">-</span>
<span class="token literal-property property">apiVersion</span><span class="token operator">:</span> v1
<span class="token literal-property property">kind</span><span class="token operator">:</span> Service
<span class="token literal-property property">metadata</span><span class="token operator">:</span>
  <span class="token literal-property property">name</span><span class="token operator">:</span> demo<span class="token operator">-</span>svc
  <span class="token literal-property property">labels</span><span class="token operator">:</span>
    <span class="token literal-property property">app</span><span class="token operator">:</span> demo<span class="token operator">-</span>app
<span class="token literal-property property">spec</span><span class="token operator">:</span>
  <span class="token literal-property property">ports</span><span class="token operator">:</span>
  <span class="token operator">-</span> name<span class="token operator">:</span> http
    <span class="token literal-property property">port</span><span class="token operator">:</span> <span class="token number">80</span>
    <span class="token literal-property property">protocol</span><span class="token operator">:</span> <span class="token constant">TCP</span>
    <span class="token literal-property property">targetPort</span><span class="token operator">:</span> <span class="token number">8080</span>
  <span class="token literal-property property">selector</span><span class="token operator">:</span>
    <span class="token literal-property property">app</span><span class="token operator">:</span> demo<span class="token operator">-</span>app
<span class="token constant">EOF</span>

Communicate with the demo app at different paths.

minikube service emissary<span class="token operator">-</span>ingress <span class="token operator">-</span>n emissary

Note: The above exposing method may not work for macOS users. They can use busybox and configure it to hit the emissary local endpoint instead.

Copy the private URL with target port 80. The URL must be IP 192.168.49.2 followed by a NodePort like http://192.168.49.2:30329. Export the NodePort value to $NODEPORT environment variable and curl to that at paths as follows:

curl http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span><span class="token keyword">public</span>

and

curl http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span>secured

OPA has not yet been added to the setup and the above curl requests are directly sent to API without any policy enforcement.

How to Install and Configure OPA?

OPA will be reading the policies fed to it via a configmap. Create the following configmap which contains a policy that allows all incoming requests only through GET method.

cat <<EOF | kubectl apply -n emissary -f  -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-policy
data: 
  policy.rego: |-
    package envoy.authz

    default allow = false

    allow {
       input.attributes.request.http.method == "GET" 
    }
EOF

OPA can be configured as an external authorization server via deploying it as an independent deployment or as a sidecar to the emissary-ingress. Here we will add it as a sidecar. Save the following YAML as opa-patch.yaml.

spec:
  template:
    spec:
      containers:
      - name: opa
        image: openpolicyagent/opa:latest-envoy
        ports:
        - containerPort: 9191
        args:
          - "run"
          - "--server"
          - "--addr=0.0.0.0:8181"
          - "--set=plugins.envoy_ext_authz_grpc.addr=0.0.0.0:9191"
          - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
          - "--set=decision_logs.console=true"
          - "--ignore=.*"
          - "/policy/policy.rego"
        volumeMounts:
          - mountPath: /policy
            name: demo-policy
            readOnly: true
      volumes:
      - name: demo-policy
        configMap:
          name: demo-policy

patch the emissary-ingress deployment and wait for the all the emissary-ingress pods to restart.

kubectl patch deployment emissary-ingress -n emissary --patch-file opa-patch.yaml

Wait until all the emissary-ingress pods come to Running state with OPA sidecar.

Create the following AuthService. AuthService is a resource which configures Emissary to communicate with an external service for Authn and Authz of incoming request. We are configuring it communicate with OPA on localhost since OPA is deployed as a sidecar.

cat <span class="token operator"><<</span><span class="token constant">EOF</span> <span class="token operator">|</span> kubectl apply <span class="token operator">-</span>f <span class="token operator">-</span>
<span class="token literal-property property">apiVersion</span><span class="token operator">:</span> getambassador<span class="token punctuation">.</span>io<span class="token operator">/</span>v3alpha1
<span class="token literal-property property">kind</span><span class="token operator">:</span> AuthService
<span class="token literal-property property">metadata</span><span class="token operator">:</span>
  <span class="token literal-property property">name</span><span class="token operator">:</span> opa<span class="token operator">-</span>ext<span class="token operator">-</span>authservice
  <span class="token literal-property property">namespace</span><span class="token operator">:</span> emissary
  <span class="token literal-property property">labels</span><span class="token operator">:</span>
    <span class="token literal-property property">product</span><span class="token operator">:</span> aes
    <span class="token literal-property property">app</span><span class="token operator">:</span> opa<span class="token operator">-</span>ext<span class="token operator">-</span>auth
<span class="token literal-property property">spec</span><span class="token operator">:</span>
  <span class="token literal-property property">proto</span><span class="token operator">:</span> grpc
  <span class="token literal-property property">auth_service</span><span class="token operator">:</span> localhost<span class="token operator">:</span><span class="token number">9191</span>
  <span class="token literal-property property">timeout_ms</span><span class="token operator">:</span> <span class="token number">5000</span>
  <span class="token literal-property property">tls</span><span class="token operator">:</span> <span class="token string">"false"</span>
  <span class="token literal-property property">allow_request_body</span><span class="token operator">:</span> <span class="token boolean">true</span>
  <span class="token literal-property property">protocol_version</span><span class="token operator">:</span> v2
  <span class="token literal-property property">include_body</span><span class="token operator">:</span>
    <span class="token literal-property property">max_bytes</span><span class="token operator">:</span> <span class="token number">8192</span>
    <span class="token literal-property property">allow_partial</span><span class="token operator">:</span> <span class="token boolean">true</span>
  <span class="token literal-property property">status_on_error</span><span class="token operator">:</span>
    <span class="token literal-property property">code</span><span class="token operator">:</span> <span class="token number">504</span>
  <span class="token literal-property property">failure_mode_allow</span><span class="token operator">:</span> <span class="token boolean">false</span>
<span class="token constant">EOF</span>

Try doing curl now, since the policy accepts requests coming through GET method and there are no restrictions on path, both the request will get a 200 OK response.

curl <span class="token operator">-</span>i http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span><span class="token keyword">public</span>
curl <span class="token operator">-</span>i http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span><span class="token keyword">private</span>

Now lets edit the policy to accept incoming requests at path /public only and request to any other path will be denied.

cat <<EOF | kubectl apply -n emissary -f  -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-policy
data: 
  policy.rego: |-
    package envoy.authz

    default allow = false

    allow {
       input.attributes.request.http.method == "GET"
       input.attributes.request.http.path == "/public" 
    }
EOF

Now restart the emissary ingress deployment for policy changes to take effect.

kubectl rollout restart deployment emissary<span class="token operator">-</span>ingress <span class="token operator">-</span>n emissary

Wait until all the emissary-ingress pods come to Running state after restart.

Now do a curl request at path /public, it will be accepted but at path /private it will be denied by OPA with a 403 response and hence the request will not reach the demo API.

curl <span class="token operator">-</span>i http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span><span class="token keyword">public</span>
curl <span class="token operator">-</span>i http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token number">192.168</span><span class="token number">.49</span><span class="token number">.2</span><span class="token operator">:</span>$<span class="token constant">NODEPORT</span><span class="token operator">/</span><span class="token keyword">private</span>

Conclusion

The decision-making about the incoming request from the client to exposed API can be decoupled to OPA as an external authorization server in the Emissary Ingress setup. OPA can be added as a plug-and-play policy enforcer to Emissary and any other gateways supporting the Envoy External Authorization API.

We hope you found this post informative and engaging. Connect with us over Twitter and Linkedin and start a conversation.

 

 

Guest post originally published on InfraCloud’s blog by Tayyab Jamadar
Source CNCF