As mentioned earlier, I use Traefik as the gateway, and all extranet requests go through the gateway.
However, some of our apis are only used by the internal network. For security, we need to ensure that these internal API are only accessible to the internal network.
However, due to the previous Settings, these apis are anonymously accessed. If you change them to require permissions, other teams that rely on our service will need to make corresponding modifications. For some reasons, this path is blocked.
There was only one way for me to install two Traefiks, one internal and one external. Installing two Traefiks is easy, and since the Traefik installation is DaemonSet, the access ports for both Traefiks must be different
--entryPoints.traefik.address=:8080
Copy the codeFor example, if the Internet traefik port is 8080, then the internal Traefik port can be 9090.
See my previous article on installing Traefik in Kubernetes.
And then that’s it? It’s not that easy, of course, but in order to do this,
The corresponding IngressRoute also needs to be modified
The previous IngressRoute was as follows
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: my-service-traefik-ingress
spec:
entryPoints:
- websecure
routes:
- match: Host(`think123.com`) && PathPrefix(`/api/anon/health`, `/api/anon/article`)
kind: Rule
services:
- name: my-service
port: 9000
Copy the codeI want the request path/API /anon/article to be Intranet accessible only, so I need to modify the configuration as follows
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: my-service-traefik-ingress
spec:
entryPoints:
- websecure
routes:
- match: Host(`think123.com`) && PathPrefix(`/api/anon/health`)
kind: Rule
services:
- name: my-service
port: 9000
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: my-service-internal-traefik-ingress
spec:
entryPoints:
- websecure
routes:
- match: Host(`think123-internal.com`) && PathPrefix(`/api/anon/article`)
kind: Rule
services:
- name: my-service
port: 9000
Copy the codeAfter configuring LoadBalance, access the corresponding domain name. But there’s a problem with that
curl -H 'Host:www.think123-internal.com' -X GET https://www.think123.com/api/anon/article
Copy the codeBy specifying the header, we find that it matches the ingressRoute my-service-internal-traefik-ingress, and finally returns the API that only Intranet users can access.
Why do I do a lot of security issues still exist?
In fact, both the internal traefik and the external traefik load all the ingressroutes, hence the above problem.
They had to find a way to load only their own IngressRoute. I checked Traefik’s documentation and found three parameters that could be used.
The first is that you can specify which Namespaces you want to monitor, so Traefik only processes namespaces requests that it monitors
--providers.kubernetescrd.namespaces=default,production
Copy the codeThe second is to label the resource, but only for Traefik Custom Resources, Kubernetes Service,Secrets, these do not work
--providers.kubernetescrd.labelselector="app=traefik"
Copy the codeThe third is to specify ingressClass
--providers.kubernetescrd.ingressclass=traefik-internal
Copy the codeWe need the corresponding resources with kubernetes. IO/ingress. The class notes, identifies resource objects to be processed.
Since we are dealing with resources under the same namespace, the namespace approach is not appropriate.
For the second or third, both are appropriate because we are handling IngressRoute. But considering other environments where nginx and Traefik coexist, we decided to use the ingressClass approach, which nginx actually recommended.
At first, I added the ingressClass setting in the DaemonSet template
--providers.kubernetescrd.ingressclass=$(ingress_class)
Copy the codeThe next step is to configure the different environment variables (configMap) in the different Traefiks. For example, Internet Traefik sets ingress_class= traefik-Internet, while Intranet traefik is empty and is treated as the default ingress controller.
Then specified in the corresponding ingressRoute kubernetes. IO/ingress. Class: traefik – the Internet. Other ingressroutes do not specify this annotation.
It looks great. Give yourself a thumbs up.
But imagination is beautiful, reality has given me a merciless blow. This modification is done, I find that when I visit https://www.think123-internal.com/api/anon/article this API, no traefik to deal with my request.
Isn’t internal Traefik my default traefik? I think the official document says so
If the parameter is non-empty, only resources containing an annotation with the same value are processed.
Otherwise, resources missing the annotation, having an empty value, or the value traefik are processed
Copy the codeIs this empty not an empty string, but an unspecified meaning? What a lack of martial virtue!
Therefore, I had to modify it in a different way. Instead of specifying ingressClass in the DaemonSet template, I added it dynamically through Kustomize. Please refer to my article on the use of Kustomize
Note that the two Traefiks are only slightly different in configuration, so we usually specify one template when installing, and the dynamic values are specified by different Traefiks
patches:
- target:
version: v1beta2
kind: DaemonSet
name: traefik
patch: |-
- op: add
path: /spec/template/spec/containers/0/args/-
value: --providers.kubernetescrd.ingressclass=traefik-internet
Copy the codeIn my configuration, the Internet traefik name is traefik, and the internal traefik name is traefik-internal, and the distinction between extranets and extranets is logical, not just by name.
We specify ingressClass only in Internet Traefik, not Intranet Traefik.
IngressRoute also needs to be modified dynamically.
# kustomization.yaml
patchesJson6902:
- target:
group: traefik.containo.us
version: v1alpha1
kind: IngressRoute
name: my-service-traefik-ingress
path: patches/patch-internet-ingressroute.yaml
# patch-internet-ingressroute.yaml
- op: add
path: /metadata/annotations
value:
kubernetes.io/ingress.class: traefik-internet
Copy the codeIn this way, the Internet traefik. It will only do kubernetes IO/ingress. Class: traefik ingressRoute – the Internet. The previous security issues would not exist.
Reference documentation
- https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/
- https://doc.traefik.io/traefik/providers/kubernetes-crd/#ingressclass