Kubernetes (K8S) as the most popular container arrangement management tool, has gradually been the operation and maintenance of major enterprises as the preferred scheme, container deployment application scheme based on K8S is also more and more. Some time ago also found time to learn this technology, with two words: really fragrant.
However, as a small business developer, there will inevitably be trouble in using K8S. That is, the default open port of K8S is 3000-32767. If you use NodePort as an external access solution, you need to use the domain name /IP with the number of the upper end, which is very ugly. LodaBalance’s solution relies on the load balancers of the major carriers, which is a costly tool, so how to access your own service directly through a domain name without ports becomes a major focus.
What is the cause of the problem?
K8S opens port 3000-32767 by default, so there is no specified HTTP(S) port other than 80 or 443 when the service is released. As a result, when we access the corresponding service, we need to specify the port number after the domain name and IP to access. As an obsessive-compulsive program, we hope to solve this problem.
solution
Ports 80 and 443 are Web servers, so consider using a reverse proxy to access the service.
Thought analysis
Based on the above ideas, I came up with two solutions:
1. Use the NGINX server as a reverse proxy
Advantages: 1, simple, fast, as long as you can write nginx configuration files can be implemented 2, more general, independent deployment of a nginx is equivalent to doing your own load balancer, for later extension and migration are very convenient disadvantages: 1, because of the separation from the control of K8S, so if there is a problem with the service, it needs to independently monitor and maintain, and management is complicated. 2, independent deployment requires independent maintenance of a set of configuration, which is quite complicatedCopy the code2. Deploy ingress-nginx on K8S for reverse proxy
Ingress-nginx is deployed on top of K8S, so K8S can monitor the operation of the service itself. It is easy to manage a set of YAML files to complete the deployment of multiple domain names. If ingress-nginx is used to access ports 80 and 443, it must be bound to the corresponding host, so that the service itself cannot be separated from the host port. If the ingress-nginx host is down, the migration cannot be normal, and kubernetes retry mechanism is not availableCopy the codeImplementation method
Nginx can be installed as a reverse proxy, or docker can be installed as a reverse proxy
I used Docker to install it
On the host to create a directory to store the content of the nginx # mkdir -p/root/docker/nginx/conf. # d mkdir -p/root/docker/nginx/HTML # mkdir -p / root/docker/nginx/log run a docker command # docker run - d - v/root/docker/nginx/conf., d/a: / etc/nginx/conf. D/v /root/docker/nginx/html/:/usr/share/nginx/html -v /root/docker/nginx/logs/:/var/log/nginx -p 80:80 -p 443:443 --name=nginx --restart=always --privileged=true # docker ps mingl # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3AE5fb0347E8 nginx "/ docker-entryPoint...." 3 weeks ago Up 7 days 0.0.0.0:80->80/ TCP, :::80->80/ TCP, 0.0.0.0:443->443/ TCP, :::443->443/ TCP Nginx configuration is complete.Copy the codeConf file in /etc/nginx, which contains all *. Conf files in conf. D, so there is no need to add a separate HTTP layer. So I didn’t use HTTP {} in subsequent configuration files.
Here is a configuration file to introduce:
In the/root/docker/nginx/conf. D/below, create a console. The conf file
Upstream console {server ip1:port1; server ip2:port2; } ## rewrite server {listen 80; / / rewrite server {listen 80; Server_name Specifies the domain name. rewrite ^(.*)$ https://$host$1; # redirect all HTTP requests via the rewrite directive to HTTPS #location / {# proxy_pass http://console; # proxy_redirect off; # proxy_set_header Host $host; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # proxy_set_header X-Forwarded-Proto $scheme; # proxy_set_header Upgrade $http_upgrade; # proxy_set_header Connection $connection_upgrade; # proxy_set_header X-Request-ID $request_id; # proxy_next_upstream error timeout; # proxy_http_version 1.1; If you want to create a cert directory on the host conf.d file, you can create a cert directory on the host conf.d file. My certificate is under host/root/docker/nginx/conf. # # because under d/cert as container mapping is - v/root/docker/nginx/conf., d/a: / etc/nginx/conf. D / # # So ssl_certificate and ssl_certificate_key configuration will be written below # # ssl_certificate/etc/nginx/conf. D/XXX. Pem; ## ssl_certificate_key /etc/nginx/conf.d/xxx.key server { listen 443 ssl; Ssl_certificate Specifies the path to the SSL pem certificate file. Ssl_certificate_key Specifies the directory for storing the KEY certificate for SSL. ssl_session_timeout 5m; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:! NULL:! aNULL:! MD5:! ADH:! RC4; # indicates the type of cipher suite used. Ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # indicates the type of TLS protocol used. ssl_prefer_server_ciphers on; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; # gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain application/javascript application/x-javascript text/javascript application/json text/css application/xml application/x-httpd-php image/jpeg image/gif image/png; gzip_vary off; gzip_disable "MSIE [1-6]\."; Server_name Specifies the domain name. location / { proxy_pass http://console; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Request-ID $request_id; proxy_next_upstream error timeout; Proxy_http_version 1.1; }} In addition, the HTTPS certificate must be the same as server_name for authentication.Copy the codeIn this way, nginx’s reverse proxy can be used to proxy our container’s services, which can be accessed by directly accessing the domain name
Scheme 2: Use ingress-nginx to perform reverse proxy processing
Install ingress-nginx on k8S:
Nginx-ingress-controller official documentation
# curl -O https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml
Copy the codeDownload the deployment file from above
2, need to modify the file through VIm
## 找到文件Deployment的位置，将类型kind改成DaemonSet
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment ==> DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
hostNetwork: true # 在spec下设置hostNetwork设置为true，如果没有就自行添加，绑定主机网络，先保证物理机上的80和443没有被占用
containers:
- name: controller
image: k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a # 因为k8s.gcr.io是google的服务器，如果有科学上网则不需要改image
# image: image: registry.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
此处的两个也是同样的替换镜像文件，科学上网忽略
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
# image: registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.0
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
# image: registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.0
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
Copy the codeAfter the configuration file is modified, run commands to install it
# kubect apply -f deploy.yaml
Copy the codeHow do I determine the installation is complete
RESTARTS AGE ingress-nginx NAME RESTARTS AGE ingress-nginx-admission create-r2fdf 0/1 kubectl get pod -n ingress-nginx NAME RESTARTS AGE ingress-nginx-admission create-r2fdf 0/1 Completed 0 3h9m ingress-nginx-admission-patch-pw84m 0/1 Completed 1 3h9m ingress-nginx-controller-6dt7g 1/1 Running 0 Ingress-nginx-controller-xxxx is Running and READY is 1/1Copy the codeAt this point, you access port 80 on the target machine
We then create a Tomcat deployment service under default :tomcat-deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: tomcat
namespace: default
labels:
app: tomcat
annotations:
deployment.kubernetes.io/revision: '1'
kubesphere.io/creator: admin
spec:
replicas: 1
selector:
matchLabels:
app: tomcat
template:
metadata:
creationTimestamp: null
labels:
app: tomcat
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
containers:
- name: container-b0zpfy
image: 'tomcat:jre8'
ports:
- name: tcp-8080
containerPort: 8080
protocol: TCP
resources: {}
volumeMounts:
- name: host-time
readOnly: true
mountPath: /etc/localtime
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
securityContext: {}
affinity: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
Copy the codeCreate a Service Service for tomcat: tomcat-svc.yaml
kind: Service
apiVersion: v1
metadata:
name: tomcat
namespace: default
labels:
app: tomcat
annotations:
kubesphere.io/creator: admin
spec:
ports:
- name: http-8080
protocol: TCP
port: 8080
targetPort: 8080
selector:
app: tomcat
type: ClusterIP
sessionAffinity: None
Copy the codeCreate a tomcat-ingress.yaml file
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-app-ingress-tomcat
spec:
rules:
- host: www.tomcat1.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tomcat
port:
number: 8080
ingressClassName: nginx
Copy the codeKubectl apply-f is executed separately in the above order to launch the files
# kubectl get pods -n default NAME READY STATUS RESTARTS AGE tomcat-54d6dc4796-llrzl 1/1 Running 0 3h7m # kubectl get SVC -n default NAME TYPE cluster-ip external-ip PORT(S) AGE Tomcat ClusterIP 10.233.22.156 < None > 8080/TCP 5M59s # kubectl get ing -n default NAME CLASS HOSTS ADDRESS PORTS AGE nginx-app-ingress-tomcat nginx www.tomcat2.com 80, 443 2m47sCopy the codeRun the preceding three commands to check whether the service is started
After the startup was completed, I saw that the HOSTS of ingress was www.tomcat2.com, because there was no domain name I, so I changed the local host to do the mapping
Then access the domain name through your browser
The current page is displayed, this is not the same as the nginx style exception page, is tomcat 404, indicating that we successfully accessed the Tomcat service through the domain name
At this point, both schemes are demonstrated
extension
1. Ingress-nginx adds HTTPS certificate non-self-signed certificate, puts the certificate on the server, and generates certificate secret from the command line
--from-file=tls. CRT = path where the CRT key is located --from-file=tls.key= path where the key is located # kubectl -n demo-echo create secret generic www-tomcat-com --from-file=tls.crt=server.crt --from-file=tls.key=server.keyCopy the codeThen add the TLS configuration in the tomcat-ingress.yaml file
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-app-ingress spec: rules: - host: www.tomcat.com http: paths: - path: / pathType: Prefix backend: service: name: tomcat port: number: 8080 tls: - hosts: SecretName: WWW -tomcat-com # The name of the secret key ingressClassName: nginxCopy the codeThe deployment will then be able to access the service via HTTPS, which will not be shown in screenshots.
The generation scheme of self-signed certificate is attached:
Echo "Generate a self-signed CA certificate" openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca. CRT -days 3560 -nodes -subj '/CN=My Cert Authority' echo "openssl req -new-newkey rsa:4096 -keyout server.key -out Csr-nodes -subj '/CN=www.tomcat.com' Here /CN= domain name Write your own domain name openssl x509 -req -sha256 -days 3650 -in server.csr-ca ca.crt -CAkey ca.key -set_serial 01 -out server.crtCopy the code2. During the operation, I tried to access services in different namespaces, but the execution failed
Error from server (InternalError): error when creating "tomcat-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": the server rejected our request for an unknown reason
Copy the codeThis exception is currently given by removing ingress-nginx-admission
# # # to perform the following command kubectl delete - A ValidatingWebhookConfiguration ingress - nginx - admissionCopy the codeCan be normal, the specific reason should be because of the permission problem
conclusion
There are two kinds of problems with k8S requiring port access
Personally, I prefer the first one, although the second one is elegant, but the first one is relatively conventional, suitable for most scenarios, good operation and maintenance can maintain these things, of course, if it is not poor money, the most recommended LoadBalance of course, which saves the trouble of operation and maintenance (hair less 🙂 smile).
Welcome to teach each other, exchange learning. If it is helpful, please send a “like”
Reference links:
Resolve ingress cross-namespace access
Set the self-signed certificate and ingress HTTPS deployment
Ingress-nginx deploys port 80/443