一、ingress-nginx-controller ConfigMap

官方链接：

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

将下述K/V配置项插入到 ingress-nginx 的 configMap 里的 data 对象下。

# 在 values.yaml 文件中修改config添加参数
load-balance: "round_robin"

1.1 优化配置

可以优化配置如下：

# 负载工作机制,轮询
load-balance: "round_robin"

# 错误日志等级设置 (debug, info, notice, warn, error, crit, alert, or emerg)
error-log-level: "notice"

# 启用Gzip资源压缩 (3k以上)
use-gzip: "true"
gzip-level: "2"
gzip-min-length: "3072"
gzip-types: "text/html text/plain text/css text/javascript application/javascript application/x-javascript application/xml application/x-httpd-php application/x-font-ttf application/json image/x-icon image/svg+xml image/avif image/webp font/ttf font/opentype"
# 不建议进行照片压缩 image/jpeg image/gif image/png 可能反而会增加其体积

# 启用Brotli资源压缩（同等条件下优于Gzip，任选一个）
# 参考:https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-brotli
enable-brotli: "true"
brotli-level: 5
brotli-types: "text/plain text/css text/javascript application/javascript application/x-javascript application/xml application/x-httpd-php application/x-font-ttf image/x-icon  image/svg+xml image/avif image/webp font/ttf font/opentype"
# 不建议进行照片压缩 image/jpeg image/gif image/png 可能反而会增加其体积

# 启用http2支持(实际上默认是开启的，如果需要关闭请将其设置为true)
use-http2: "true"

# ssl 会话复用
ssl_session_cache: "shared:SSL:10m;"
ssl-session-timeout: "10m"

# worker每个工作进程可以打开的最大文件数与同时打开最大连接数设置
# 参考: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-connections
worker-processes: "auto"
max-worker-open-files: "10240"
#默认值16384
max-worker-connections: "32767"

# 连接复用
enable-multi-accept: "true"

# keep-alive连接超时和最大请求数调整
keep-alive: "75"
keep-alive-requests: "10000"

# 参考: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#upstream-keepalive-connections

# upstream-keepalive 与上游Pod连接超时与最大请求数调整
upstream-keepalive-time: "30m"
upstream-keepalive-timeout: "60"
upstream-keepalive-requests: "10000"
upstream-keepalive-connections: "512"

# 参考：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-read-timeout
# 参考：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-send-timeout
# proxy-connect 设置 ingress-nginx 与 upstream pod 之间连接请求超时实践。
# 默认值60s
proxy-connect-timeout: "1"

# 设置将请求传输到代理服务器的超时时间（以秒为单位）（超时仅在两个连续的写操作之间设置，而不是为整个请求的传输设置）
proxy-send-timeout: "3"

# 设置从代理服务器读取响应的超时时间（以秒为单位）
proxy-read-timeout: "3"

下面以优化负载工作机制为例进行演示，其他相同

# 修改values.yaml文件，添加优化配置信息
[root@master01 ~]# cd /root/6/ingress-nginx
[root@master01 ingress-nginx]# vim values.yaml
# 找到第42行，去掉第42行的{}，并在第42行下面添加如下配置信息
  config:
    load-balance: "round_robin"

# 配置生效
[root@master01 ingress-nginx]# helm upgrade ingress-nginx -f values.yaml -n ingress-nginx .

# 补充：最新修改完成后完整配置文件
[root@master01 ingress-nginx]# egrep -v "^$|#" values.yaml
commonLabels: {}
controller:
  name: controller
  image:
    chroot: false
    registry: registry.cn-hangzhou.aliyuncs.com
    image: google_containers/nginx-ingress-controller
    tag: "v1.7.0"
    digestChroot: sha256:e84ef3b44c8efeefd8b0aa08770a886bfea1f04c53b61b4ba9a7204e9f1a7edc
    pullPolicy: IfNotPresent
    runAsUser: 101
    allowPrivilegeEscalation: true
  existingPsp: ""
  containerName: controller
  containerPort:
    http: 80
    https: 443
  config:
    load-balance: "round_robin"
  configAnnotations: {}
  proxySetHeaders: {}
  addHeaders: {}
  dnsConfig: {}
  hostname: {}
  dnsPolicy: ClusterFirstWithHostNet
  reportNodeInternalIp: false
  watchIngressWithoutClass: false
  ingressClassByName: false
  enableTopologyAwareRouting: false
  allowSnippetAnnotations: true
  hostNetwork: true
  hostPort:
    enabled: false
    ports:
      http: 80
      https: 443
  electionID: ""
  ingressClassResource:
    name: nginx
    enabled: true
    default: false
    controllerValue: "k8s.io/ingress-nginx"
    parameters: {}
  ingressClass: nginx
  podLabels: {}
  podSecurityContext: {}
  sysctls: {}
  publishService:
    enabled: true
    pathOverride: ""
  scope:
    enabled: false
    namespace: ""
    namespaceSelector: ""
  configMapNamespace: ""
  tcp:
    configMapNamespace: ""
    annotations: {}
  udp:
    configMapNamespace: ""
    annotations: {}
  maxmindLicenseKey: ""
  extraArgs: {}
  extraEnvs: []
  kind: DaemonSet
  annotations: {}
  labels: {}
  updateStrategy: {}
  minReadySeconds: 0
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  terminationGracePeriodSeconds: 300
  nodeSelector:
    kubernetes.io/os: linux
    ingress: "true"
  livenessProbe:
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 5
  readinessProbe:
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3
  healthCheckPath: "/healthz"
  healthCheckHost: ""
  podAnnotations: {}
  replicaCount: 1
  minAvailable: 1
  resources:
    requests:
      cpu: 100m
      memory: 90Mi
  autoscaling:
    apiVersion: autoscaling/v2
    enabled: false
    annotations: {}
    minReplicas: 1
    maxReplicas: 11
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
    behavior: {}
  autoscalingTemplate: []
  keda:
    apiVersion: "keda.sh/v1alpha1"
    enabled: false
    minReplicas: 1
    maxReplicas: 11
    pollingInterval: 30
    cooldownPeriod: 300
    restoreToOriginalReplicaCount: false
    scaledObject:
      annotations: {}
    triggers: []
    behavior: {}
  enableMimalloc: true
  customTemplate:
    configMapName: ""
    configMapKey: ""
  service:
    enabled: true
    appProtocol: true
    annotations: {}
    labels: {}
    externalIPs: []
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    enableHttp: true
    enableHttps: true
    ipFamilyPolicy: "SingleStack"
    ipFamilies:
      - IPv4
    ports:
      http: 80
      https: 443
    targetPorts:
      http: http
      https: https
    type: LoadBalancer
    nodePorts:
      http: ""
      https: ""
      tcp: {}
      udp: {}
    external:
      enabled: true
    internal:
      enabled: false
      annotations: {}
      loadBalancerSourceRanges: []
  shareProcessNamespace: false
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  extraInitContainers:
  - name: sysctl
    image: registry.cn-hangzhou.aliyuncs.com/abroad_images/alpine:3.10
    imagePullPolicy: IfNotPresent
    command:
      - sh
      - -c
      - |
        mount -o remount rw /proc/sys
        sysctl -w net.core.somaxconn=65535
        sysctl -w net.ipv4.tcp_tw_reuse=1
        sysctl -w net.ipv4.ip_local_port_range="1024 65535"
        sysctl -w fs.file-max=1048576
        sysctl -w fs.inotify.max_user_instances=16384
        sysctl -w fs.inotify.max_user_watches=524288
        sysctl -w fs.inotify.max_queued_events=16384
    securityContext:
      privileged: true
  extraModules: []
  opentelemetry:
    enabled: false
    image: registry.k8s.io/ingress-nginx/opentelemetry:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:40f766ac4a9832f36f217bb0e98d44c8d38faeccbfe861fbc1a76af7e9ab257f
    containerSecurityContext:
      allowPrivilegeEscalation: false
  admissionWebhooks:
    annotations: {}
    enabled: true
    extraEnvs: []
    failurePolicy: Fail
    port: 8443
    certificate: "/usr/local/certificates/cert"
    key: "/usr/local/certificates/key"
    namespaceSelector: {}
    objectSelector: {}
    labels: {}
    existingPsp: ""
    networkPolicyEnabled: false
    service:
      annotations: {}
      externalIPs: []
      loadBalancerSourceRanges: []
      servicePort: 443
      type: ClusterIP
    createSecretJob:
      securityContext:
        allowPrivilegeEscalation: false
      resources: {}
    patchWebhookJob:
      securityContext:
        allowPrivilegeEscalation: false
      resources: {}
    patch:
      enabled: true
      image:
        registry: registry.cn-hangzhou.aliyuncs.com
        image: google_containers/kube-webhook-certgen
        tag: v20230312-helm-chart-4.5.2-28-g66a760794
        pullPolicy: IfNotPresent
      priorityClassName: ""
      podAnnotations: {}
      nodeSelector:
        kubernetes.io/os: linux
      tolerations: []
      labels: {}
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
        fsGroup: 2000
    certManager:
      enabled: false
      rootCert:
        duration: ""
      admissionCert:
        duration: ""
  metrics:
    port: 10254
    portName: metrics
    enabled: false
    service:
      annotations: {}
      labels: {}
      externalIPs: []
      loadBalancerSourceRanges: []
      servicePort: 10254
      type: ClusterIP
    serviceMonitor:
      enabled: false
      additionalLabels: {}
      namespace: ""
      namespaceSelector: {}
      scrapeInterval: 30s
      targetLabels: []
      relabelings: []
      metricRelabelings: []
    prometheusRule:
      enabled: false
      additionalLabels: {}
      rules: []
  lifecycle:
    preStop:
      exec:
        command:
          - /wait-shutdown
  priorityClassName: ""
revisionHistoryLimit: 10
defaultBackend:
  enabled: false
  name: defaultbackend
  image:
    registry: registry.k8s.io
    image: defaultbackend-amd64
    tag: "1.5"
    pullPolicy: IfNotPresent
    runAsUser: 65534
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
  existingPsp: ""
  extraArgs: {}
  serviceAccount:
    create: true
    name: ""
    automountServiceAccountToken: true
  extraEnvs: []
  port: 8080
  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
  readinessProbe:
    failureThreshold: 6
    initialDelaySeconds: 0
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5
  updateStrategy: {}
  minReadySeconds: 0
  tolerations: []
  affinity: {}
  podSecurityContext: {}
  containerSecurityContext: {}
  podLabels: {}
  nodeSelector:
    kubernetes.io/os: linux
  podAnnotations: {}
  replicaCount: 1
  minAvailable: 1
  resources: {}
  extraVolumeMounts: []
  extraVolumes: []
  autoscaling:
    apiVersion: autoscaling/v2
    annotations: {}
    enabled: false
    minReplicas: 1
    maxReplicas: 2
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
  service:
    annotations: {}
    externalIPs: []
    loadBalancerSourceRanges: []
    servicePort: 80
    type: ClusterIP
  priorityClassName: ""
  labels: {}
rbac:
  create: true
  scope: false
podSecurityPolicy:
  enabled: false
serviceAccount:
  create: true
  name: ""
  automountServiceAccountToken: true
  annotations: {}
imagePullSecrets: []
tcp: {}
udp: {}
portNamePrefix: ""
dhParam: ""

ingress-nginx 资源查看

# 查看 Ingress-nginx全局配置参数，观察到优化配置已经成功添加
[root@master01 ingress-nginx]#kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
...
...
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  load-balance: round_robin
...
...

Nginx全局配置通过 configmap 配置(Nginx Ingress Controller 会 watch 并自动 reload 配置)。

1.2 安全配置

ModSecurity (http://modsecurity.org/) 是一个开源的Web Application防火墙, 可以为 一组特定的入口位置启用它。

必须首先通过在 ConfigMap 中启用 ModSecurity 来启用 ModSecurity 模块。

建议的配置参考 ：https://github.com/SpiderLabs/ModSecurity/blob/v3/master/modsecurity.conf-recommended

#启用ModSecurity waf模块拦截常规web攻击
enable-modsecurity: "true"

1.3 分布式跟踪

参考：https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/o pentracing/

启用NGINX服务的请求，通过OpenTracing项目进行分布式跟踪。

使用第三方模块opentracing-contrib/nginx-opentracing，NGINX 入口控制器可以配置 NGINX 以启用OpenTracing检测。 默认情况下，此功能处于禁用状态。

用法：

要启用检测，我们必须在配置 ConfigMap 中启用 OpenTracing：

data:
  enable-opentracing: "true"

要为单个 Ingress 启用或禁用检测，请使用 enable-opentracing注解：

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentracing: "true"

1.4 总结

  • 在 Nginx 中的 upstream 主要是配置均衡池和调度方法.

  • 在 Nginx 中的 proxy_pass 主要是配置代理服务器ip或服务器组的名字.

  • 常用的压缩算法是 gzip（Ingress-nginx也是默认使用gzip)，据说brotli要比gzip高 出20％至30％的压缩率。默认的压缩算法是gzip，压缩级别为1，如需要启用 brotli，需要配置以下三个参数：

  • enable-brotli: true 或 false，是否启用brotli压缩算法。

  • brotli-level: 压缩级别，范围1~11，默认为4，级别越高，越消耗CPU性能。

  • brotli-types: 由brotli即时压缩的MIME类型。

  • 在高并发场景下，我们需配置upstream-keepalive-*相关参数, 使得 nginx 尽可 能快速处理 HTTP 请求（尽量少释放并重建 TCP 连接)，同时控制 nginx 内存使用量。

  • Ingress nginx 与 upstream pod 建立TCP连接并进行通信，其中涉及3个超时配 置我们需要重点关注。
  • proxy-read-timeout / proxy-send-timeout： 设置 nginx 与 upstream pod 之 间读操作的超时时间，默认设置为 60s，当业务方服务异常导致响应耗时飙涨时，异常请求会长时间夯住ingress网关，我们在拉取所有服务正常请求的 P99.99 耗时之后，将网关与 upstream pod 之间读写超时均缩短到3s，使得nginx可以及时掐断异常请求，避免长时间被夯住。
  • proxy-connect-timeout： 设置 nginx 与 upstream pod 连接建立的超时时间，默认设置为 5s，由于在 nginx 和业务均在内网同机房通信，我们将此超时时间缩短到1s。