Ingress Nginx support Progressive Delivery

Progressive delivery is a practice that allows organizations to control how and when new software features or changes are delivered. It builds on the capabilities and practices of feature flag management and deployment strategies like blue-green and canary deployments. Ultimately, progressive delivery combines software development and delivery practices allowing organizations to deliver with control.

• Canary Release

• Blue-Green Deployment

• A/B Testing

• Gray Release

• Dark Launch

• …

In some cases, you may want to “canary” a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: "true" is set:

• nginx.ingress.kubernetes.io/canary-by-header: The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always, it will be routed to the canary. When the header is set to never, it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence.

• nginx.ingress.kubernetes.io/canary-by-header-value: The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn’t have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined.

• nginx.ingress.kubernetes.io/canary-by-header-pattern: This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching.

• nginx.ingress.kubernetes.io/canary-by-cookie: The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always, it will be routed to the canary. When the cookie is set to never, it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence.

nginx.ingress.kubernetes.io/canary-weight: The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total.

nginx.ingress.kubernetes.io/canary-weight-total: The total weight of traffic. If unspecified, it defaults to 100.

Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight

Examples

Canary Release

Version 1

manifests/Deployment-1.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# manifests/Deployment-1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-1
  labels:
    app: nginx-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-1
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

1

$ kubectl apply -f manifests/Deployment-1.yaml

manifests/Service-1.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# manifests/Service-1.yaml

apiVersion: v1
kind: Service
metadata:
  name: service-1
spec:
  ports:
    - name: '80'
      protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: nginx-1

1

$ kubectl apply -f manifests/Service-1.yaml

manifests/Ingress-1.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# manifests/Ingress-1.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-1
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: nginx
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-1
                port:
                  number: 80

1

$ kubectl apply -f manifests/Ingress-1.yaml

Version canary

manifests/Deployment-canary.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# manifests/Deployment-canary.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-canary
  labels:
    app: nginx-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-canary
  template:
    metadata:
      labels:
        app: nginx-canary
    spec:
      containers:
      - name: nginx
        image: nginx:1.21.4
        ports:
        - containerPort: 80

1

$ kubectl apply -f manifests/Deployment-canary.yaml

manifests/Service-canary.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13

# manifests/Service-canary.yaml

apiVersion: v1
kind: Service
metadata:
  name: service-canary
  ports:
    - name: '80'
      protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: nginx-canary

1

$ kubectl apply -f manifests/Service-canary.yaml

manifests/Ingress-2.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# manifests/Ingress-canary.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-canary
  annotations:
    kubernetes.io/ingress.class: nginx
    
    nginx.ingress.kubernetes.io/canary: 'true'
    nginx.ingress.kubernetes.io/canary-by-header: Canary
spec:
  rules:
    - host: nginx
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-2
                port:
                  number: 80

1

$ kubectl apply -f manifests/Ingress-canary.yaml

1
2
3
4
5

# curl Verion 1
$ curl ngi

# curl Version canary
$ curl -H "Canary: always" nginx

Others

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# manifests/Ingress-canary.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-canary
  annotations:
    kubernetes.io/ingress.class: nginx

    nginx.ingress.kubernetes.io/canary: "true"

    # # canary-by-header
    # -H "Region: au" or -H "Region: au"
    # nginx.ingress.kubernetes.io/canary-by-header: "Region"
    # nginx.ingress.kubernetes.io/canary-by-header-value: "au"

    # # canary-by-header
    # -H "Region: au"
    # nginx.ingress.kubernetes.io/canary-by-header: "Region"
    # nginx.ingress.kubernetes.io/canary-by-header-pattern: "au|us"

    # # canary-by-cookie
    # # --cookie "user_from_au=always"
    # nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_au"

    # # canary-weight
    # nginx.ingress.kubernetes.io/canary: "true"
    # nginx.ingress.kubernetes.io/canary-weight: "10"

Known Limitations

• Not support automatic failover. Even if the traffic is completely cut to Canary Ingress, the old version of the service must still exist, otherwise an error 503 will be reported.

• Only one Canary Ingress of the same service can be defined, so the back-end service supports up to two versions.

• The domain name must be configured in Ingress, otherwise it will not be effective.

• All the other non-canary annotations will be ignored. Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance, nginx.ingress.kubernetes.io/upstream-hash-by, and annotations related to session affinity. If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition.

• Currently a maximum of one canary ingress can be applied per Ingress rule.

References

[1] Annotations - NGINX Ingress Controller - https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary

[2]] Welcome - NGINX Ingress Controller - https://kubernetes.github.io/ingress-nginx/

Ruby 3.0.2 has been released.

This release includes security fixes. Please check the topics below for details.

• CVE-2021-31810: Trusting FTP PASV responses vulnerability in Net::FTP - https://www.ruby-lang.org/en/news/2021/07/07/trusting-pasv-responses-in-net-ftp/

• CVE-2021-32066: A StartTLS stripping vulnerability in Net::IMAP - https://www.ruby-lang.org/en/news/2021/07/07/starttls-stripping-in-net-imap/

• CVE-2021-31799: A command injection vulnerability in RDoc - https://www.ruby-lang.org/en/news/2021/05/02/os-command-injection-in-rdoc/

bundler-leak

The best tool to find leaky gems in your dependencies. Make sure memory leaks are not in your gem dependencies.

faraday

Faraday is an HTTP client library that provides a common interface over many adapters (such as Net::HTTP) and embraces the concept of Rack middleware when processing the request/response cycle.

working_hours

A modern ruby gem allowing to do time calculation with working hours.

ruby-git

The Git Gem provides an API that can be used to create, read, and manipulate Git repositories by wrapping system calls to the git binary. The API can be used for working with Git in complex interactions including branching and merging, object inspection and manipulation, history, patch generation and more.

Exception: Zlib::DataError: unknown compression method

Recently encountered a problem when we used rest-client, faraday gem or other HTTP client libraries to post data to WeChat Pay Unifiedorder API - https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=9_1.

RVM( or Rbenv), RubyGems and Bundler

RVM( or Rbenv), RubyGems, and Bundler work together to give us a lot of control over our code’s environment. If you know how they work, you’ll be better prepared to troubleshoot any problems you encounter.

Docker Desktop, Docker Compose and Kubernetes (K8S) on macOS

Docker Desktop is an easy-to-install application for your Mac or Windows environment that enables you to build and share containerized applications and microservices. Docker Desktop includes Docker Engine, Docker CLI client, Docker Compose, Docker Content Trust, Kubernetes, and Credential Helper.

update-kube-cert

update-kube-cert is used to extended kubernetes cluster certificates that have expired or are about to expire.

This script is suitable for all k8s version cluster certificate update (using kubeadm initialized cluster).