[Ingress Nginx] Ingress Nginx support Progressive Delivery: Canary Release, Blue-Green Deployment or A/B Testing

Posted on Edited on In , , Views: Word count in article: 7.1k Reading time ≈ 6 mins.

Ingress Nginx support Progressive Delivery

Progressive delivery is a practice that allows organizations to control how and when new software features or changes are delivered. It builds on the capabilities and practices of feature flag management and deployment strategies like blue-green and canary deployments. Ultimately, progressive delivery combines software development and delivery practices allowing organizations to deliver with control.

  • Canary Release

  • Blue-Green Deployment

  • A/B Testing

  • Gray Release

  • Dark Launch

  • …

In some cases, you may want to “canary” a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: "true" is set:

  • nginx.ingress.kubernetes.io/canary-by-header: The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always, it will be routed to the canary. When the header is set to never, it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence.

  • nginx.ingress.kubernetes.io/canary-by-header-value: The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn’t have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined.

  • nginx.ingress.kubernetes.io/canary-by-header-pattern: This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching.

  • nginx.ingress.kubernetes.io/canary-by-cookie: The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always, it will be routed to the canary. When the cookie is set to never, it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence.

nginx.ingress.kubernetes.io/canary-weight: The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total.

nginx.ingress.kubernetes.io/canary-weight-total: The total weight of traffic. If unspecified, it defaults to 100.

Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight

Examples

Canary Release

Version 1

manifests/Deployment-1.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# manifests/Deployment-1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-1
  labels:
    app: nginx-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-1
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
1
$ kubectl apply -f manifests/Deployment-1.yaml
manifests/Service-1.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# manifests/Service-1.yaml

apiVersion: v1
kind: Service
metadata:
  name: service-1
spec:
  ports:
    - name: '80'
      protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: nginx-1
1
$ kubectl apply -f manifests/Service-1.yaml
manifests/Ingress-1.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# manifests/Ingress-1.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-1
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: nginx
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-1
                port:
                  number: 80
1
$ kubectl apply -f manifests/Ingress-1.yaml

Version canary

manifests/Deployment-canary.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# manifests/Deployment-canary.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-canary
  labels:
    app: nginx-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-canary
  template:
    metadata:
      labels:
        app: nginx-canary
    spec:
      containers:
      - name: nginx
        image: nginx:1.21.4
        ports:
        - containerPort: 80
1
$ kubectl apply -f manifests/Deployment-canary.yaml
manifests/Service-canary.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
# manifests/Service-canary.yaml

apiVersion: v1
kind: Service
metadata:
  name: service-canary
  ports:
    - name: '80'
      protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: nginx-canary
1
$ kubectl apply -f manifests/Service-canary.yaml
manifests/Ingress-2.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# manifests/Ingress-canary.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-canary
  annotations:
    kubernetes.io/ingress.class: nginx
    
    nginx.ingress.kubernetes.io/canary: 'true'
    nginx.ingress.kubernetes.io/canary-by-header: Canary
spec:
  rules:
    - host: nginx
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-2
                port:
                  number: 80
1
$ kubectl apply -f manifests/Ingress-canary.yaml
1
2
3
4
5
# curl Verion 1
$ curl ngi

# curl Version canary
$ curl -H "Canary: always" nginx

Others

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# manifests/Ingress-canary.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-canary
  annotations:
    kubernetes.io/ingress.class: nginx

    nginx.ingress.kubernetes.io/canary: "true"

    # # canary-by-header
    # -H "Region: au" or -H "Region: au"
    # nginx.ingress.kubernetes.io/canary-by-header: "Region"
    # nginx.ingress.kubernetes.io/canary-by-header-value: "au"

    # # canary-by-header
    # -H "Region: au"
    # nginx.ingress.kubernetes.io/canary-by-header: "Region"
    # nginx.ingress.kubernetes.io/canary-by-header-pattern: "au|us"

    # # canary-by-cookie
    # # --cookie "user_from_au=always"
    # nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_au"

    # # canary-weight
    # nginx.ingress.kubernetes.io/canary: "true"
    # nginx.ingress.kubernetes.io/canary-weight: "10"

Known Limitations

  • Not support automatic failover. Even if the traffic is completely cut to Canary Ingress, the old version of the service must still exist, otherwise an error 503 will be reported.

  • Only one Canary Ingress of the same service can be defined, so the back-end service supports up to two versions.

  • The domain name must be configured in Ingress, otherwise it will not be effective.

  • All the other non-canary annotations will be ignored. Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance, nginx.ingress.kubernetes.io/upstream-hash-by, and annotations related to session affinity. If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition.

  • Currently a maximum of one canary ingress can be applied per Ingress rule.

References

[1] Annotations - NGINX Ingress Controller - https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary

[2]] Welcome - NGINX Ingress Controller - https://kubernetes.github.io/ingress-nginx/