From 5152c165179c672778f19b6f97a37c405eaf64c6 Mon Sep 17 00:00:00 2001 From: Martin Hickey Date: Fri, 15 May 2020 14:05:17 +0000 Subject: [PATCH 1/3] Add deprecated Kubernetes API doc Signed-off-by: Martin Hickey --- docs/README.md | 1 + docs/kubernetes_apis.md | 149 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 docs/kubernetes_apis.md diff --git a/docs/README.md b/docs/README.md index ed13cc22a..37c1c6346 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,6 +2,7 @@ - [Quick Start](quickstart.md) - Read me first! - [Installing Helm](install.md) - Install Helm and Tiller + - [Deprecated Kubernetes APIs](kubernetes_apis.md) - [Kubernetes Distribution Notes](kubernetes_distros.md) - [Frequently Asked Questions](install_faq.md) - [Using Helm](using_helm.md) - Learn the Helm tools diff --git a/docs/kubernetes_apis.md b/docs/kubernetes_apis.md new file mode 100644 index 000000000..405be6ef6 --- /dev/null +++ b/docs/kubernetes_apis.md @@ -0,0 +1,149 @@ +# Deprecated Kubernetes APIs + +Kubernetes is an API-driven system and the API evolves over time to reflect +the evolving understanding of the problem space. This is common practice +across systems and their APIs. An important part of evolving APIs is a good +deprecation policy and process to inform users of how changes to APIs are +implemented. In other words, consumers of your API need to know in advance and +in what release an API will be removed or changed. This removes the element of +surprise and breaking changes to consumers. + +The [Kubernetes deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) +documents how Kubernetes handles the changes to its API versions. The policy +for deprecation states the timeframe that API versions will be supported following +a deprecation announcement. It is therefore important to be aware of deprecation +announcements and know when API versions will be removed, to help minimize the +effect. + +This is an example of an announcement [for the removal of deprecated API versions in Kubernetes 1.16](https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/) +and was advertised a few months prior to the release. These API versions would +have been announced for deprecation prior to this again. This shows that there +is a good policy in place which informs consumers of API version support. + +Helm templates specify a [Kubernetes API group](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups) +when defining a Kubernetes object, similar to a Kubernetes manifest file. It is +specified in the `apiVersion` field of the template and it identifies the API +version of the Kubernetes object. This means that Helm users and chart maintainers +need to be aware when Kubernetes API versions have been deprecated and in what +Kubernetes version they will be removed. + +## Chart Maintainers + +You should audit your charts checking for Kubernetes API versions that are +deprecated or are removed in a Kubernetes version. The API versions found as +due to be or that are now out of support, should be updated to the supported +version and a new version of the chart released. The API version is defined by +the `kind` and `apiVersion` fields. For example, here is a removed `Deployment` +object API version in Kubernetes 1.16: + +```yaml +apiVersion: apps/v1beta1 +kind: Deployment +``` + +## Helm Users + +You should audit the charts that you use (similar to [chart maintainers](#chart-maintainers)) +and identify any charts where API versions are deprecated or removed in a +Kubernetes version. For the charts identified, you need to check for the latest +version of the chart (which has supported API versions) or update the chart +yourself. + +Additionally, you also need to audit any charts deployed (i.e. Helm releases) +checking again for any deprecated or removed API versions. This can be done by +getting details of a release using the `helm get manifest` command. + +The means for updating a Helm release to supported APIs depends on your findings +as follows: + +1. If you find deprecated API versions only then: + - Perform a `helm upgrade` with a version of the chart with supported Kubernetes + API versions + - Add a description in the upgrade, something along the lines to not perform a + rollback to a Helm version prior to this current version +2. If you find any API version(s) that is/are removed in a Kubernetes version +then: + - If you are running a Kubernetes version where the API version(s) are still + available (for example, you are on Kubernetes 1.15 and found you use APIs that + will be removed in Kubernetes 1.16): + - Follow the step 1 procedure + - Otherwise (for example, you are already running a Kubernetes version where + some API versions reported by `helm get manifest` are no longer available): + - You need to edit the release manifest that is stored in the cluster to + update the API versions to supported APIs. See + [Updating API Versions of a Release Manifest](#updating-api-versions-of-a-release-manifest) + for more details + +> Note: In all cases of updating a Helm release with supported APIs, you should +never rollback the release to a version prior to the release version with the +supported APIs. + +> Recommendation: The best practice is to upgrade releases using deprecated API +versions to supported API versions, prior to upgrading to a kubernetes cluster +that removes those API versions. + +If you don't update a release as suggested previously, you will have an error +similar to the following when trying to upgrade a release in a Kubernetes version +where its API version(s) is/are removed: + +``` +Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s) +for this kubernetes version and it is therefore unable to build the kubernetes +objects for performing the diff. error from kubernetes: unable to recognize "": +no matches for kind "Deployment" in version "apps/v1beta1" +``` + +Helm fails in this scenario because it attempts to create a diff patch between +the current deployed release (which contains the Kubernetes APIs that are removed +in this Kubernetes version) against the chart you are passing with the +updated/supported API versions. The underlying reason for failure is that when +Kubernetes removes an API version, the Kubernetes Go client library can no longer +parse the deprecated objects and Helm therefore fails when calling the library. +Helm unfortunately is unable to recover from this situation and is no longer able +to manage such a release. +See [Updating API Versions of a Release Manifest](#updating-api-versions-of-a-release-manifest) +for more details on how to recover from this scenario. + +## Updating API Versions of a Release Manifest + +The manifest is a property of the Helm release object which is stored in the data +field of a ConfigMap (default) or Secret in the cluster. The data field contains +a gzipped [protobuf object](developers#grpc-and-protobuf) which is base 64 +encoded (there is an additional base 64 encoding for a Secret). There is +a Secret/ConfigMap per release version/revision in the namespace of the release. + +You can use the Helm [mapkubeapis](https://github.com/hickeyma/helm-mapkubeapis) +plugin to perform the update of a release to supported APIs. Check out the +readme for more details. + +Alternatively, you can follow these manual steps to perform an update of the API +versions of a release manifest: + +- Prerequsiities: + - HELM_PROTOBUF_SCHEMA: [Helm protobuf schema](https://github.com/helm/helm/tree/dev-v2/_proto) + - PROTOBUF_SCHEMA: [Protobuf base schema](https://github.com/protocolbuffers/protobuf/tree/master/src) +- Get the name of the latest deployed release: + - ConfigMap backend: `kubectl get configmap -l OWNER=TILLER,STATUS=DEPLOYED,NAME= --namespace | awk '{print $1}' | grep -v NAME` + - Secrets backend: `kubectl get secret -l OWNER=TILLER,STATUS=DEPLOYED,NAME= --namespace | awk '{print $1}' | grep -v NAME` +- Get latest deployed release details: + - ConfigMap backend: `kubectl get configmap -n -o yaml > release.yaml` + - Secrets backend: `kubectl get secret -n -o yaml > release.yaml` +- Backup the release incase you need to restore if something goes wrong: + - `cp release.yaml release.bak` + - In case of emergency, restore: `kubectl apply -f release.bak -n ` +- Decode the release object: + - ConfigMap backend: `cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | gzip -d | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --decode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* > release.data.decoded` + - Secrets backend:`cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --decode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* > release.data.decoded` +- Change API versions of the manifests. Can use any tool (e.g. editor) to make +the changes. This is in the `manifest` field of your decoded release +object (`release.data.decoded`) +- Encode the release object: + - ConfigMap backend: `cat release.data.decoded | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --encode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* | gzip | base64 --wrap 0` + - Secrets backend: `cat release.data.decoded | protoc --proto_path ${HELM_PROTOBUF_SCHEMA} --proto_path ${PROTOBUF_SCHEMA} --encode hapi.release.Release ${HELM_PROTOBUF_SCHEMA}/hapi/**/* | gzip | base64 | base64 --wrap 0` +- Replace `data.release` property value in the deployed release file (`release.yaml`) +with the new encoded release object +- Apply file to namespace: `kubectl apply -f release.yaml -n ` +- Perform a `helm upgrade` with a version of the chart with supported Kubernetes +API versions +- Add a description in the upgrade, something along the lines to not perform a +rollback to a Helm version prior to this current version From 46807c6729708973b1115242297fc702334eb1b4 Mon Sep 17 00:00:00 2001 From: Martin Hickey Date: Wed, 20 May 2020 10:30:49 +0000 Subject: [PATCH 2/3] Update review comment Review comment: - https://github.com/helm/helm/pull/8135#issuecomment-629527051 Signed-off-by: Martin Hickey --- docs/kubernetes_apis.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/kubernetes_apis.md b/docs/kubernetes_apis.md index 405be6ef6..785859a92 100644 --- a/docs/kubernetes_apis.md +++ b/docs/kubernetes_apis.md @@ -119,7 +119,7 @@ readme for more details. Alternatively, you can follow these manual steps to perform an update of the API versions of a release manifest: -- Prerequsiities: +- Prerequisites: - HELM_PROTOBUF_SCHEMA: [Helm protobuf schema](https://github.com/helm/helm/tree/dev-v2/_proto) - PROTOBUF_SCHEMA: [Protobuf base schema](https://github.com/protocolbuffers/protobuf/tree/master/src) - Get the name of the latest deployed release: @@ -147,3 +147,5 @@ with the new encoded release object API versions - Add a description in the upgrade, something along the lines to not perform a rollback to a Helm version prior to this current version + +> Note: Ensure to use the `protobuf schema` for the deployed Tiller version, otherwise the decoding might fail From 9247be3b786d417f4ca76bdc168f08f7933cdf6e Mon Sep 17 00:00:00 2001 From: Martin Hickey Date: Fri, 22 May 2020 10:56:26 +0000 Subject: [PATCH 3/3] Pull review comments from v3 doc Pulled in review comments from https://github.com/helm/helm-www/pull/639 Signed-off-by: Martin Hickey --- docs/kubernetes_apis.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/kubernetes_apis.md b/docs/kubernetes_apis.md index 785859a92..63948cc99 100644 --- a/docs/kubernetes_apis.md +++ b/docs/kubernetes_apis.md @@ -117,18 +117,19 @@ plugin to perform the update of a release to supported APIs. Check out the readme for more details. Alternatively, you can follow these manual steps to perform an update of the API -versions of a release manifest: +versions of a release manifest. Depending on your configuration you will follow +the steps for the ConfigMap or Secret backend. - Prerequisites: - HELM_PROTOBUF_SCHEMA: [Helm protobuf schema](https://github.com/helm/helm/tree/dev-v2/_proto) - PROTOBUF_SCHEMA: [Protobuf base schema](https://github.com/protocolbuffers/protobuf/tree/master/src) -- Get the name of the latest deployed release: +- Get the name of the ConfigMap or Secret associated with the latest deployed release: - ConfigMap backend: `kubectl get configmap -l OWNER=TILLER,STATUS=DEPLOYED,NAME= --namespace | awk '{print $1}' | grep -v NAME` - Secrets backend: `kubectl get secret -l OWNER=TILLER,STATUS=DEPLOYED,NAME= --namespace | awk '{print $1}' | grep -v NAME` - Get latest deployed release details: - - ConfigMap backend: `kubectl get configmap -n -o yaml > release.yaml` - - Secrets backend: `kubectl get secret -n -o yaml > release.yaml` -- Backup the release incase you need to restore if something goes wrong: + - ConfigMap backend: `kubectl get configmap -n -o yaml > release.yaml` + - Secrets backend: `kubectl get secret -n -o yaml > release.yaml` +- Backup the release in case you need to restore if something goes wrong: - `cp release.yaml release.bak` - In case of emergency, restore: `kubectl apply -f release.bak -n ` - Decode the release object: