diff --git a/docs/workflow/developer-workflows.md b/docs/workflow/developer-workflows.md new file mode 100644 index 000000000..238837fd8 --- /dev/null +++ b/docs/workflow/developer-workflows.md @@ -0,0 +1,515 @@ +# Helm/DM Developer Experience Workflows + +This document outlines the individual workflows that Helm is designed to +solve. + +In this document we examine several workflows that we feel are central to +the experience of deploying, managing, and building applications using +Helm. Each workflow is followed by a basic model of where the processing +of commands occurs. + +### User Workflow + +The user workflow is the common case. This answers stories for the "tire +kicker" and "standard user" personas. + +#### Installation + +Currently, the client can be used with no installation. However, installing the server side component is done like this: + +``` +$ helm dm install +``` + +- Client uses existing `kubectl` configuration to install built-in manifests. + +General pattern: +``` +helm dm install +``` + +#### Searching + +``` +$ helm search bar +helm:example.com/foo/bar - A basic chart +helm:example.com/foo/barracuda - A fishy chart +helm:example.com/foo/barbecue - A smoky chart +``` + +- Client submits the query to the manager +- Server searches the available chart repos + +General pattern: +``` +helm search PATTERN +``` + + +#### Simple deployment: + +``` +$ helm deploy helm:example.com/foo/bar +Created wonky-panda +``` + +- The client sends the server a request to deploy `helm:example.com/foo/bar`. +- The server assigns a random name `wonky-panda`, fetches the chart from + object storage, and goes about the deployment process. + +General patterns: +``` +helm deploy [-f CONFIG] [-n NAME] [CHART] +``` + + +#### Find out about params: + +In this operation, helm reads a chart and returns the list of parameters +that can be supplied in a template: + +``` +$ helm chart show helm:example.com/foo/bar +Params: +- bgcolor: The background color for the home page (hex code or HTML colors) +- title: The title of the app +``` + +- The client sends the request to the API server +- The API server fetches the chart, analyzes it, and returns the list of + parameters. + +General pattern: +``` +helm chart show CHART +``` + + +#### Generate the params for me: + +In this operation, helm generates a parameter values file for the user. + +``` +$ helm chart params helm:example.com/foo/bar +Created: values.yaml +$ edit values.yaml +``` + +- The client sends the request to the server +- The server returns a stub file +- The client writes the file to disk + +General pattern: +``` +helm chart params CHART [CHART...] +``` + +#### Deploy with params: + +In this operation, the user deploys a chart with an associated values +file. + +``` +$ helm deploy -f values.yaml helm:example.com/foo/bar +Created taco-tuesday +``` + +- The client sends a request with the name of the chart and the values + file. +- The API server generates a name (`taco-tuesday`) +- The API server fetches the request chart, sends the chart and values + through the template resolution phase, and deploys the results. + +If we allow the user to pass in a name (overriding the generated name), +then the server must first guarantee that this name is unique. + +Alternately, just the values file may be specified, since it contains a reference to the base chart. + +``` +$ helm deploy -f values.yaml +Created taco-wednesday +``` + +#### Get the info about named deployment. + +A deployment, as we have seen, is a named instance of a chart. +Operations that operate on these instances use the name to refer to the +instance. + +``` +$ helm status taco-tuesday +OK +Located at: 10.10.10.77:8080 +``` + +- The client sends the API server a request for the status of + `taco-tuesday`. +- The API server looks up pertinent data and returns it. +- The client displays the data. + +General pattern: +``` +helm status NAME +``` + +#### Edit and redeploy: + +Redeployment is taking an existing _instance_ and changing its template +values, and then re-deploying it. + +``` +$ edit values.yaml +$ helm redeploy -f values.yaml taco-tuesday +Redeployed taco-tuesday +``` + +- The client sends the instance name and the new values to the server +- The server coalesces the values into the existing instance and then + restarts. + +General pattern: +``` +helm redeploy [-f CONFIG] NAME +``` + + +#### Delete: + +``` +$ helm delete taco-tuesday +Destroyed taco-tuesday +``` + +- The client sends the DELETE instance name command +- The API server destroys the resource + +General pattern: +``` +helm uninstall NAME [NAME...] +``` + + +### Power User Features + +Users familiar with the system may desire additional tools. + +#### Name a deploy + +``` +$ helm deploy -name skinny-pigeon example.com/foo/bar +``` + +This follows the deployment process above. The server _must_ ensure that +the name is unique. + +#### Get values for an app: +``` +$ helm deployment params taco-tuesday +Stored in values.yaml +$ helm redeploy taco-tuesday values.yaml +``` + +- The client sends the instance name to the values endpoint +- The server returns the values file used to generate the instance. +- The client writes this to a file (or, perhaps, to STDOUT) + +General pattern: +``` +helm deployment params NAME [NAME...] +``` + +When more than one name is specified, the resulting file will contain configs for all names. + +#### Get fully generated manifest files + +``` +$ helm deployment manifest taco-tuesday +Created manifest.yaml +``` + +- The client sends the instance name to the manifests endpoint +- The server returns the manifests, as generated during the + deploy/redeploy cycle done prior. + +General pattern: +``` +helm deployment manifest NAME [NAME...] +``` + +#### Auto-detecting helm problems + +``` +$ helm doctor +``` + +- The client performs local diagnostics and diagnostics of Kubernetes and the DM server. + +General pattern: +``` +helm doctor +``` + +#### Listing all installed charts + +``` +$ helm chart list +helm:example.com/foo/bar#1.1.1 +helm:example.com/foo/bar#1.1.2 +helm:example.com/foo/barbecue#0.1.0 +``` + +- The client sends the server a request for all charts installed +- The server computes and responds + +General pattern: +``` +helm chart list +``` + +#### Get instances of a chart + +NB: We might rename this `helm chart instances`, as that is less vague. + +``` +$ helm chart get helm:example.com/foo/bar +taco-tuesday +taco-wednesday +``` + +- The client sends a request to the API server. +- The server responds with a list of deployment instance names. + +This retrieves a shallow list, and does not inspect instances for ancestor charts. + +General pattern: +``` +helm chart get CHART +``` + +### Listing deployments + +``` +$ helm deployment list +skinny-pigeon +taco-tuesday +taco-wednesday +``` + +- The client requests a list from the server +- The server returns the list + +General pattern: +``` +helm deployment list +``` + +### Getting details of a deployment + +_NB: Might not need this._ + +``` +$ helm deployment show skinny-pigeon +DETAILS +``` + +### Developer Workflow + +This section covers the experience of the chart developer. + +In this case, when the client detects that it is working with a local +chart, it bundles the chart, and sends the entire chart, not just the +values. + +``` +$ helm create mychart +Created mychart/Chart.yaml +$ helm lint mychart +OK +$ helm deploy . +Uploading ./mychart as localchart/mychart-0.0.1.tgz +Created skinny-pigeon +$ helm status skinny-pigeon +OK +$ edit something +$ helm redeploy skinny-pigeon +Redeployed skinny-pigeon +``` + +- `helm create` and `helm lint` are client side operations +- `helm deploy`, `helm status`, and `helm redeploy` are explained above. + +General pattern for create: +``` +helm create [--from NAME] CHARTNAME +``` + +Where `NAME` will result in fetching the generated values from the cluster. + +General pattern for lint: +``` +helm lint PATH +``` + +#### Packaging and Releasing packages + +Package a chart: + +``` +$ helm package . +Created foo-1.1.2.tgz +``` + +Releasing a chart: + +``` +$ helm release -u https://example.com/bucket ./foo-1.1.2.tgz +Uploaded to https://example.com/bucket/foo-1.1.2.tgz +``` + +General pattern: +``` +helm package PATH +helm release [-u destination] PATH|FILE +``` + + +### Helm Cluster Management Commands + +#### Install +``` +$ helm dm install +``` +- Client installs using the current kubectl configuration + +General pattern: +``` +helm dm install +``` + +#### Uninstall + +``` +$ helm dm uninstall +``` + +- The client interacts with the Kubernetes API server + +General pattern: +``` +helm dm uninstall +``` + +#### Check which cluster is the current target for helm + +``` +$ helm dm target +API Endpoint: https://10.21.21.21/ +``` + +- The client interacts with the local Kubernetes config and the Kubernetes API server + +General pattern: +``` +helm search PATTERN +``` + +#### View status of DM service + +``` +$ helm dm status +OK +``` + +- The client interacts with the Kubernetes API server + +General pattern: +``` +helm dm status +``` + +### Repository Configuration + +#### Listing repositories + +``` +$ helm repo list +``` + +- The client request info from the server +- The server provides information about which repositories it is aware of + +General pattern: +``` +helm repo list +``` + +#### Adding credentials + +``` +$ helm credential add aff34... 89897a... +Created token-foo +``` + +- The client sends a request to the server +- The server creates a new token and returns a name + +General pattern: +``` +helm credential add TOKEN SECRET +``` + +#### Adding a repository with credentials + +``` +$ helm repo add -c token-foo https://example.com/charts +``` + +The URL is of the form `PROTO://HOST[:PORT]/BUCKET`. + +General pattern: +``` +helm repo add [-c TOKEN_NAME] REPO_URL +``` + +#### Removing repositories + +``` +$ helm repo rm https://example.com/charts +``` + +- The client sends a request to the server +- The server removes the repo from the known list + +General pattern: +``` +helm repo rm REPO_URL +``` + +#### Listing Credentials + +``` +$ helm credential list +token-foo: TOKEN +``` + +- The client requests a list of tokens +- The server returns the name and the token, but not the secret + +General pattern: +``` +helm credential list [PATTERN] +``` + +#### Removing credentials + +``` +$ helm credential rm token-foo +``` + +- The client sends a request to the server +- The server deletes the credential + +General pattern: +``` +helm credential rm CREDENTIAL_NAME +``` diff --git a/docs/workflow/helm-dm-diagrams.src.md b/docs/workflow/helm-dm-diagrams.src.md new file mode 100644 index 000000000..03c3ecb9a --- /dev/null +++ b/docs/workflow/helm-dm-diagrams.src.md @@ -0,0 +1,84 @@ +# Helm-DM Workflow Diagrams + +# Helm Official +@startuml helm-official-workflow.png +autonumber +actor Member +actor Core +participant GitHub +participant CI +database Repository +Member->GitHub: PR New Chart +GitHub->CI: Run automated tests +CI->GitHub: Tests pass +GitHub-->Member: Tests pass +GitHub-->Core: Tests pass +Core<->Member: Discussion about chart +Core->GitHub: Code review +Core->GitHub: Release approval +GitHub->CI: Issue release +CI->Repository: Push release +Member->Repository: Get release +@enduml + +# Public Chart Repository +An example workflow using Launchpad and BZR. +@startuml public-chart-repo.png +autonumber +actor Developer +actor Maintainer +participant Launchpad +Developer->Launchpad: Push branch with chart +Maintainer->Launchpad: Fetch branch +Maintainer->Maintainer: Locally test +Maintainer->Launchpad: Merge branch +... later ... +Maintainer->Launchpad: Create global release +Maintainer->Maintainer: Regenerate all versioned charts +Maintainer->Repository: Push charts +@enduml + +# Private chart repository +An example workflow using an internal Gerrit Git repo and Jenkins. +@startuml private-chart-repo.png +autonumber +actor "Developer A" as A +actor "Developer B" as B +actor "Internal user" as Internal +participant Gerrit +participant Jenkins +participant "Docker registry" as Docker +participant "Chart repository" as Chart +A->Gerrit: Clone repo +A->A: Local development +A->Gerrit: Push branch +Gerrit->Jenkins: Test branch +Jenkins->Jenkins: Build and test +Jenkins->Docker: Snapshot image push +Jenkins->Chart: Snapshot chart +Jenkins->Jenkins: Integration tests +Jenkins-->A: Build OK +B->Gerrit: Code review +B->A: Code approval +A->Gerrit: Tag the new chart +Gerrit->Jenkins: Release +Jenkins->Jenkins: Build and test +Jenkins->Docker: Production image push +Jenkins->Chart: Production chart +Internal->Chart: Get production chart +@enduml + +# Private Charts without repository +An example using SVN for local dev, and no chart repository. +@startuml private-chart-no-repo.png +autonumber +actor "Developer A" as A +actor "Developer B" as B +participant SVN +participant Kubernetes +A->SVN: Checkout code +A->A: Develop locally +A->SVN: Check in code +B<->SVN: Update local +B->Kubernetes: Run helm deploy on local chart +@enduml diff --git a/docs/workflow/helm-official-workflow.png b/docs/workflow/helm-official-workflow.png new file mode 100644 index 000000000..049c9f68b Binary files /dev/null and b/docs/workflow/helm-official-workflow.png differ diff --git a/docs/workflow/private-chart-no-repo.png b/docs/workflow/private-chart-no-repo.png new file mode 100644 index 000000000..4919d63bc Binary files /dev/null and b/docs/workflow/private-chart-no-repo.png differ diff --git a/docs/workflow/private-chart-repo.png b/docs/workflow/private-chart-repo.png new file mode 100644 index 000000000..bf9d17e89 Binary files /dev/null and b/docs/workflow/private-chart-repo.png differ diff --git a/docs/workflow/public-chart-repo.png b/docs/workflow/public-chart-repo.png new file mode 100644 index 000000000..31c9f73a8 Binary files /dev/null and b/docs/workflow/public-chart-repo.png differ diff --git a/docs/workflow/team-workflows.md b/docs/workflow/team-workflows.md new file mode 100644 index 000000000..8fcb7b342 --- /dev/null +++ b/docs/workflow/team-workflows.md @@ -0,0 +1,226 @@ +# Team Workflows + +This document explains the four team workflows that Helm is designed to +solve. + +## An Overview of Four Team Workflows + +The remainder of this document is broken into two major sections: The Overview covers the +broad characteristics of the four workflows. The Development Cycle +section covers how the developer cycle plays out for each of the four +workflows. + +This section provides an introduction to the four workflows outlined +above, calling out characteristics of each workflow that define it +against other workflows. + +### Helm Official + +The _Helm Official_ project focuses on maintaining a repository of high-quality production-ready charts. Charts may be contributed by anyone in the broad community, and they are vetted and maintained by the Helm Official core contributors. + +Stage | Devel | Review | Release | Store | Use +------|-------|--------|---------|-------|----- +Operations | Create/modify chart | Code review | Sign, version, package | Store releases | Get, Use +Who? | Developer | Us | Us | Us | Anyone +How? | Dev PR | We review | We release and sign | We upload to managed storage | Chart is public + +Characteristics of the Helm Official workflow: + +- Source code for charts is maintained in a GitHub repository +- Chart source is linted and tested by automated tools and human beings +- Charts are released by core contributors to the project +- All Charts are made available under the Apache 2 license +- When a chart is released, an official binary distribution is uploaded to the official chart repository in Google Cloud Storage. In addition, Helm Official charts have the following characteristics: + - They released by community members + - They are versioned independently (each chart has its own version) + - They are signed/provenanced by one of the official Helm GPG keys + - They are made available with no authentication or registration requirements + +### Public Unofficial + +The Helm project need not be the only repository available to users. Other organizations may wish to host their own repositories, and load whatever charts they so choose. + +We attempt to provide flexibility that allows such organizations to build repositories as they see fit, but keep the end user's experience roughly similar. + +Stage | Devel | Review | Release | Store | Use +------|-------|--------|---------|-------|----- +Operations | Create/modify chart | Code review | Sign, version, package | Store releases | Get, Use +Who? | Developer | Org? | Org? | Org | Anyone +How? | Dev Contribution | Org defines this | Org defines this | Org hosts | Chart is public + + +A public unofficial chart repository is free to structure the development cycle as they see fit. + +- We have no opinions about the source control (if any) used for chart development +- We provide `helm release` as a tool for releasing a chart + - We strongly advise public repository maintainers to use provenance files + - We strongly advise public repository maintainers to make public keys available to the general public. +- Charts must be stored in one of the supported object storage repositories, including Google Cloud Storage and S3 compatible storage +- Helm/DM provides the following tools for working with such repositories: + - `helm release` takes local source and builds a chart. + - `helm repo add|list|rm` allows users to add, list, and remove repositories from the recognized list. + - `helm repo push` allows new charts to be pushed to a remote server + + +### Private Chart Repositories + +We anticipate that some Helm users will desire to have private repositories over which they have control over both the development cycle and the availability of the released charts. + +Similar to our approach to unofficial chart repositories, we attempt to provide a tool general enough that it can be used for private chart repositories. + +Stage | Devel | Review | Release | Store | Use +------|-------|--------|---------|-------|----- +Operations | Create/modify chart | Code review | Sign, version, package | Store releases | Get, Use +Who? | Org | Org | Org | Org | Org +How? | Org defines this | Org defines this | Org defines this | Org hosts | Org controls access + + +We assume the following about this workflow: + +- We have no opinions about the source control (if any) used for chart development +- We provide `helm release` as a tool for releasing a chart + - While we suggest signing charts, this is a discretionary exercise for private repositories +- Charts may be stored in private GCS, S3, or compatible object storage. Helm is tested on Minio (an S3 compatible server for private hosting) to ensure a viable private cloud mechanism. + - Private repositories may require authentication via token/secret, as well as access controls. +- Helm/DM provides the following tools for working with such repositories: + - `helm release` takes local source and builds a chart. + - `helm repo add|list|rm` allows users to add, list, and remove repositories from the recognized list. + - `helm repo push` allows new charts to be pushed to a remote server + + +### Private Charts without Repositories + +In some cases, an organization may wish to create charts, but not store them in a chart repository. While this is not the preferred Helm workflow, we support tooling for this method. + +Stage | Devel | Review | Release | Store | Use +------|-------|--------|---------|-------|----- +Operations | Create/modify chart | Code review | Sign, version, package | Store releases | Get, Use +Who? | Org | Org | Org | N/A | Org +How? | Org defines this | Org defines this | Org defines this | N/A | Org defines this + +We operate on a different set of assumptions for this workflow. + +- We have no opinions about the source code control (if any) used. +- We have no opinions about access control when repositories are not involved. +- Charts must be pushed into the cluster (the cluster will not pull from a non-repository location) +- Helm/DM provides the following tools for this workflow: + - `helm release` builds a package + - `helm deploy` pushes a package into a cluster + +## Development Cycles for Each Workflow + +In this section, we explain the developer cycle for each of the four workflows above. + +### 0. Pre-Submission Workflow (aka Local Development) + +Prior to submitting a chart for release, developers may use the following workflow. This workflow is assumed in all four sections below. + +1. Create a chart with `helm create MYCHART` (or manually) +2. Edit the chart +3. Test the chart's standards conformance with `helm lint MYCHART` +4. Run a test deployment using `helm deploy MYCHART` + +### 1. The Helm Official Development Cycle + +The Helm Official project maintains source code in a location that is readily available to all of the community. Source code is stored on GitHub in the official (`github.com/helm/charts` or `github.com/kubernetes/charts`) repository. GitHub facilitates a development workflow that this project uses for chart maintenance. + +There are two general classes of user that are important to this workflow: + +- Core Contributors: Core contributors are developers that have been given special stewardship responsibilities over the repository. They have the following responsibilities: + - Review submissions to the repository + - Approve charts for release + - Respond to issues with existing charts +- Community Members: Any user of the Helm Official project who is not a core contributor. + +![Helm Official Workflow](helm-official-workflow.png) + +1. PULL REQUEST: A new chart (or an update to an existing chart) is contributed by a core maintainer or community member. This is done using GitHub pull requests. +2. AUTOMATED TESTING: Automated testing tools evaluate the contribution for the following: + - CLA approval of the submitter + - Style/format adherence + - Unit, functional, and/or integration tests pass +3. DISCUSSION: Any discussion on the pull request may happen using GitHub's commentary features +3. CODE REVIEW: Two or more core contributors must review the code and sign off on it. +4. RELEASE APPROVAL: If a chart is approved for release, a core maintainer may mark it as such +5. AUTOMATED RELEASE: Once a chart is approved for release, an automated tool will bundle the chart, sign it using an official signature, and upload it into the Helm Official repository + +### 2. Unofficial Public Repositories + +Unofficial repositories do not have a well-defined development workflow, but have a semi-rigid release workflow. + +- The public repository must use a supported object storage system +- Charts must be in the same format + +#### A Hypothetical Dev Workflow + +An organization has a Bazaar (bzr) project maintained at _launchpad.net_, and has no automated tooling around the project. + +![Unofficial Public Repo](public-chart-repo.png) + +1. Developers work off of copies of the Bazaar code base +2. Developers push branches into the code review system when ready +2. A central maintainer approves and merges reviewed code +3. At a fixed point in time, the project administrator releases a new version of the entire repository +4. During this process, all charts are version, packaged/signed, and uploaded to the project's S3 repository, where they are made available to users + +The above workflow illustrates how a development team may conduct public shared development, and release to a public repository, but with a workflow that diverges substantially from the model of the Helm Official repository. + +### 3. Private Chart Repository + +In this model, the entire process is managed by an organization. + +#### A Hypothetical Dev Workflow + +The organization uses an internally hosted Git server, Gerrit for code review, and Jenkins for automation. They host an internal repository on Minio. This repository has a combination of pre-approved Helm Official packages (copied from upstream) and internal packages. + +In this workflow, charts are not stored together. Instead, each chart is stored alongside the Docker image source code. For example, the repository named `corpcalendar.git` is laid out as follows: + +``` +corpcalendar/ + Dockerfile + src/ + ... # corpcalendar source code + chart/ + corpcalendar + Chart.yaml + ... # Helm Chart files +``` + +Each separate project is structured in this way. + +![Private Repo](private-chart-repo.png) + +1. Developers clone a desired repo (`git clone .../corpcalendar.git`) +2. Developers work on the code and chart together +3. Upon each commit, Jenkins does the following: + - Project tests are run + - A snapshot Docker image is built and stored in a snapshot Docker registry + - A snapshot chart is built and stored on a snapshot Helm repository + - Integration tests are run +4. When the developer is ready for a release, she tags the repository (`git tag v1.2.3`) and pushes the tag +5. Upon a tag commit, Jenkins does the following: + - Runs tests + - Creates a final Docker image and uploads to the internal release Docker registry + - Creates a final chart and uploads to the internal Helm repository +6. Internal users may then access the chart at the internal Helm repository + +This workflow represents a method common in enterprises, and also illustrates how chart development need not occur in one aggregated repository. + +### 4. Private Charts without Repositories + +This model is used in cases where an organization chooses not to host a Chart repository. While we don't advise using this method, a workflow exists. + +While we don't offer an opinionated VCS workflow, we also suggest that one is used. In our example below, we draw on VCS usage. + +#### A Hypothetical Dev Workflow + +A small development team uses Subversion (SVN) to host their internal projects. They do not use a chart repository, nor do they employ any automated testing tools outside of Helm. + +![Private Charts with No Repo](private-chart-no-repo.png) + +1. Developer Andy checks out a copy of the SVN repository (`svn co ...`) +2. Developer Andy edits charts locally, and test locally +3. When a chart is ready for sharing, the developer checks in the revised chart (`svn ci ...`) +4. Developer Barb updates her local copy (`svn up`) +5. Developer Barb then uses `helm deploy ./localchart` to deploy this chart into production +