mirror of https://github.com/helm/helm
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
376 lines
12 KiB
376 lines
12 KiB
# Deployment Manager Design
|
|
|
|
## Overview
|
|
Deployment Manager is a service that runs in a Kubernetes cluster. It
|
|
provides a declarative configuration language to describe Kubernetes
|
|
resources, and a mechanism for deploying, updating, and deleting configurations.
|
|
This document describes the configuration language, object model, and
|
|
architecture of the service in detail.
|
|
|
|
## Configuration Language
|
|
The configuration language in Deployment Manager consists of two parts: a
|
|
YAML-based language for describing resources, and a templating mechanism for
|
|
creating abstract parameterizable types.
|
|
|
|
A configuration consists of a list of resources in YAML. Resources have three
|
|
properties:
|
|
|
|
* name: the name to use when managing the resource
|
|
* type: the type of the resource being managed
|
|
* properties: the configuration properties of the resource
|
|
|
|
An example snippet of a configuration looks like:
|
|
|
|
```
|
|
resources:
|
|
- name: my-rc
|
|
type: ReplicationController
|
|
properties:
|
|
metadata:
|
|
name: my-rc
|
|
spec:
|
|
replicas: 1
|
|
...
|
|
- name: my-service
|
|
type: Service
|
|
properties:
|
|
...
|
|
```
|
|
|
|
### References
|
|
Resources can reference values from other resources. The version of Deployment
|
|
Manager running in the Google Cloud Platform uses references to understand
|
|
dependencies between resources and properly order the operations it performs on
|
|
a configuration. (This version of DM doesn't yet order operations to satisfy
|
|
dependencies, but it will soon.)
|
|
|
|
A reference follows this syntax: **$(ref.NAME.PATH)**, where _NAME_ is the name
|
|
of the resource being referenced, and _PATH_ is a JSON path to the value in the
|
|
resource object.
|
|
|
|
For example:
|
|
|
|
```
|
|
$(ref.my-service.metadata.name)
|
|
```
|
|
|
|
In this case, _my-service_ is the name of the resource, and _metadata.name_ is
|
|
the JSON path to the value being referenced.
|
|
|
|
### Configurable Resources
|
|
|
|
Configurable resources are the primitive resources that can be configured in
|
|
Deployment Manager, including:
|
|
|
|
* Pod
|
|
* ReplicationController
|
|
* Service
|
|
|
|
Deployment Manager processes configurable resources by passing their
|
|
configuration properties directly to kubectl to create, update, or delete them
|
|
in the cluster.
|
|
|
|
### Templates
|
|
|
|
Templates are abstract types that can be created using Python or
|
|
[Jinja](http://jinja.pocoo.org/). A template takes a set of properties as input,
|
|
and must output a valid YAML configuration string. Properties are bound to
|
|
values when a template is instantiated in a configuration.
|
|
|
|
Templates are expanded in a pre-processing step before configurable resources
|
|
are processed. They can output configurations containing configurable resources,
|
|
or additional nested templates. Nested templates are processed recursively.
|
|
|
|
An example of a template in python is:
|
|
|
|
```
|
|
import yaml
|
|
|
|
def GenerateConfig(context):
|
|
resources = [{
|
|
'name': context.env['name'] + '-service',
|
|
'type': 'Service',
|
|
'properties': {
|
|
'prop1': context.properties['prop1'],
|
|
...
|
|
}
|
|
}]
|
|
|
|
return yaml.dump({'resources': resources})
|
|
```
|
|
|
|
and in Jinja:
|
|
|
|
```
|
|
resources:
|
|
- name: {{ env['name'] }}-service
|
|
type: Service
|
|
properties:
|
|
prop1: {{ properties['prop1'] }}
|
|
...
|
|
```
|
|
|
|
Templates provide access to multiple sets of data, which can be used for
|
|
parameterizing or further customizing configurations:
|
|
|
|
* env: a map of key/value pairs from the environment, including pairs
|
|
defined by Deployment Manager, such as _deployment_, _name_, and _type_
|
|
* properties: a map of the key/value pairs passed in the properties section when
|
|
instantiating the template
|
|
* imports: a map of import file names to file contents of all imports
|
|
originally specified for the configuration
|
|
|
|
In Python, this data is available from the _context_ object passed into the
|
|
_GenerateConfig_ method.
|
|
|
|
### Template Schemas
|
|
A schema can be optionally provided for a template. The schema describes
|
|
the template in more detail, including:
|
|
|
|
* info: more information about the template, including long description and
|
|
title
|
|
* imports: any sub-imports used by this template (may be relative path or URL)
|
|
* required: properties which are required when instantiating the template
|
|
* properties: JSON Schema descriptions of each property the template accepts
|
|
|
|
Here's an example of a template schema:
|
|
|
|
```
|
|
info:
|
|
title: The Example
|
|
description: A template being used as an example to illustrate concepts.
|
|
|
|
imports:
|
|
- path: helper.py
|
|
|
|
required:
|
|
- prop1
|
|
|
|
properties:
|
|
prop1:
|
|
description: The first property
|
|
type: string
|
|
default: prop-value
|
|
```
|
|
|
|
Schemas are used by Deployment Manager to validate properties during
|
|
template instantiation, and to provide default values.
|
|
|
|
Schemas must be imported with the templates they describe, when passing
|
|
configuration to Deployment Manager.
|
|
|
|
### Instantiating Templates
|
|
|
|
Templates can be used in two different ways: either passed to the API as an
|
|
imported file, or used from a public HTTP endpoint.
|
|
|
|
#### Imported Templates
|
|
|
|
Templates can be imported as part of the target configuration, and used
|
|
directly, for example:
|
|
|
|
```
|
|
imports:
|
|
- path: example.py
|
|
|
|
resources:
|
|
- name: example
|
|
type: example.py
|
|
properties:
|
|
prop1: prop-value
|
|
```
|
|
|
|
The _imports_ list is not understood by the Deployment Manager service.
|
|
It's a directive used by client-side tools to specify what additional files
|
|
should be included when passing a configuration to the API.
|
|
|
|
Using the Deployment Manager API, these templates can be included in the
|
|
imports section of the _configuration_.
|
|
|
|
#### External Templates
|
|
|
|
Templates can also be used from a public HTTP endpoint. For example:
|
|
|
|
```
|
|
resources:
|
|
- name: example
|
|
type: https://raw.githubusercontent.com/example/example.py
|
|
properties:
|
|
prop1: prop-value
|
|
```
|
|
|
|
The service will process external templates as follows:
|
|
|
|
1. Fetch the external template as an import
|
|
1. Attempt to fetch the schema for the template, using
|
|
_<full template path>.schema_ as the schema path
|
|
1. Repeat for any sub-imports found in the schema file
|
|
|
|
When fetching schema files and sub-imports, the base path of the external
|
|
template is used for relative paths.
|
|
|
|
## API Model
|
|
|
|
Deployment Manager exposes a set of RESTful collections over HTTP/JSON.
|
|
|
|
### Deployments
|
|
|
|
Deployments are the primary resource in the Deployment Manager service. The
|
|
inputs to a deployment are:
|
|
|
|
* name
|
|
* configuration
|
|
|
|
When creating a deployment, users pass their configuration,
|
|
as well as any import files (templates, datafiles, etc.), all encoded in `YAML`,
|
|
in as the _configuration_.
|
|
|
|
Creating, updating or deleting a deployment creates a new manifest for the
|
|
deployment. When deleting a deployment, the deployment is first updated to
|
|
an empty manifest containing no resources, and then removed from the system.
|
|
|
|
Deployments are available at the HTTP endpoint:
|
|
|
|
```
|
|
http://manager-service/deployments
|
|
```
|
|
|
|
### Manifests
|
|
|
|
A manifest is created for a deployment every time it is changed. It contains
|
|
three key components:
|
|
|
|
* inputConfig: the original input configuration for the manifest
|
|
* expandedConfig: the expanded configuration to be used when processing resources
|
|
for the manifest
|
|
* layout: the hierarchical structure of the manifest
|
|
|
|
Manifests are available at the HTTP endpoint:
|
|
|
|
```
|
|
http://manager-service/deployments/<deployment>/manifests
|
|
```
|
|
|
|
#### Expanded Configuration
|
|
|
|
Given a new _inputConfig_, Deployment Manager expands all template
|
|
instantiations recursively until there is a flat set of configurable resources.
|
|
This final set is stored as the _expandedConfig_ and is used during resource
|
|
processing.
|
|
|
|
#### Layout
|
|
|
|
Users can use templates to build a rich, deep hierarchical architecture in their
|
|
configuration. Expansion flattens this hierarchy and removes the template
|
|
relationships from the configuration to create a format optimized for the process
|
|
of instantiating the resources. However, the structural information contained in
|
|
the original configuration has many uses, so rather than discard it, Deployment
|
|
Manager preserves it in the form of a _layout_.
|
|
|
|
The _layout_ looks very much like an input configuration. It is a YAML list of
|
|
resources, where each resource contains the following information:
|
|
|
|
* name: name of the resource
|
|
* type: type of the resource
|
|
* properties: properties of the resource, set only for templates
|
|
* resources: sub-resources from expansion, set only for templates
|
|
|
|
An example layout is:
|
|
|
|
```
|
|
resources:
|
|
- name: rs
|
|
type: replicatedservice.py
|
|
propertes:
|
|
replicas: 2
|
|
resources:
|
|
- name: rs-rc
|
|
type: ReplicationController
|
|
- name: rs-service
|
|
type: Service
|
|
```
|
|
|
|
The layout can be used for visualizing the architecture of resources, including
|
|
their hierarchical structure and reference relationships.
|
|
|
|
### Types
|
|
The types API provides information about existing types being used the cluster.
|
|
|
|
It can be used to list all known types that are in use in existing deployments:
|
|
|
|
```
|
|
http://manager-service/types
|
|
```
|
|
|
|
It can be used to list all active instances of a specific type in the cluster:
|
|
|
|
```
|
|
http://manager-service/types/<type>/instances
|
|
```
|
|
|
|
Passing _all_ as the type shows all instances of all types in the cluster. Type
|
|
instances include the following information:
|
|
|
|
* name: name of resource
|
|
* type: type of resource
|
|
* deployment: name of deployment in which the resource resides
|
|
* manifest: name of manifest in which the resource configuration resides
|
|
* path: JSON path to the entry for the resource in the manifest layout
|
|
|
|
## Architecture
|
|
The Deployment Manager service is built to run as a service within a Kubernetes
|
|
cluster. It has three major components to manage deployments. The following
|
|
diagram illustrates the relationships between the components, which are described
|
|
in more detail below.
|
|
|
|
|
|
|
|
Currently there are two caveats in the design of the service:
|
|
|
|
* Synchronous API: the API currently blocks on all processing for
|
|
a deployment request. In the future, this design will change to an
|
|
asynchronous operation-based mode.
|
|
* Non-persistence: the service currently stores all metadata in memory,
|
|
so it will lose all knowledge of deployments and their metadata on restart.
|
|
In the future, the service will persist all deployment metadata.
|
|
|
|
### Manager
|
|
|
|
The **manager** service acts as both the API server and the workflow engine for
|
|
processing deployments. It uses the following process:
|
|
|
|
1. Create a new deployment with a manifest containing _inputConfig_ from the
|
|
user request
|
|
1. Call out to the **expandybird** service to expand the _inputConfig_
|
|
1. Store the resulting _expandedConfig_ and _layout_
|
|
1. Call out to the **resourcifier** service to perform processing on resources
|
|
from the _expandedConfig_
|
|
1. Respond with success or error messages to the original API request
|
|
|
|
The manager is responsible for saving the metadata associated with
|
|
deployments, manifests, type instances, and other resources in the Deployment
|
|
Manager model.
|
|
|
|
### Expandybird
|
|
|
|
The **expandybird** service takes in input configurations, performs all template
|
|
expansions, and returns the resulting flat configuration and layout. It is completely
|
|
stateless.
|
|
|
|
Because templates are written in Python or Jinja, the actual expansion process
|
|
is performed in a sub-process that runs a Python interpreter. A new sub-process
|
|
is created for every request to expandybird.
|
|
|
|
Currently, expansion is not sandboxed, but templates should be reproducable,
|
|
hermetically sealed entities. Future designs may therefore, introduce a sandbox to
|
|
limit external interaction, such as network or disk access, during expansion.
|
|
|
|
### Resourcifier
|
|
|
|
The **resourcifier** service takes in flat expanded configurations containing
|
|
only configurable resources, and makes the respective kubectl calls to process
|
|
each resource. It is totally stateless, and handles requests synchronously.
|
|
|
|
The resourcifier returns either success or error messages encountered during
|
|
resource processing.
|