|
|
|
@ -1,9 +1,9 @@
|
|
|
|
|
# Deployment Manager Design
|
|
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
Deployment Manager is a service which can be run in a Kubernetes cluster that
|
|
|
|
|
Deployment Manager is a service that runs in a Kubernetes cluster. It
|
|
|
|
|
provides a declarative configuration language to describe Kubernetes
|
|
|
|
|
resources and a mechanism for deploying, updating, and deleting configurations.
|
|
|
|
|
resources, and a mechanism for deploying, updating, and deleting configurations.
|
|
|
|
|
This document describes the configuration language, object model, and
|
|
|
|
|
architecture of the service in detail.
|
|
|
|
|
|
|
|
|
@ -41,8 +41,8 @@ resources:
|
|
|
|
|
Resources can reference values from other resources. The version of Deployment
|
|
|
|
|
Manager running in the Google Cloud Platform uses references to understand
|
|
|
|
|
dependencies between resources and properly order the operations it performs on
|
|
|
|
|
a configuration. This version doesn't yet have this functionality, but will have
|
|
|
|
|
it shortly.
|
|
|
|
|
a configuration. (This version of DM doesn't yet order operations to satisfy
|
|
|
|
|
dependencies, but it will soon.)
|
|
|
|
|
|
|
|
|
|
A reference follows this syntax: **$(ref.NAME.PATH)**, where _NAME_ is the name
|
|
|
|
|
of the resource being referenced, and _PATH_ is a JSON path to the value in the
|
|
|
|
@ -58,6 +58,7 @@ In this case, _my-service_ is the name of the resource, and _metadata.name_ is
|
|
|
|
|
the JSON path to the value being referenced.
|
|
|
|
|
|
|
|
|
|
### Configurable Resources
|
|
|
|
|
|
|
|
|
|
Configurable resources are the primitive resources that can be configured in
|
|
|
|
|
Deployment Manager, including:
|
|
|
|
|
|
|
|
|
@ -66,18 +67,19 @@ Deployment Manager, including:
|
|
|
|
|
* Service
|
|
|
|
|
|
|
|
|
|
Deployment Manager processes configurable resources by passing their
|
|
|
|
|
configuration properties directly to kubectl on the cluster to create, update,
|
|
|
|
|
or delete the resource.
|
|
|
|
|
configuration properties directly to kubectl to create, update, or delete them
|
|
|
|
|
in the cluster.
|
|
|
|
|
|
|
|
|
|
### Templates
|
|
|
|
|
|
|
|
|
|
Templates are abstract types that can be created using Python or
|
|
|
|
|
[Jinja](http://jinja.pocoo.org/). Templates take a set of properties and must
|
|
|
|
|
output a valid YAML configuration string. Properties are bound to values when a
|
|
|
|
|
template is instantiated in a configuration.
|
|
|
|
|
[Jinja](http://jinja.pocoo.org/). A template takes a set of properties as input,
|
|
|
|
|
and must output a valid YAML configuration string. Properties are bound to
|
|
|
|
|
values when a template is instantiated in a configuration.
|
|
|
|
|
|
|
|
|
|
Templates are expanded as a pre-processing step before configurable resources
|
|
|
|
|
Templates are expanded in a pre-processing step before configurable resources
|
|
|
|
|
are processed. They can output configurations containing configurable resources,
|
|
|
|
|
or additional nested templates. Nested templates will be processed recursively.
|
|
|
|
|
or additional nested templates. Nested templates are processed recursively.
|
|
|
|
|
|
|
|
|
|
An example of a template in python is:
|
|
|
|
|
|
|
|
|
@ -97,7 +99,7 @@ def GenerateConfig(context):
|
|
|
|
|
return yaml.dump({'resources': resources})
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
and in Jinja is:
|
|
|
|
|
and in Jinja:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
resources:
|
|
|
|
@ -108,22 +110,22 @@ resources:
|
|
|
|
|
...
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Templates provide access to several sets of data, which can be used for
|
|
|
|
|
parameterizing or further customizing a configuration:
|
|
|
|
|
Templates provide access to multiple sets of data, which can be used for
|
|
|
|
|
parameterizing or further customizing configurations:
|
|
|
|
|
|
|
|
|
|
* env: a map of values defined by Deployment Manager, including _deployment_,
|
|
|
|
|
_name_, and _type_
|
|
|
|
|
* env: a map of key/value pairs from the environment, including pairs
|
|
|
|
|
defined by Deployment Manager, such as _deployment_, _name_, and _type_
|
|
|
|
|
* properties: a map of the key/value pairs passed in the properties section when
|
|
|
|
|
instantiating the template
|
|
|
|
|
* imports: a map of import file name to file contents of all imports originally
|
|
|
|
|
specified for the configuration
|
|
|
|
|
* imports: a map of import file names to file contents of all imports
|
|
|
|
|
originally specified for the configuration
|
|
|
|
|
|
|
|
|
|
In Python, this data is available from the _context_ object passed into the
|
|
|
|
|
_GenerateConfig_ method.
|
|
|
|
|
|
|
|
|
|
### Template Schemas
|
|
|
|
|
A schema can be provided for a template. The schema describes the template in
|
|
|
|
|
more details, including:
|
|
|
|
|
A schema can be optionally provided for a template. The schema describes
|
|
|
|
|
the template in more detail, including:
|
|
|
|
|
|
|
|
|
|
* info: more information about the template, including long description and
|
|
|
|
|
title
|
|
|
|
@ -131,7 +133,7 @@ more details, including:
|
|
|
|
|
* required: properties which are required when instantiating the template
|
|
|
|
|
* properties: JSON Schema descriptions of each property the template accepts
|
|
|
|
|
|
|
|
|
|
An example of a template schema is:
|
|
|
|
|
Here's an example of a template schema:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
info:
|
|
|
|
@ -151,19 +153,20 @@ properties:
|
|
|
|
|
default: prop-value
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Schemas are used by Deployment Manager to validate properties being used during
|
|
|
|
|
template instantiation and provide default value semantics on properties.
|
|
|
|
|
Schemas are used by Deployment Manager to validate properties during
|
|
|
|
|
template instantiation, and to provide default values.
|
|
|
|
|
|
|
|
|
|
Schemas must be imported along-side the templates which they describe when
|
|
|
|
|
passing configuration to Deployment Manager.
|
|
|
|
|
Schemas must be imported with the templates they describe, when passing
|
|
|
|
|
configuration to Deployment Manager.
|
|
|
|
|
|
|
|
|
|
### Instantiating Templates
|
|
|
|
|
Templates can be instantiated in the same way that a configurable resource is
|
|
|
|
|
used. They can be used in two different ways, either passed to the API as an
|
|
|
|
|
|
|
|
|
|
Templates can be used in two different ways: either passed to the API as an
|
|
|
|
|
imported file, or used from a public HTTP endpoint.
|
|
|
|
|
|
|
|
|
|
#### Imported Templates
|
|
|
|
|
Templates may be imported as part of the target configuration and used
|
|
|
|
|
|
|
|
|
|
Templates can be imported as part of the target configuration, and used
|
|
|
|
|
directly, for example:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
@ -177,15 +180,16 @@ resources:
|
|
|
|
|
prop1: prop-value
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The _imports_ list is not understood by the Deployment Manager service, but is a
|
|
|
|
|
directive to client-side tooling to specify what additional files should be
|
|
|
|
|
included when passing a configuration to the API.
|
|
|
|
|
The _imports_ list is not understood by the Deployment Manager service.
|
|
|
|
|
It's a directive used by client-side tools to specify what additional files
|
|
|
|
|
should be included when passing a configuration to the API.
|
|
|
|
|
|
|
|
|
|
Using the Deployment Manager API, these templates can be included in the
|
|
|
|
|
imports section of the _targetConfig_.
|
|
|
|
|
imports section of the _configuration_.
|
|
|
|
|
|
|
|
|
|
#### External Templates
|
|
|
|
|
Templates may also be used from a public HTTP endpoint, for example:
|
|
|
|
|
|
|
|
|
|
Templates can also be used from a public HTTP endpoint. For example:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
resources:
|
|
|
|
@ -197,31 +201,33 @@ resources:
|
|
|
|
|
|
|
|
|
|
The service will process external templates as follows:
|
|
|
|
|
|
|
|
|
|
1. fetch the external template as an import
|
|
|
|
|
1. attempt to fetch the schema for the template, using
|
|
|
|
|
1. Fetch the external template as an import
|
|
|
|
|
1. Attempt to fetch the schema for the template, using
|
|
|
|
|
_<full template path>.schema_ as the schema path
|
|
|
|
|
1. repeat for any sub-imports found in the schema file
|
|
|
|
|
1. Repeat for any sub-imports found in the schema file
|
|
|
|
|
|
|
|
|
|
When fetching schema files and sub-imports, the base path of the external
|
|
|
|
|
template is used for relative paths.
|
|
|
|
|
|
|
|
|
|
## API Model
|
|
|
|
|
|
|
|
|
|
Deployment Manager exposes a set of RESTful collections over HTTP/JSON.
|
|
|
|
|
|
|
|
|
|
### Deployments
|
|
|
|
|
|
|
|
|
|
Deployments are the primary resource in the Deployment Manager service. The
|
|
|
|
|
inputs to a deployment are:
|
|
|
|
|
|
|
|
|
|
* name
|
|
|
|
|
* targetConfig
|
|
|
|
|
* configuration
|
|
|
|
|
|
|
|
|
|
When creating a deployment, users pass their YAML configuration, as well as any
|
|
|
|
|
import files (templates, datafiles, etc.) in as the _targetConfig_.
|
|
|
|
|
When creating a deployment, users pass their configuration,
|
|
|
|
|
as well as any import files (templates, datafiles, etc.), all encoded in `YAML`,
|
|
|
|
|
in as the _configuration_.
|
|
|
|
|
|
|
|
|
|
Creating, updating and deleting a deployment creates a new manifest for the
|
|
|
|
|
deployment, and then processes the new configuration. In the case of deleting a
|
|
|
|
|
deployment, the deployment is first updated to an empty manifest containing no
|
|
|
|
|
resources, and then is removed from the system.
|
|
|
|
|
Creating, updating or deleting a deployment creates a new manifest for the
|
|
|
|
|
deployment. When deleting a deployment, the deployment is first updated to
|
|
|
|
|
an empty manifest containing no resources, and then removed from the system.
|
|
|
|
|
|
|
|
|
|
Deployments are available at the HTTP endpoint:
|
|
|
|
|
|
|
|
|
@ -230,15 +236,13 @@ http://manager-service/deployments
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Manifests
|
|
|
|
|
A manifest is created for a deployment every time it is mutated, including
|
|
|
|
|
creation, update, and deletion.
|
|
|
|
|
|
|
|
|
|
A manifest contains three major pieces of data:
|
|
|
|
|
A manifest is created for a deployment every time it is changed. It contains
|
|
|
|
|
three key components:
|
|
|
|
|
|
|
|
|
|
* inputConfig: the original input configuration for the manifest, including YAML
|
|
|
|
|
configuration and imports
|
|
|
|
|
* expandedConfig: the final expanded configuration to be used when processing
|
|
|
|
|
resources for the manifest
|
|
|
|
|
* inputConfig: the original input configuration for the manifest
|
|
|
|
|
* expandedConfig: the expanded configuration to be used when processing resources
|
|
|
|
|
* for the manifest
|
|
|
|
|
* layout: the hierarchical structure of the manifest
|
|
|
|
|
|
|
|
|
|
Manifests are available at the HTTP endpoint:
|
|
|
|
@ -248,12 +252,14 @@ http://manager-service/deployments/<deployment>/manifests
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
#### Expanded Configuration
|
|
|
|
|
|
|
|
|
|
Given a new _inputConfig_, Deployment Manager expands all template
|
|
|
|
|
instantiations recursively until there is a flat set of configurable resources.
|
|
|
|
|
This final set is stored as the _expandedConfig_ and is used during resource
|
|
|
|
|
processing.
|
|
|
|
|
|
|
|
|
|
#### Layout
|
|
|
|
|
|
|
|
|
|
Users can use templates to build a rich, deep hierarchical architecture in their
|
|
|
|
|
configuration. Expansion flattens this hierarchy and removes the template
|
|
|
|
|
relationships from the configuration to create a format optimized for the process
|
|
|
|
@ -321,53 +327,49 @@ in more detail below.
|
|
|
|
|
|
|
|
|
|
Currently there are two caveats in the design of the service:
|
|
|
|
|
|
|
|
|
|
* Synchronous API: the API is currently designed to block on all processing for
|
|
|
|
|
* Synchronous API: the API currently blocks on all processing for
|
|
|
|
|
a deployment request. In the future, this design will change to an
|
|
|
|
|
asynchronous operation-based mode.
|
|
|
|
|
* Non-persistence: the service currently stores all metadata in memory, so will
|
|
|
|
|
lose all knowledge of deployments and their metadata on restart. In the
|
|
|
|
|
future, the service will persist all deployment metadata in the cluster.
|
|
|
|
|
* Non-persistence: the service currently stores all metadata in memory,
|
|
|
|
|
so it will lose all knowledge of deployments and their metadata on restart.
|
|
|
|
|
In the future, the service will persist all deployment metadata.
|
|
|
|
|
|
|
|
|
|
### Manager
|
|
|
|
|
|
|
|
|
|
The **manager** service acts as both the API server and the workflow engine for
|
|
|
|
|
processing deployments. The process for a deployment is:
|
|
|
|
|
processing deployments. It uses the following process:
|
|
|
|
|
|
|
|
|
|
1. Create a new deployment with a manifest containing _inputConfig_ from the
|
|
|
|
|
user request
|
|
|
|
|
1. Call out to **expandybird** service to perform expansion on the _inputConfig_
|
|
|
|
|
1. Call out to he **expandybird** service to expand the _inputConfig_
|
|
|
|
|
1. Store the resulting _expandedConfig_ and _layout_
|
|
|
|
|
1. Call out to **resourcifier** service to perform processing on resources from
|
|
|
|
|
the _expandedConfig_
|
|
|
|
|
1. Call out to the **resourcifier** service to perform processing on resources
|
|
|
|
|
from the _expandedConfig_
|
|
|
|
|
1. Respond with success or error messages to the original API request
|
|
|
|
|
|
|
|
|
|
The manager is responsible for all persistence of metadata associated with
|
|
|
|
|
The manager is responsible for saving the metadata associated with
|
|
|
|
|
deployments, manifests, type instances, and other resources in the Deployment
|
|
|
|
|
Manager model.
|
|
|
|
|
|
|
|
|
|
### Expandybird
|
|
|
|
|
The **expandybird** service takes in input configurations, including the YAML
|
|
|
|
|
configuration and import files, performs all template expansion, and returns the
|
|
|
|
|
resulting flat configuration and layout. It is completely stateless and handles
|
|
|
|
|
requests synchronously.
|
|
|
|
|
|
|
|
|
|
Because templates are Python or Jinja, the actual expansion process is performed
|
|
|
|
|
in a sub-process running a Python interpreter. A new sub-process is created for
|
|
|
|
|
every request to expandybird.
|
|
|
|
|
The **expandybird** service takes in input configurations, performs all template
|
|
|
|
|
expansions, and returns the resulting flat configuration and layout. It is completely
|
|
|
|
|
stateless.
|
|
|
|
|
|
|
|
|
|
Currently expansion is not sandboxed, but the intention of templates is to be
|
|
|
|
|
reproducable hermetically sealed entities, so future designs may
|
|
|
|
|
introduce a sandbox to limit external interaction like network and disk access
|
|
|
|
|
during expansion.
|
|
|
|
|
Because templates are written in Python or Jinja, the actual expansion process
|
|
|
|
|
is performed in a sub-process that runs a Python interpreter. A new sub-process
|
|
|
|
|
is created for every request to expandybird.
|
|
|
|
|
|
|
|
|
|
Currently, expansion is not sandboxed, but templates should be reproducable,
|
|
|
|
|
hermetically sealed entities. Future designs may therefore, introduce a sandbox to
|
|
|
|
|
limit external interaction, such as network or disk access, during expansion.
|
|
|
|
|
|
|
|
|
|
### Resourcifier
|
|
|
|
|
|
|
|
|
|
The **resourcifier** service takes in flat expanded configurations containing
|
|
|
|
|
only configurable resources, and makes the respective kubectl calls to process
|
|
|
|
|
each resource. It is completely stateless and handles requests synchronously.
|
|
|
|
|
|
|
|
|
|
Processing may be to create, update, or delete a resource,
|
|
|
|
|
depending on the request. The resourcifier handles references, and is the major
|
|
|
|
|
workflow engine for resource processing. In the future. it will also handle
|
|
|
|
|
dependencies between resources, as described earlier.
|
|
|
|
|
each resource. It is totally stateless, and handles requests synchronously.
|
|
|
|
|
|
|
|
|
|
The resourcifier service returns either success or error messages encountered
|
|
|
|
|
during resource processing.
|
|
|
|
|
The resourcifier returns either success or error messages encountered during
|
|
|
|
|
resource processing.
|
|
|
|
|