From fdb3a7d2aee101c2e47e7b77682093775a288409 Mon Sep 17 00:00:00 2001 From: jackgr Date: Sun, 8 Nov 2015 21:25:56 -0800 Subject: [PATCH 01/27] Readme fixes. --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 42eebff96..9357bde3d 100644 --- a/README.md +++ b/README.md @@ -20,11 +20,12 @@ that second example correctly. It uses DM to deploy itself. See [examples/bootstrap/README.md](examples/bootstrap/README.md) for more information.) DM runs server side, in your Kubernetes cluster, so it can tell you what types -you've instantiated there, and even what instances you've created of a given type. -So, you can ask questions like: +you've instantiated there, what instances you've created of a given type, and even +how the instances are organized. So, you can ask questions like: -* Show me all the Redis slaves running in this cluster. -* Show me all the resources used by Redis. +* What Redis instances are running in this cluster? +* What Redis master and slave services are part of this Redis instance? +* What pods are part of this Redis slave? Because DM stores its state in the cluster, not on your workstation, you can ask those questions from any client at any time. @@ -86,7 +87,7 @@ dm --properties workers=3 deploy redis/v1 ``` When you deploy a type, `dm` generates a template from the type and input -paramaters, and then deploys it. +parameters, and then deploys it. You can also deploy an existing template, or read one from `stdin`. This command deploys the canonical Guestbook example from the examples directory: From 35816200b6ef94af11cf25cf43cbd86f91f2af19 Mon Sep 17 00:00:00 2001 From: Jack Greenfield Date: Sun, 8 Nov 2015 23:21:23 -0800 Subject: [PATCH 02/27] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9357bde3d..7ba78fce9 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Follow these 3 steps to install DM: 1. Make sure your Kubernetes cluster is up and running, and that you can run `kubectl` commands against it. 1. Clone this repository into the src folder of your GOPATH, if you haven't already. -1. Use `kubectl` to intall DM into your cluster: +1. Use `kubectl` to install DM into your cluster: ``` kubectl create -f install.yaml From 98989f83d0246926e8d2eea781236b8b234cb159 Mon Sep 17 00:00:00 2001 From: Jack Greenfield Date: Sun, 8 Nov 2015 23:30:54 -0800 Subject: [PATCH 03/27] Update README.md --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 7ba78fce9..d1615985d 100644 --- a/README.md +++ b/README.md @@ -149,13 +149,12 @@ make push ## Design of Deployment Manager -There is a more detailed [design document](docs/design/design.md) -available. +There is a more detailed [design document](docs/design/design.md) available. ## Status of the project -The project is still under active development, so you might run into issues. If -you do, please don't be shy about letting us know, or better yet, contributing a +This project is still under active development, so you might run into issues. If +you do, please don't be shy about letting us know, or better yet, contribute a fix or feature. We use the same [development process](CONTRIBUTING.md) as the main Kubernetes repository. From fad7519678fa5fe66f576f57443900cdca01825c Mon Sep 17 00:00:00 2001 From: jackgr Date: Sun, 8 Nov 2015 23:34:59 -0800 Subject: [PATCH 04/27] Add command list. --- README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d1615985d..e3a42954d 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,21 @@ For more information about this example, see [examples/guestbook/README.md](exam ## Additional commands The command line tool makes it easy to configure a cluster from a set of predefined -types. +types. Here's a list of available commands: + +``` +expand Expands the supplied template(s) +deploy Deploys the supplied type or template(s) +list Lists the deployments in the cluster +get Retrieves the supplied deployment +delete Deletes the supplied deployment +update Updates a deployment using the supplied template(s) +deployed-types Lists the types deployed in the cluster +deployed-instances Lists the instances of the supplied type deployed in the cluster +types Lists the types in the current registry +describe Describes the supplied type in the current registry + +``` ## Uninstalling Deployment Manager From f2fcd66fc1f0265b348ea9cfe53376a5aed220e1 Mon Sep 17 00:00:00 2001 From: Jack Greenfield Date: Sun, 8 Nov 2015 23:41:01 -0800 Subject: [PATCH 05/27] Update design.md --- docs/design/design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/design.md b/docs/design/design.md index ed0f99b98..a145a71c9 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -41,8 +41,8 @@ resources: Resources can reference values from other resources. The version of Deployment Manager running in the Google Cloud Platform uses references to understand dependencies between resources and properly order the operations it performs on -a configuration. This version doesn't yet have this functionality, but will have -it shortly. +a configuration. (This version doesn't yet order operations to satisfy dependencies, +but it will soon.) A reference follows this syntax: **$(ref.NAME.PATH)**, where _NAME_ is the name of the resource being referenced, and _PATH_ is a JSON path to the value in the From 44c264b9ed47e2a5af6716a56b41b580fa8bcb30 Mon Sep 17 00:00:00 2001 From: Jack Greenfield Date: Mon, 9 Nov 2015 00:14:19 -0800 Subject: [PATCH 06/27] Update design.md --- docs/design/design.md | 152 ++++++++++++++++++++---------------------- 1 file changed, 73 insertions(+), 79 deletions(-) diff --git a/docs/design/design.md b/docs/design/design.md index a145a71c9..3ed47cd0f 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -1,9 +1,9 @@ # Deployment Manager Design ## Overview -Deployment Manager is a service which can be run in a Kubernetes cluster that +Deployment Manager is a service that runs in a Kubernetes cluster. It provides a declarative configuration language to describe Kubernetes -resources and a mechanism for deploying, updating, and deleting configurations. +resources, and a mechanism for deploying, updating, and deleting configurations. This document describes the configuration language, object model, and architecture of the service in detail. @@ -41,8 +41,8 @@ resources: Resources can reference values from other resources. The version of Deployment Manager running in the Google Cloud Platform uses references to understand dependencies between resources and properly order the operations it performs on -a configuration. (This version doesn't yet order operations to satisfy dependencies, -but it will soon.) +a configuration. (This version of DM doesn't yet order operations to satisfy +dependencies, but it will soon.) A reference follows this syntax: **$(ref.NAME.PATH)**, where _NAME_ is the name of the resource being referenced, and _PATH_ is a JSON path to the value in the @@ -58,6 +58,7 @@ In this case, _my-service_ is the name of the resource, and _metadata.name_ is the JSON path to the value being referenced. ### Configurable Resources + Configurable resources are the primitive resources that can be configured in Deployment Manager, including: @@ -67,17 +68,17 @@ Deployment Manager, including: Deployment Manager processes configurable resources by passing their configuration properties directly to kubectl on the cluster to create, update, -or delete the resource. +or delete the resources. ### Templates Templates are abstract types that can be created using Python or -[Jinja](http://jinja.pocoo.org/). Templates take a set of properties and must -output a valid YAML configuration string. Properties are bound to values when a -template is instantiated in a configuration. +[Jinja](http://jinja.pocoo.org/). A template takes a set of properties as input, +and must output a valid YAML configuration string. Properties are bound to +values when a template is instantiated in a configuration. -Templates are expanded as a pre-processing step before configurable resources +Templates are expanded in a pre-processing step before configurable resources are processed. They can output configurations containing configurable resources, -or additional nested templates. Nested templates will be processed recursively. +or additional nested templates. Nested templates are processed recursively. An example of a template in python is: @@ -97,7 +98,7 @@ def GenerateConfig(context): return yaml.dump({'resources': resources}) ``` -and in Jinja is: +and in Jinja: ``` resources: @@ -108,22 +109,22 @@ resources: ... ``` -Templates provide access to several sets of data, which can be used for -parameterizing or further customizing a configuration: +Templates provide access to multiple sets of data, which can be used for +parameterizing or further customizing configurations: -* env: a map of values defined by Deployment Manager, including _deployment_, - _name_, and _type_ +* env: a map of key/value pairs from the environment, including pairs +defined by Deployment Manager, such as _deployment_, _name_, and _type_ * properties: a map of the key/value pairs passed in the properties section when instantiating the template -* imports: a map of import file name to file contents of all imports originally - specified for the configuration +* imports: a map of import file names to file contents of all imports +originally specified for the configuration In Python, this data is available from the _context_ object passed into the _GenerateConfig_ method. ### Template Schemas -A schema can be provided for a template. The schema describes the template in -more details, including: +A schema can be optionally provided for a template. The schema describes +the template in more detail, including: * info: more information about the template, including long description and title @@ -131,7 +132,7 @@ more details, including: * required: properties which are required when instantiating the template * properties: JSON Schema descriptions of each property the template accepts -An example of a template schema is: +Here's an example of a template schema: ``` info: @@ -151,19 +152,18 @@ properties: default: prop-value ``` -Schemas are used by Deployment Manager to validate properties being used during -template instantiation and provide default value semantics on properties. +Schemas are used by Deployment Manager to validate properties during +template instantiation, and to provide default values. -Schemas must be imported along-side the templates which they describe when -passing configuration to Deployment Manager. +Schemas must be imported with the templates they describe, when passing +configuration to Deployment Manager. ### Instantiating Templates -Templates can be instantiated in the same way that a configurable resource is -used. They can be used in two different ways, either passed to the API as an +Templates can be used in two different ways: either passed to the API as an imported file, or used from a public HTTP endpoint. #### Imported Templates -Templates may be imported as part of the target configuration and used +Templates can be imported as part of the target configuration, and used directly, for example: ``` @@ -177,15 +177,15 @@ resources: prop1: prop-value ``` -The _imports_ list is not understood by the Deployment Manager service, but is a -directive to client-side tooling to specify what additional files should be -included when passing a configuration to the API. +The _imports_ list is not understood by the Deployment Manager service. +It's a directive used by client-side tools to specify what additional files +should be included when passing a configuration to the API. Using the Deployment Manager API, these templates can be included in the -imports section of the _targetConfig_. +imports section of the _configuration_. #### External Templates -Templates may also be used from a public HTTP endpoint, for example: +Templates can also be used from a public HTTP endpoint. For example: ``` resources: @@ -197,10 +197,10 @@ resources: The service will process external templates as follows: -1. fetch the external template as an import -1. attempt to fetch the schema for the template, using - _.schema_ as the schema path -1. repeat for any sub-imports found in the schema file +1. Fetch the external template as an import +1. Attempt to fetch the schema for the template, using +_.schema_ as the schema path +1. Repeat for any sub-imports found in the schema file When fetching schema files and sub-imports, the base path of the external template is used for relative paths. @@ -213,15 +213,15 @@ Deployments are the primary resource in the Deployment Manager service. The inputs to a deployment are: * name -* targetConfig +* configuration -When creating a deployment, users pass their YAML configuration, as well as any -import files (templates, datafiles, etc.) in as the _targetConfig_. +When creating a deployment, users pass their configuration, +as well as any import files (templates, datafiles, etc.), all encoded in `YAML`, +in as the _configuration_. -Creating, updating and deleting a deployment creates a new manifest for the -deployment, and then processes the new configuration. In the case of deleting a -deployment, the deployment is first updated to an empty manifest containing no -resources, and then is removed from the system. +Creating, updating or deleting a deployment creates a new manifest for the +deployment. When deleting a deployment, the deployment is first updated to +an empty manifest containing no resources, and then removed from the system. Deployments are available at the HTTP endpoint: @@ -230,15 +230,12 @@ http://manager-service/deployments ``` ### Manifests -A manifest is created for a deployment every time it is mutated, including -creation, update, and deletion. - -A manifest contains three major pieces of data: +A manifest is created for a deployment every time it is changed. It contains +three key components: -* inputConfig: the original input configuration for the manifest, including YAML - configuration and imports -* expandedConfig: the final expanded configuration to be used when processing - resources for the manifest +* inputConfig: the original input configuration for the manifest +* expandedConfig: the expanded configuration to be used when processing resources +* for the manifest * layout: the hierarchical structure of the manifest Manifests are available at the HTTP endpoint: @@ -248,6 +245,7 @@ http://manager-service/deployments//manifests ``` #### Expanded Configuration + Given a new _inputConfig_, Deployment Manager expands all template instantiations recursively until there is a flat set of configurable resources. This final set is stored as the _expandedConfig_ and is used during resource @@ -321,53 +319,49 @@ in more detail below. Currently there are two caveats in the design of the service: -* Synchronous API: the API is currently designed to block on all processing for +* Synchronous API: the API currently blocks on all processing for a deployment request. In the future, this design will change to an asynchronous operation-based mode. -* Non-persistence: the service currently stores all metadata in memory, so will - lose all knowledge of deployments and their metadata on restart. In the - future, the service will persist all deployment metadata in the cluster. +* Non-persistence: the service currently stores all metadata in memory, + so it will lose all knowledge of deployments and their metadata on restart. + In the future, the service will persist all deployment metadata. ### Manager + The **manager** service acts as both the API server and the workflow engine for -processing deployments. The process for a deployment is: +processing deployments. It uses the following process: 1. Create a new deployment with a manifest containing _inputConfig_ from the user request -1. Call out to **expandybird** service to perform expansion on the _inputConfig_ +1. Call out to he **expandybird** service to expand the _inputConfig_ 1. Store the resulting _expandedConfig_ and _layout_ -1. Call out to **resourcifier** service to perform processing on resources from - the _expandedConfig_ +1. Call out to the **resourcifier** service to perform processing on resources +from the _expandedConfig_ 1. Respond with success or error messages to the original API request -The manager is responsible for all persistence of metadata associated with +The manager is responsible for saving the metadata associated with deployments, manifests, type instances, and other resources in the Deployment Manager model. ### Expandybird -The **expandybird** service takes in input configurations, including the YAML -configuration and import files, performs all template expansion, and returns the -resulting flat configuration and layout. It is completely stateless and handles -requests synchronously. -Because templates are Python or Jinja, the actual expansion process is performed -in a sub-process running a Python interpreter. A new sub-process is created for -every request to expandybird. +The **expandybird** service takes in input configurations, performs all template +expansions, and returns the resulting flat configuration and layout. It is completely +stateless. -Currently expansion is not sandboxed, but the intention of templates is to be -reproducable hermetically sealed entities, so future designs may -introduce a sandbox to limit external interaction like network and disk access -during expansion. +Because templates are written in Python or Jinja, the actual expansion process +is performed in a sub-process that runs a Python interpreter. A new sub-process +is created for every request to expandybird. + +Currently, expansion is not sandboxed, but templates should be reproducable, +hermetically sealed entities. Future designs may therefore, introduce a sandbox to +limit external interaction, such as network or disk access, during expansion. ### Resourcifier + The **resourcifier** service takes in flat expanded configurations containing only configurable resources, and makes the respective kubectl calls to process -each resource. It is completely stateless and handles requests synchronously. - -Processing may be to create, update, or delete a resource, -depending on the request. The resourcifier handles references, and is the major -workflow engine for resource processing. In the future. it will also handle -dependencies between resources, as described earlier. +each resource. It is totally stateless, and handles requests synchronously. -The resourcifier service returns either success or error messages encountered -during resource processing. +The resourcifier returns either success or error messages encountered during +resource processing. From 79387903e72ce451d868c6a245a700bb6aabbc25 Mon Sep 17 00:00:00 2001 From: Jack Greenfield Date: Mon, 9 Nov 2015 00:21:19 -0800 Subject: [PATCH 07/27] Update design.md --- docs/design/design.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/design/design.md b/docs/design/design.md index 3ed47cd0f..6a7e69f9b 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -67,10 +67,11 @@ Deployment Manager, including: * Service Deployment Manager processes configurable resources by passing their -configuration properties directly to kubectl on the cluster to create, update, -or delete the resources. +configuration properties directly to kubectl to create, update, or delete them +in the cluster. ### Templates + Templates are abstract types that can be created using Python or [Jinja](http://jinja.pocoo.org/). A template takes a set of properties as input, and must output a valid YAML configuration string. Properties are bound to @@ -159,10 +160,12 @@ Schemas must be imported with the templates they describe, when passing configuration to Deployment Manager. ### Instantiating Templates + Templates can be used in two different ways: either passed to the API as an imported file, or used from a public HTTP endpoint. #### Imported Templates + Templates can be imported as part of the target configuration, and used directly, for example: @@ -185,6 +188,7 @@ Using the Deployment Manager API, these templates can be included in the imports section of the _configuration_. #### External Templates + Templates can also be used from a public HTTP endpoint. For example: ``` @@ -206,9 +210,11 @@ When fetching schema files and sub-imports, the base path of the external template is used for relative paths. ## API Model + Deployment Manager exposes a set of RESTful collections over HTTP/JSON. ### Deployments + Deployments are the primary resource in the Deployment Manager service. The inputs to a deployment are: @@ -230,6 +236,7 @@ http://manager-service/deployments ``` ### Manifests + A manifest is created for a deployment every time it is changed. It contains three key components: @@ -252,6 +259,7 @@ This final set is stored as the _expandedConfig_ and is used during resource processing. #### Layout + Users can use templates to build a rich, deep hierarchical architecture in their configuration. Expansion flattens this hierarchy and removes the template relationships from the configuration to create a format optimized for the process From da8ba9564cc3b9e00223854095e50bc76f661b81 Mon Sep 17 00:00:00 2001 From: vaikas-google Date: Mon, 9 Nov 2015 13:55:16 -0800 Subject: [PATCH 08/27] check for existence of schema file when fetching the download URL --- registry/github_registry.go | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/registry/github_registry.go b/registry/github_registry.go index e812587df..4b9265eab 100644 --- a/registry/github_registry.go +++ b/registry/github_registry.go @@ -60,21 +60,32 @@ func (g *GithubRegistry) List() ([]Type, error) { return retTypes, nil } -// GetURL fetches the download URL for a given Type. +// GetURL fetches the download URL for a given Type and checks for existence of a schema file. func (g *GithubRegistry) GetURL(t Type) (string, error) { _, dc, _, err := g.client.Repositories.GetContents(g.owner, g.repository, TypesDir+"/"+t.Name+"/"+t.Version, nil) if err != nil { log.Printf("Failed to list types : %v", err) return "", err } + var downloadURL, typeName, schemaName string for _, f := range dc { if *f.Type == "file" { if *f.Name == t.Name+".jinja" || *f.Name == t.Name+".py" { - return *f.DownloadURL, nil + typeName = *f.Name + downloadURL = *f.DownloadURL + } + if *f.Name == t.Name+".jinja.schema" || *f.Name == t.Name+".py.schema" { + schemaName = *f.Name } } } - return "", fmt.Errorf("Can not find type %s:%s", t.Name, t.Version) + if downloadURL == "" { + return "", fmt.Errorf("Can not find type %s:%s", t.Name, t.Version) + } + if schemaName == typeName + ".schema" { + return downloadURL, nil + } + return "", fmt.Errorf("Can not find schema for %s:%s, expected to find %s", t.Name, t.Version, typeName + ".schema") } func (g *GithubRegistry) getDirs(dir string) ([]string, error) { From 7650b916b912c6f62bec89c61c53d1f20aa75082 Mon Sep 17 00:00:00 2001 From: vaikas-google Date: Mon, 9 Nov 2015 13:59:20 -0800 Subject: [PATCH 09/27] add newlines to listing types --- dm/dm.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dm/dm.go b/dm/dm.go index 812a40e56..3952f669b 100644 --- a/dm/dm.go +++ b/dm/dm.go @@ -98,15 +98,15 @@ func main() { log.Fatalf("Cannot list %v err") } - fmt.Printf("Types:") + fmt.Printf("Types:\n") for _, t := range types { - fmt.Printf("%s:%s", t.Name, t.Version) + fmt.Printf("%s:%s\n", t.Name, t.Version) downloadURL, err := git.GetURL(t) if err != nil { log.Printf("Failed to get download URL for type %s:%s", t.Name, t.Version) } - fmt.Printf("\tdownload URL: %s", downloadURL) + fmt.Printf("\tdownload URL: %s\n", downloadURL) } case "describe": fmt.Printf("this feature is not yet implemented") From 87fba0e22fd8a89fb2d1eed7332fde540e95228c Mon Sep 17 00:00:00 2001 From: Eric Ho Date: Tue, 10 Nov 2015 12:58:08 +0800 Subject: [PATCH 10/27] Build and push image to other registry Command: `make push DOCKER_REGISTRY=index.docker.io PROJECT=(your account)` --- expandybird/Makefile | 11 ++++++++--- manager/Makefile | 13 +++++++++---- resourcifier/Makefile | 9 +++++++-- 3 files changed, 24 insertions(+), 9 deletions(-) diff --git a/expandybird/Makefile b/expandybird/Makefile index 6667a064a..eb6ecc41a 100644 --- a/expandybird/Makefile +++ b/expandybird/Makefile @@ -1,17 +1,22 @@ -# Makefile for the Docker image gcr.io/$(PROJECT)/expandybird +# Makefile for the Docker image $(DOCKER_REGISTRY)/$(PROJECT)/expandybird # MAINTAINER: Jack Greenfield # If you update this image please check the tag value before pushing. .PHONY : all build test push container clean -PREFIX := gcr.io/$(PROJECT) +DOCKER_REGISTRY := gcr.io +PREFIX := $(DOCKER_REGISTRY)/$(PROJECT) IMAGE := expandybird TAG := latest DIR := . push: container +ifeq ($(DOCKER_REGISTRY),gcr.io) gcloud docker push $(PREFIX)/$(IMAGE):$(TAG) +else + docker push $(PREFIX)/$(IMAGE):$(TAG) +endif container: expandybird cp $(shell which expandybird) . @@ -25,4 +30,4 @@ expandybird: clean: -docker rmi $(PREFIX)/$(IMAGE):$(TAG) rm -f expandybird - + diff --git a/manager/Makefile b/manager/Makefile index 7c100ce60..afbaeaf88 100644 --- a/manager/Makefile +++ b/manager/Makefile @@ -1,10 +1,11 @@ -# Makefile for the Docker image gcr.io/$(PROJECT)/manager +# Makefile for the Docker image $(DOCKER_REGISTRY)/$(PROJECT)/manager # MAINTAINER: Jack Greenfield # If you update this image please check the tag value before pushing. .PHONY : all build test push container clean .project -PREFIX := gcr.io/$(PROJECT) +DOCKER_REGISTRY := gcr.io +PREFIX := $(DOCKER_REGISTRY)/$(PROJECT) IMAGE := manager TAG := latest @@ -12,11 +13,15 @@ ROOT_DIR := $(abspath ./..) DIR = $(ROOT_DIR) push: container +ifeq ($(DOCKER_REGISTRY),gcr.io) gcloud docker push $(PREFIX)/$(IMAGE):$(TAG) +else + docker push $(PREFIX)/$(IMAGE):$(TAG) +endif -container: +container: docker build -t $(PREFIX)/$(IMAGE):$(TAG) -f Dockerfile $(DIR) clean: -docker rmi $(PREFIX)/$(IMAGE):$(TAG) - \ No newline at end of file + diff --git a/resourcifier/Makefile b/resourcifier/Makefile index 3ae2743e1..54956e717 100644 --- a/resourcifier/Makefile +++ b/resourcifier/Makefile @@ -1,10 +1,11 @@ -# Makefile for the Docker image gcr.io/$(PROJECT)/resourcifier +# Makefile for the Docker image $(DOCKER_REGISTRY)/$(PROJECT)/resourcifier # MAINTAINER: Jack Greenfield # If you update this image please check the tag value before pushing. .PHONY : all build test push container clean -PREFIX := gcr.io/$(PROJECT) +DOCKER_REGISTRY := gcr.io +PREFIX := $(DOCKER_REGISTRY)/$(PROJECT) IMAGE := resourcifier TAG := latest @@ -12,7 +13,11 @@ ROOT_DIR := $(abspath ./..) DIR = $(ROOT_DIR) push: container +ifeq ($(DOCKER_REGISTRY),gcr.io) gcloud docker push $(PREFIX)/$(IMAGE):$(TAG) +else + docker push $(PREFIX)/$(IMAGE):$(TAG) +endif container: docker build -t $(PREFIX)/$(IMAGE):$(TAG) -f Dockerfile $(DIR) From ebd4ab57b17caed767771c4184b6fc99495c71b0 Mon Sep 17 00:00:00 2001 From: Brendan Melville Date: Wed, 11 Nov 2015 16:58:22 -0800 Subject: [PATCH 11/27] Resourcifier and expandybird errors now propagate through API. This also cleans up some error messages that were adding lots of newlines to the end of errors as they passed through the system. --- expandybird/service/service.go | 4 ++-- manager/manager/deployer.go | 12 ++++++------ manager/manager/expander.go | 32 ++++++++++++++++---------------- 3 files changed, 24 insertions(+), 24 deletions(-) diff --git a/expandybird/service/service.go b/expandybird/service/service.go index c27aba6b8..3e1c0a78a 100644 --- a/expandybird/service/service.go +++ b/expandybird/service/service.go @@ -68,14 +68,14 @@ func NewExpansionHandler(backend expander.Expander) restful.RouteFunction { output, err := backend.ExpandTemplate(template) if err != nil { - message := fmt.Sprintf("error (%s) expanding template:\n%v\n", err, template) + message := fmt.Sprintf("error expanding template: %s", err) logAndReturnErrorFromHandler(http.StatusBadRequest, message, resp) return } response, err := expander.NewExpansionResponse(output) if err != nil { - message := fmt.Sprintf("error (%s) marshaling output:\n%v\n", err, output) + message := fmt.Sprintf("error marshaling output: %s", err) logAndReturnErrorFromHandler(http.StatusBadRequest, message, resp) return } diff --git a/manager/manager/deployer.go b/manager/manager/deployer.go index aca278557..430b22367 100644 --- a/manager/manager/deployer.go +++ b/manager/manager/deployer.go @@ -100,7 +100,7 @@ func (d *deployer) PutConfiguration(configuration *Configuration) error { func (d *deployer) callServiceWithConfiguration(method, operation string, configuration *Configuration) error { callback := func(e error) error { - return fmt.Errorf("cannot %s configuration (%s)", operation, e) + return fmt.Errorf("cannot %s configuration: %s", operation, e) } y, err := yaml.Marshal(configuration) @@ -129,14 +129,14 @@ func (d *deployer) callService(method, url string, reader io.Reader, callback fo } defer response.Body.Close() - if response.StatusCode < http.StatusOK || - response.StatusCode >= http.StatusMultipleChoices { - err := fmt.Errorf("deployer service response:\n%v\n", response) + body, err := ioutil.ReadAll(response.Body) + if err != nil { return nil, callback(err) } - body, err := ioutil.ReadAll(response.Body) - if err != nil { + if response.StatusCode < http.StatusOK || + response.StatusCode >= http.StatusMultipleChoices { + err := fmt.Errorf("resourcifier response:\n%s", body) return nil, callback(err) } diff --git a/manager/manager/expander.go b/manager/manager/expander.go index 0a80f9a65..3456f8571 100644 --- a/manager/manager/expander.go +++ b/manager/manager/expander.go @@ -54,7 +54,7 @@ func (e *expander) getBaseURL() string { } func expanderError(t *Template, err error) error { - return fmt.Errorf("cannot expand template named %s (%s):\n%s\n", t.Name, err, t.Content) + return fmt.Errorf("cannot expand template named %s (%s):\n%s", t.Name, err, t.Content) } // ExpanderResponse gives back a layout, which has nested structure @@ -117,7 +117,7 @@ func (e *expander) ExpandTemplate(t Template) (*ExpandedTemplate, error) { // 4. If type resolution resulted in new imports being available, return to 2. config := &Configuration{} if err := yaml.Unmarshal([]byte(t.Content), config); err != nil { - e := fmt.Errorf("Unable to unmarshal configuration (%s): %s\n", err, t.Content) + e := fmt.Errorf("Unable to unmarshal configuration (%s): %s", err, t.Content) return nil, e } @@ -127,7 +127,7 @@ func (e *expander) ExpandTemplate(t Template) (*ExpandedTemplate, error) { // Start things off by attempting to resolve the templates in a first pass. newImp, err := e.typeResolver.ResolveTypes(config, t.Imports) if err != nil { - e := fmt.Errorf("type resolution failed:%s\n", err) + e := fmt.Errorf("type resolution failed: %s", err) return nil, expanderError(&t, e) } @@ -137,7 +137,7 @@ func (e *expander) ExpandTemplate(t Template) (*ExpandedTemplate, error) { // Now expand with everything imported. result, err := e.expandTemplate(&t) if err != nil { - e := fmt.Errorf("template expansion:%s\n", err) + e := fmt.Errorf("template expansion: %s", err) return nil, expanderError(&t, e) } @@ -152,7 +152,7 @@ func (e *expander) ExpandTemplate(t Template) (*ExpandedTemplate, error) { newImp, err = e.typeResolver.ResolveTypes(result.Config, nil) if err != nil { - e := fmt.Errorf("type resolution failed:%s\n", err) + e := fmt.Errorf("type resolution failed: %s", err) return nil, expanderError(&t, e) } @@ -167,7 +167,7 @@ func (e *expander) ExpandTemplate(t Template) (*ExpandedTemplate, error) { content, err = yaml.Marshal(result.Config) t.Content = string(content) if err != nil { - e := fmt.Errorf("Unable to unmarshal response from expander (%s): %s\n", + e := fmt.Errorf("Unable to unmarshal response from expander (%s): %s", err, result.Config) return nil, expanderError(&t, e) } @@ -183,31 +183,31 @@ func (e *expander) expandTemplate(t *Template) (*ExpandedTemplate, error) { response, err := http.Post(e.getBaseURL(), "application/json", ioutil.NopCloser(bytes.NewReader(j))) if err != nil { - e := fmt.Errorf("http POST failed:%s\n", err) + e := fmt.Errorf("http POST failed: %s", err) return nil, e } - defer response.Body.Close() - - if response.StatusCode != http.StatusOK { - err := fmt.Errorf("expander service response:%v", response) - return nil, err - } + defer response.Body.Close() body, err := ioutil.ReadAll(response.Body) if err != nil { - e := fmt.Errorf("error reading response:%s\n", err) + e := fmt.Errorf("error reading response: %s", err) return nil, e } + if response.StatusCode != http.StatusOK { + err := fmt.Errorf("expandybird response:\n%s", body) + return nil, err + } + er := &ExpansionResponse{} if err := json.Unmarshal(body, er); err != nil { - e := fmt.Errorf("cannot unmarshal response body (%s):%s\n", err, body) + e := fmt.Errorf("cannot unmarshal response body (%s):%s", err, body) return nil, e } template, err := er.Unmarshal() if err != nil { - e := fmt.Errorf("cannot unmarshal response yaml (%s):%v\n", err, er) + e := fmt.Errorf("cannot unmarshal response yaml (%s):%v", err, er) return nil, e } From 4965c3ac0ca27f4038cf4ce08907add8fb2dd07d Mon Sep 17 00:00:00 2001 From: Brendan Melville Date: Wed, 11 Nov 2015 17:00:17 -0800 Subject: [PATCH 12/27] Adding an example config to the redis type. This will allow for testing of the type locally. --- types/redis/v1/redis.yaml | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 types/redis/v1/redis.yaml diff --git a/types/redis/v1/redis.yaml b/types/redis/v1/redis.yaml new file mode 100644 index 000000000..12901829d --- /dev/null +++ b/types/redis/v1/redis.yaml @@ -0,0 +1,7 @@ +imports: +- path: redis.jinja + +resources: +- name: redis + type: redis.jinja + properties: null From f750be679732e8eb91be11e87a7737bc6462c9a0 Mon Sep 17 00:00:00 2001 From: Gaoce Date: Thu, 12 Nov 2015 09:13:36 +0800 Subject: [PATCH 13/27] fix typo --- docs/design/design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/design.md b/docs/design/design.md index 6a7e69f9b..a6b19e5ef 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -341,7 +341,7 @@ processing deployments. It uses the following process: 1. Create a new deployment with a manifest containing _inputConfig_ from the user request -1. Call out to he **expandybird** service to expand the _inputConfig_ +1. Call out to the **expandybird** service to expand the _inputConfig_ 1. Store the resulting _expandedConfig_ and _layout_ 1. Call out to the **resourcifier** service to perform processing on resources from the _expandedConfig_ From 08568d8e61bf554a8cdb2a057ff209f57b206952 Mon Sep 17 00:00:00 2001 From: vaikas-google Date: Mon, 9 Nov 2015 13:55:16 -0800 Subject: [PATCH 14/27] check for existence of schema file when fetching the download URL --- registry/github_registry.go | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/registry/github_registry.go b/registry/github_registry.go index e812587df..4b9265eab 100644 --- a/registry/github_registry.go +++ b/registry/github_registry.go @@ -60,21 +60,32 @@ func (g *GithubRegistry) List() ([]Type, error) { return retTypes, nil } -// GetURL fetches the download URL for a given Type. +// GetURL fetches the download URL for a given Type and checks for existence of a schema file. func (g *GithubRegistry) GetURL(t Type) (string, error) { _, dc, _, err := g.client.Repositories.GetContents(g.owner, g.repository, TypesDir+"/"+t.Name+"/"+t.Version, nil) if err != nil { log.Printf("Failed to list types : %v", err) return "", err } + var downloadURL, typeName, schemaName string for _, f := range dc { if *f.Type == "file" { if *f.Name == t.Name+".jinja" || *f.Name == t.Name+".py" { - return *f.DownloadURL, nil + typeName = *f.Name + downloadURL = *f.DownloadURL + } + if *f.Name == t.Name+".jinja.schema" || *f.Name == t.Name+".py.schema" { + schemaName = *f.Name } } } - return "", fmt.Errorf("Can not find type %s:%s", t.Name, t.Version) + if downloadURL == "" { + return "", fmt.Errorf("Can not find type %s:%s", t.Name, t.Version) + } + if schemaName == typeName + ".schema" { + return downloadURL, nil + } + return "", fmt.Errorf("Can not find schema for %s:%s, expected to find %s", t.Name, t.Version, typeName + ".schema") } func (g *GithubRegistry) getDirs(dir string) ([]string, error) { From 34c9e42d27470242a60790f8ca82c8f02a4a4d98 Mon Sep 17 00:00:00 2001 From: vaikas-google Date: Mon, 9 Nov 2015 13:59:20 -0800 Subject: [PATCH 15/27] add newlines to listing types --- dm/dm.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dm/dm.go b/dm/dm.go index 812a40e56..3952f669b 100644 --- a/dm/dm.go +++ b/dm/dm.go @@ -98,15 +98,15 @@ func main() { log.Fatalf("Cannot list %v err") } - fmt.Printf("Types:") + fmt.Printf("Types:\n") for _, t := range types { - fmt.Printf("%s:%s", t.Name, t.Version) + fmt.Printf("%s:%s\n", t.Name, t.Version) downloadURL, err := git.GetURL(t) if err != nil { log.Printf("Failed to get download URL for type %s:%s", t.Name, t.Version) } - fmt.Printf("\tdownload URL: %s", downloadURL) + fmt.Printf("\tdownload URL: %s\n", downloadURL) } case "describe": fmt.Printf("this feature is not yet implemented") From 5b5278631a0791576448beed4e0a81d12bc39f16 Mon Sep 17 00:00:00 2001 From: jackgr Date: Thu, 12 Nov 2015 05:52:37 -0800 Subject: [PATCH 16/27] Fix two bugs in replicatedservice.py --- types/replicatedservice/v1/replicatedservice.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/types/replicatedservice/v1/replicatedservice.py b/types/replicatedservice/v1/replicatedservice.py index 5c63dd32e..b5a1a52aa 100644 --- a/types/replicatedservice/v1/replicatedservice.py +++ b/types/replicatedservice/v1/replicatedservice.py @@ -41,9 +41,9 @@ def GenerateConfig(context): 'properties': { 'apiVersion': 'v1', 'kind': 'Service', - 'namespace': namespace, 'metadata': { 'name': service_name, + 'namespace': namespace, 'labels': GenerateLabels(context, service_name), }, 'spec': { @@ -63,9 +63,9 @@ def GenerateConfig(context): 'properties': { 'apiVersion': 'v1', 'kind': 'ReplicationController', - 'namespace': namespace, 'metadata': { 'name': rc_name, + 'namespace': namespace, 'labels': GenerateLabels(context, rc_name), }, 'spec': { @@ -101,11 +101,11 @@ def GenerateConfig(context): def GenerateLabels(context, name): """Generates labels either from the context.properties['labels'] or - generates a default label 'name':name + generates a default label 'app':name We make a deep copy of the context.properties['labels'] section to avoid linking in the yaml document, which I believe reduces readability of the - expanded template. If no labels are given, generate a default 'name':name. + expanded template. If no labels are given, generate a default 'app':name. Args: context: Template context, which can contain the following properties: @@ -115,7 +115,7 @@ def GenerateLabels(context, name): A dict containing labels in a name:value format """ tmp_labels = context.properties.get('labels', None) - ret_labels = {'name': name} + ret_labels = {'app': name} if isinstance(tmp_labels, dict): for key, value in tmp_labels.iteritems(): ret_labels[key] = value From c7af9ab145e360fb0af6135157a329fbd31bf708 Mon Sep 17 00:00:00 2001 From: Brendan Melville Date: Thu, 12 Nov 2015 11:30:38 -0800 Subject: [PATCH 17/27] Fixing jinja template for redis to allow for no properties. Previously it assumed properties would exist. --- types/redis/v1/redis.jinja | 2 +- types/redis/v1/redis.yaml | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/types/redis/v1/redis.jinja b/types/redis/v1/redis.jinja index b7d8451f3..0a19c863f 100644 --- a/types/redis/v1/redis.jinja +++ b/types/redis/v1/redis.jinja @@ -1,5 +1,5 @@ {% set REDIS_PORT = 6379 %} -{% set WORKERS = properties['workers'] or 2 %} +{% set WORKERS = properties['workers'] if properties and properties['workers'] else 2 %} resources: - name: redis-master diff --git a/types/redis/v1/redis.yaml b/types/redis/v1/redis.yaml index 12901829d..0b1b87b69 100644 --- a/types/redis/v1/redis.yaml +++ b/types/redis/v1/redis.yaml @@ -4,4 +4,3 @@ imports: resources: - name: redis type: redis.jinja - properties: null From 5fc6d3493d16adf060332e1e14aa6e53d74b3a5b Mon Sep 17 00:00:00 2001 From: Brendan Melville Date: Thu, 12 Nov 2015 14:24:39 -0800 Subject: [PATCH 18/27] Adding documentation about type registries. --- docs/types/registry.md | 82 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/types/registry.md diff --git a/docs/types/registry.md b/docs/types/registry.md new file mode 100644 index 000000000..ca5859ff7 --- /dev/null +++ b/docs/types/registry.md @@ -0,0 +1,82 @@ +# Type Registry + +The Deployment Manager client allows you to deploy +[template types](https://github.com/kubernetes/deployment-manager/blob/master/docs/design/design.md#templates) +directly from a Github repository. In order for a repository to integrate with +Deployment Manager, it must store Deployment Manager templates in a manner that +conforms to the required **Type Registry** structure detailed below. + +## File structure +The repository must use the following file structure to store Deployment +Manager template types: + +``` +/ + types/ + / + / + + / + ... + / + ... +``` + +## Versions +Each type may contain any number of versions. Versions may be as simple as +`v1`, `latest`, or `head`, or may be more complex to contain major and minor +version numbers. Version format is user-defined. + +## Type files +Each type version must contain a top-level Deployment Manager template, named +either `.py` or `.jinja`, depending on the templating +language used for the type. + +A +[template schema](https://github.com/kubernetes/deployment-manager/blob/master/docs/design/design.md#template-schemas) +must also be present, named `