If you’ve worked with Kubernetes, odds are you’ve seen YAML manifest files that can be thousands of lines long. At Proton, we’re heavy Kubernetes users, and at one point relied on static resource definitions for all of our services. Because we don’t change these often — and never as part of a regular deployment cycle — it was clear these long YAML files would easily become outdated.

Further, they were a big pain to develop with: trying to figure out why a pod won’t start, and finally realizing that you’d changed the name of a configmap without modifying the volume 50 lines below, or debugging incorrect behavior when you’d really just made a typo copying an environment variable across multiple definitions. We needed a solution for more dynamic templating.

We tried kustomize first, and realized it couldn’t deal with exactly what we were hoping to abstract from our manifests. There wasn’t enough flexibility, and all in all we would wind up with 15 different files for one application. Other tools like Helm, ksonnet, or yq also didn’t perfectly fit our use case.

In the end, we came up with our own solution. At the heart of it, Kubernetes manifests don’t need to be in YAML. It’s also possible to use JSON. The Jsonnet templating language gives us the ability to use variables, conditionals, functions, etc. to generate JSON, and feels more like writing JavaScript in some cases than writing a template at all. This ticked all our boxes: giving us the repeatability of a templating environment with the power of something closer to a programming language. We combine Jsonnet with ArgoCD to scale our deployments across thousands of microservices.

Geting Started

Consider this simplified application running as a Kubernetes deployment. We’ll need a few environments for development, testing, and QA, so the deployment will be running across multiple namespaces, and with different versions of our image. The rest of the application should remain consistent across environments, except we’ll also want to ensure each running container is pointed at the correct database. To deploy this with our Jsonnet framework we start by importing custom libraries containing Kubernetes resource definitions, common fields used across applications, and config values for the application. Since the image tag is updated frequently and the namespace is necessary to decide what the rest of the specification will look like, we use the image-tag and namespace variables on the command line to output the correct definitions.

local common = import "../common.libsonnet";
local k8s = import "../k8s.libsonnet";
local namespace = std.extVar("namespace");
local image_tag = std.extVar("image-tag");

Now, a container can be defined like so:

local env_vars = [
    { "name": "DATABASE_HOST",
    "value": if std.setMember(namespace,["dev", "sandbox"]) then common.dev_db_host else common.db_host }
];

local liveness = k8s.probe(8000, delay=10, period=30, failure_threshold=2, type="http")
### etc...

local container = k8s.container_v1(
    image="%sapi:%s" % [common.image_repo, image_tag]
    name="api",
    env_vars=env_vars,
    command=["/bin/bash"],
    args=["-c", "source /home/startup.sh"],
    ports=ports,
    readiness=readiness,
    liveness=liveness,
    startup=startup,
    resources=resources
)

If we need to add an environment variable across all namespaces we add it to the env_vars variable. We can modify how much memory pods are able to consume by updating the resource definition. If it’s necessary to replicate this in a brand new namespace, we can simply specify another namespace string.

And now to make the container into a deployment, with the associated service:

[
    k8s.service(
        metadata=k8s.metadata("api", namespace),
        port=8000,
      load_balancer=true
    ),

    k8s.deployment(
        metadata=k8s.metadata("api", namespace),
      containers=[container],
      replicas=config.replicas[namespace],
      max_surge="100%",
      service_account="secret-reader"
    )
]

Now, instead of modifying and searching for individual YAML files, or commenting out what you don’t want to apply, each configuration can be separately obtained from the same file with the command:

jsonnet -y -V namespace=dev -V image-tag=1.0.1 api.jsonnet | kubectl apply -f -

This uses our Jsonnet setup to generate data Kubernetes understands, allowing us to deploy our service. This framework is a big improvement and makes it much easier to recreate resources or update them if they’re out of date.

A Deeper Example

This, though, may not impress you. Couldn’t you achieve the same thing with kustomize? Or patching? In the above example, yes. In actual production deployments, we may have more complicated needs.

At Proton, we often maintain different deployments for different companies we serve to tailor systems to their requirements. In the simplest case, that means we need to run N models per client, and giving us roughly N*|clients|, just in production.

Imagine we need to manage 15 of these microservices, each with a different configuration — that alone generates 1000 lines of YAML. This is where Jsonnet really shines. For example, we could specify our deployment like this:

local config = import "../models.libsonnet";
local k8s = import "../../k8s.libsonnet";
local networking = import "../../networking/networking.libsonnet";
local serving = import "../serving.libsonnet";
local namespace = std.extVar("namespace");
local company = std.extVar("company");

local company_models = std.objectFieldsAll(config[company][namespace]);
local paths = { [m]: std.extVar(m) for m in company_models };

[ serving.model_deployment(
    company,
    m,
    namespace,
    filepath=paths[m],
    replicas=0
  ) for m in company_models
] +

[ serving.model_service(
    company,
    m,
    namespace
  ) for m in company_models
] +

[ networking.canary(
    k8s.metadata(
      serving.name(company, m),
      namespace
    ),
    8500,
    port_name="grpc",
    port_discovery=true,
    interval="10s",
    max_weight=100,
    step_weight=5,
    webhooks=serving.model_canary_webhooks(company, m, namespace)
  ) for m in company_models
]

A lot is abstracted away into libsonnet files, where we define Kubernetes resources specific to these services. As with the previous service we looked at, you can make changes in one file and have them propagate across each one of the individual services. We can also add minor safeguards by setting defaults — for example, anything in production here has at least two replicas, resource limits, and common annotations, and you’d need to deliberately edit multiple conditionals to change that. If you were creating a new specification in YAML, there’s nothing to hinder not following best practices.

At the top of the file, we define two external variables (company and namespace). These lines allows us to define additional runtime variables, as needed, per the company/namespace configuration:

local company_models = std.objectFieldsAll(config[company][namespace]);
local paths = { [m]: std.extVar(m) for m in company_models };

If that seems like a headache to use on the command line, it is. You’d need to check the configuration file, see which variables you need to input, and know what each value should be before applying any changes. Enter a configuration management tool to tie everything together.

Synchronizing with ArgoCD

Now that we have dynamic Kubernetes “manifests”, how do we ensure that what is defined in source control is also running in our cluster? We close the loop with ArgoCD. It’s a “declarative, GitOps continuous delivery tool for Kubernetes”. With ArgoCD, we can take the Jsonnet specification from above, and manage those services without looking at Jsonnet at all. Here’s what the app manifest looks like with everything tied together:

project: dev
source:
  repoURL: 'git@github.com:mycorp/myproject.git'
  path: kubernetes/serving
  targetRevision: master
  directory:
    jsonnet:
      extVars:
        - name: namespace
          value: dev
        - name: company
          value: proton
        - name: modelx
          value: x/1608991066-v3.pb
        - name: modely
          value: y/1608288912-v3.pb
        - name: modelz
          value: z/1608248241-v1.pb
      #....
destination:
  server: '<https://kubernetes.default.svc>'
  namespace: dev

If we want to update just one of the models, you can now do that with one simple command, without understanding the other pieces of the system:

argocd app set proton-models-dev --jsonnet-ext-var-str modelx=$MODEL_VERSION_PATH

Assuming auto-sync is on, that’s it! Your service is updated, and you can visit the ArgoCD dashboard to see a nice green heart indicating it’s healthy.

This pattern is miles away from where we started. No more managing YAML files. We can simply and flexibly specify our deployments with Jsonnet, and then layer on ArgoCD let us use them. It makes managing a constellation of microservices a cinch.

Managing a complex web of microservices would be impossible without Kubernetes. But as our needs grew, we realized we needed something else on top of it. Using Jsonnet to create flexible Kubernetes definitions and managing them with ArgoCD has let us scale up without missing a beat. We now use this system to power dozens of deployments that control hundreds of pods delivering our tools to users. We think this is a great way to scale our technology, and we hope it’s given you some ideas, too.