•••
This is an opinionated guide on how to write extensible logging middleware for Go web services.
I’ve had a number of requests to add a built-in logger to gorilla/mux and to extend what is logged by gorilla/handlers, and they’re hard to triage. Many of the asks are for different things, since “what” to log, how much to log, and which library to use are not agreed-upon by all. Further, and especially in mux’s case, logging is not the focus of the library, and writing your own logging “middleware” can be simpler than you expect.
The patterns in this guide can be extended to any HTTP middleware use-cases, including authentication & authorization, metrics, tracing, and web security. Logging just happens to be one of the most common use-cases and makes for a great example.
Why is Middleware Useful?
If you’ve been writing Go for a while, you can skip to the code at the end of this post.
Middleware allows us to separate concerns and write composable applications—and in a world of micro-services, allow clearer lines of ownership for specific components.
Specifically:
- Authentication and authorization (“authn” and “authz”) can be handled uniformly: we can both keep it separate from our primary business logic, and/or share the same authn/authz handling across our organization. Separating this can make adding new authentication providers easier, or (importantly) fixing potential security issues easier as a team grows.
- Similar to authn & authz, we can define a set of re-usable logging, metrics & tracing middleware for our applications, so that troubleshooting across services and/or teams isn’t a pot-luck.
- Testing becomes simpler, as we can draw clearer boundaries around each component: noting that integration testing is still important for end-to-end validation.
With this in mind, let’s see how defining “re-usable” middleware in Go actually works.
A Common Middleware Interface
One thing that’s important when writing any middleware is that it be loosely coupled from your choice of framework or router-specific APIs. Handlers should be usable by any HTTP-speaking Go service: if team A chooses net/http, team B chooses gorilla/mux, and team C wants to use Twirp, then our middleware shouldn’t force a choice or be constrained within a particular framework.
Go’s net/http library defines the http.Handler interface, and satisfying this makes it easy to write portable HTTP handling code.
The only method required to satisfy http.Handler is ServeHTTP(http.ResponseWriter, *http.Request) - and the concrete http.HandlerFunc type means that you can convert any type with a matching signature into a type that satisfies http.Handler.
Example:
func ExampleMiddleware(next http.Handler) http.Handler {
// We wrap our anonymous function, and cast it to a http.HandlerFunc
// Because our function signature matches ServeHTTP(w, r), this allows
// our function (type) to implicitly satisify the http.Handler interface.
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
// Logic before - reading request values, putting things into the
// request context, performing authentication
// Important that we call the 'next' handler in the chain. If we don't,
// then request handling will stop here.
next.ServeHTTP(w, r)
// Logic after - useful for logging, metrics, etc.
//
// It's important that we don't use the ResponseWriter after we've called the
// next handler: we may cause conflicts when trying to write the response
}
)
}
This is effectively the recipe for any middleware we want to build. Each middleware component (which is just a http.Handler implementation!) wraps another, performs any work it needs to, and then calls the handler it wrapped via next.ServeHTTP(w, r).
If we need to pass values between handlers, such as the ID of the authenticated user, or a request or trace ID, we can the use the context.Context attached to the *http.Request via the *Request.Context() method introduced back in Go 1.7.
A stack of middleware would look like the below:
router := http.NewServeMux()
router.HandleFunc("/", indexHandler)
// Requests traverse LoggingMiddleware -> OtherMiddleware -> YetAnotherMiddleware -> final handler
configuredRouter := LoggingMiddleware(OtherMiddleware(YetAnotherMiddleware(router))))
log.Fatal(http.ListenAndServe(":8000", configuredRouter))
This looks composable (check!), but what about if we want to inject dependencies or otherwise customize the behaviour of each handler in the stack?
Injecting Dependencies
In the above ExampleMiddleware, we created a simple function that accepted a http.Handler and returned a http.Handler. But what if we wanted to provide our own logger implementation, inject other config, and/or not rely on global singletons?
Let’s take a look at how we can achieve that while still having our middleware accept (and return) http.Handler.
func NewExampleMiddleware(someThing string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// Logic here
// Call the next handler
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
By returning a func(http.Handler) http.Handler we can make the dependencies of our middleware clearer, and allow consumers of our middleware to configure it to their needs.
In our logging example, we make want to pass an application-level logger with some existing configuration—say, the service name, and a timestamp format—to our LoggingMiddleware, without having to copy-paste it or otherwise rely on package globals, which make our code harder to reason about & test.
The Code: LoggingMiddleware
Let’s take everything we’ve learned above, with a middleware function that logs:
- The request method & path
- The status code written to the response, using our own implementation of
http.ResponseWriter (more on this below)
- The duration of the HTTP request & response - until the last bytes are written to the response
- Allows us to inject our own
logger.Log instance from kit/log.
Source on GitHub
// request_logger.go
import (
"net/http"
"runtime/debug"
"time"
log "github.com/go-kit/kit/log"
)
// responseWriter is a minimal wrapper for http.ResponseWriter that allows the
// written HTTP status code to be captured for logging.
type responseWriter struct {
http.ResponseWriter
status int
wroteHeader bool
}
func wrapResponseWriter(w http.ResponseWriter) *responseWriter {
return &responseWriter{ResponseWriter: w}
}
func (rw *responseWriter) Status() int {
return rw.status
}
func (rw *responseWriter) WriteHeader(code int) {
if rw.wroteHeader {
return
}
rw.status = code
rw.ResponseWriter.WriteHeader(code)
rw.wroteHeader = true
return
}
// LoggingMiddleware logs the incoming HTTP request & its duration.
func LoggingMiddleware(logger log.Logger) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
defer func() {
if err := recover(); err != nil {
w.WriteHeader(http.StatusInternalServerError)
logger.Log(
"err", err,
"trace", debug.Stack(),
)
}
}()
start := time.Now()
wrapped := wrapResponseWriter(w)
next.ServeHTTP(wrapped, r)
logger.Log(
"status", wrapped.status,
"method", r.Method,
"path", r.URL.EscapedPath(),
"duration", time.Since(start),
)
}
return http.HandlerFunc(fn)
}
}
Review:
- We implement our own
responseWriter type that captures the status code of a response, allowing us to log it (since it’s not known until the response is written). Importantly, we don’t have to re-implement every method of the http.ResponseWriter - we embed the one we receive, and override only the Status() int and WriteHeader(int) methods, so we can carry state in our .status and .wroteHeader struct fields.
- http.HandlerFunc converts our return type into a http.HandlerFunc, which automatically allows it to satisfy the
ServeHTTP method of http.Handler.
- Our Logger also logs panics (optional, but useful) so we can capture them in our logging system too.
- Because we directly inject the
log.Logger - we can both configure it, and mock it during tests.
- Calling
.Log() allows us to pass whichever values we need - we may not want to log all values at once, but it’s also easy to expand as necessary. There is no “one size fits all” logger.
Notably, I use kit/log here, although you could use any logger you like, including the standard library - noting that you’d be missing the benefits of structured logging if you went down that path.
A Full Example
Below is a full (runnable!) example, using a version of LoggingMiddleware we defined earlier from the elithrar/admission-control package:
// server.go
package main
import (
"fmt"
stdlog "log"
"net/http"
"os"
"github.com/elithrar/admission-control"
log "github.com/go-kit/kit/log"
)
func myHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "hello!")
}
func main() {
router := http.NewServeMux()
router.HandleFunc("/", myHandler)
var logger log.Logger
// Logfmt is a structured, key=val logging format that is easy to read and parse
logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
// Direct any attempts to use Go's log package to our structured logger
stdlog.SetOutput(log.NewStdlibAdapter(logger))
// Log the timestamp (in UTC) and the callsite (file + line number) of the logging
// call for debugging in the future.
logger = log.With(logger, "ts", log.DefaultTimestampUTC, "loc", log.DefaultCaller)
// Create an instance of our LoggingMiddleware with our configured logger
loggingMiddleware := admissioncontrol.LoggingMiddleware(logger)
loggedRouter := loggingMiddleware(router)
// Start our HTTP server
if err := http.ListenAndServe(":8000", loggedRouter); err != nil {
logger.Log("status", "fatal", "err", err)
os.Exit(1)
}
}
If we run this server, and then make a request against it, we’ll see our log line output to stderr:
$ go run server.go
# Make a request with: curl localhost:8000/
ts=2020-03-21T18:30:58.8816186Z loc=server.go:62 status=0 method=GET path=/ duration=7.6µs
If we wanted to log more information - such as *Request.Host, a value from *Request.Context() (e.g. a trace ID), or specific response headers, we could easily do that by extending the call to logger.Log as needed in our own version of the middleware.
Summary
We were able to build a flexible, re-usable middleware component by:
- Satisfying Go’s existing
http.Handler interface, allowing our code to be loosely coupled from underlying framework choices
- Returning closures to inject our dependencies and avoid global (package-level) config
- Using composition - when we defined a wrapper around the
http.ResponseWriter interface - to override specific methods, as we did with our logging middleware.
Taking this, you can hopefully see how you might provide the basis for authentication middleware, or metrics middleware that counts status codes and response sizes.
And because we used http.Handler as our foundation, the middleware we author can be easily consumed by others!
Pretty good, huh?
Postscript: Logs vs Metrics vs Traces
It’s worth taking a moment to define what we mean by “logging”. Logging is about capturing (hopefully) structured event data, and logs are good for detailed investigation, but are large in volume and can be slow(er) to query. Metrics are directional (think: # of requests, login failures, etc) and good for monitoring trends, but don’t give you the full picture. Traces track the lifecycle of a request or query across systems.
Although this article talks about better logging for Go web services, a production application should consider all dimensions. I recommend reading Peter Bourgon’s post on Metrics, tracing & logging for a deeper dive on this topic.
•••
Admission Control (GitHub) is a micro-framework written in Go for building and deploying dynamic admission controllers for your Kubernetes clusters. It reduces the boilerplate needed to inspect, validate and/or reject the admission of objects to your cluster, allowing you to focus on writing the specific business logic you want to enforce.
The framework was born out of the need to cover a major gap with most managed Kubernetes providers: namely, that a LoadBalancer is public-by-default. As I started to prototype an admission controller that could validate-and-reject public load balancer Services, I realized that I was writing a lot of boilerplate in order to satisfy Kubernetes’ admission API and (importantly) stand up a reliable controller.
What is an Admission Controller?: When you deploy, update or otherwise change the state of a Kubernetes (k8s) cluster, your change needs to be validated by the control plane. By default, Kubernetes has a number of built-in “admission controllers” that validate and (in some cases) enforce resource quotas, service account automation, and other cluster-critical tasks. Usefully, Kubernetes also supports dynamic admission controllers: that is, admission controllers you can write yourself.
For example, you can write admission controllers for:
- Validating that specific annotations are present on all of your Services - such as a valid DNS hostname on your company domain.
- Rejecting
Ingress or Service objects that would create a public-facing load-balancer/VIP as part of a defense-in-depth approach for a private cluster.
- Mutating fields: resolving container image tags into hashes for security, or generating side-effects such as pushing state or status updates into another system.
The last example - a MutatingWebhookConfiguration - can be extremely powerful, but you should consider how mutating live objects might make troubleshooting more challenging down the road vs. rejecting admission outright.
Writing Your Own
Writing your own dynamic admission controller is fairly simple, and has three key parts:
- The admission controller itself: a service running somewhere (in-cluster or otherwise)
- An
admissioncontrol.AdmitFunc that performs the validation. An AdmitFunc has a http.Handler compatible wrapper that allows you to BYO Go webserver library.
- A
ValidatingWebhookConfiguration (or Mutating...) that defines what Kinds of objects are checked against the controller, what methods (create, update, etc) and how failure should be handled.
If you’re already familiar with Go, Kubernetes, and want to see the framework in action, here’s a simple example that requires any Service have a specific annotation (key, value).
Note that the README contains step-by-step instructions for creating, configuring and running an admission controller on your cluster, as well as sample configurations to help you get started.
// ServiceHasAnnotation is a simple validating AdmitFunc that inspects any kind:
// Service for a static annotation key & value. If the annotation does not
// match, or a non-Service object is sent to the AdmitFunc, admission will be
// rejected.
func ServiceHasAnnotation(requiredKey, requiredVal string) AdmitFunc {
// Return a function of type AdmitFunc
return func(admissionReview *admission.AdmissionReview) (*admission.AdmissionResponse, error) {
kind := admissionReview.Request.Kind.Kind
// Create an *admission.AdmissionResponse that denies by default.
resp := &admission.AdmissionResponse{
Allowed: false,
Result: &metav1.Status{},
}
// Create an object to deserialize our requests' object into.
// If we get a type we can't decode - we will reject admission.
// Our ValidatingWebhookConfiguration will be configured to only ...
svc := core.Service{}
deserializer := serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer()
if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &svc); err != nil {
return nil, err
}
for k, v := svc.ObjectMeta.Annotations {
if k == requiredKey && v == requiredVal {
// Set resp.Allowed to true before returning your AdmissionResponse
resp.Allowed = true
break
}
}
if !resp.Allowed {
return resp, xerrors.Errorf("submitted %s is missing annotation (%s: %s)",
kind, requiredKey, requiredVal)
}
return resp, nil
}
}
We can now use the AdmissionHandler wrapper to translate HTTP request & responses for us. In this example, we’re using gorilla/mux as our routing library, but since we satisfy the http.Handler type, you could use net/http as well.
You would deploy this as Service to your cluster: an admission controller is ultimately just a webserver that knows how to handle an AdmissionRequest and return an AdmissionResponse.
r := mux.NewRouter().StrictSlash(true)
admissions := r.PathPrefix("/admission-control").Subrouter()
admissions.Handle("/enforce-static-annotation", &admissioncontrol.AdmissionHandler{
AdmitFunc: admissioncontrol.ServiceHasAnnotation("k8s.example.com", "hello-world"),
Logger: logger,
}).Methods(http.MethodPost)
You can hopefully see how powerful this is already.
We can decode our request into a native Kubernetes object (or a custom resource), parse an object, and match on any field we want to in order to enforce our business logic. We could easily make this more dynamic by feeding the admission controller itself a ConfigMap of values we want it to check for, instead of hard-coding the values into the service itself.
Writing Our ValidatingWebhookConfiguration
A ValidatingWebhookConfiguration is what determines which admissions are sent to your webhook.
Using our example above, we’ll create a simple configuration that validates all Service objects deployed in any Namespace across our cluster with an enforce-annotations: "true" label.
apiVersion: v1
kind: Namespace
metadata:
# Create a namespace that we'll match on
name: enforce-annotations-example
labels:
enforce-annotations: "true"
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: enforce-static-annotations
webhooks:
- name: enforce-static-annotations.questionable.services
sideEffects: None
# "Equivalent" provides insurance against API version upgrades/changes - e.g.
# extensions/v1beta1 Ingress -> networking.k8s.io/v1beta1 Ingress
# matchPolicy: Equivalent
rules:
- apiGroups:
- "*"
apiVersions:
- "*"
operations:
- "CREATE"
- "UPDATE"
resources:
- "services"
namespaceSelector:
matchExpressions:
# Any Namespace with a label matching the below will have its
# annotations validated by this admission controller
- key: "enforce-annotations"
operator: In
values: ["true"]
failurePolicy: Fail
clientConfig:
service:
# This is the hostname our certificate needs in its Subject Alternative
# Name array - name.namespace.svc
# If the certificate does NOT have this name, TLS validation will fail.
name: admission-control-service # the name of the Service when deployed in-cluster
namespace: default
path: "/admission-control/enforce-static-annotation"
# This should be the CA certificate from your Kubernetes cluster
# Use the below to generate the certificate in a valid format:
# $ kubectl config view --raw --minify --flatten \
# -o jsonpath='{.clusters[].cluster.certificate-authority-data}'
caBundle: "<snip>"
# You can alternatively supply a URL to the service, as long as its reachable by the cluster.
# url: "https://admission-control-example.questionable.services/admission-control/enforce-pod-annotations""
A Service that would match this configuration and be successfully validated would look like the below:
apiVersion: v1
kind: Service
metadata:
name: public-service
namespace: enforce-annotations
annotations:
"k8s.example.com": "hello-world"
spec:
type: LoadBalancer
selector:
app: hello-app
ports:
- port: 8000
protocol: TCP
targetPort: 8080
Deploying a Service without the required annotation would return an error similar to the below:
Error from server: submitted Service is missing required annotation (k8s.example.com: hello-world)
… and reject admission. Because we also have UPDATE in our .rules.operations list, removing or otherwise modifying a previously-admitted Service would also be rejected if the annotation did not match.
Things to Watch Out For
One important thing worth noting is that a “Pod” is not always a “Pod” - if you want to enforce (for example) that the value of containers.image in any created Pod references a specific registry URL, you’ll need to write logic that inspects the PodTemplate of a Deployment, StatefulSet, DaemonSet and other types that can indirectly create a Pod.
There is not currently (as of Kubernetes v1.17) a way to reference a type regardless of how it is embedded in other objects: in order to combat this, default deny objects that you don’t have explicit handling for.
Other best practices:
- You should also scope admission controllers to namespaces using the
.webhooks.namespaceSelector field: this will allow you to automate which namespaces have certain admission controls applied. Applying controls to kube-system and other cluster-wide administrative namespaces can break your deployments.
- Make sure your admission controllers are reliable: running your admission controller as a
Deployment with its own replicas will prevent downtime from the controller being unavailable.
- Test, test, test. Run both unit tests and integration tests to make sure your AdmitFuncs are behaving as expected. The Kubernetes API surface is large, and there are often multiple versions of an object in play (v1beta1, v1, etc) for a given Kubernetes version. See the framework tests for an example of how to test your own AdmitFuncs.
Note: a project with a similar goal is Open Policy Agent, which requires you to write policies in Rego, a query language/DSL. This can be useful for simpler policies, but I would argue that once you get into more complex policy matching, the ability to use k8s packages, types and a Turing-complete language (Go) is long-term beneficial to a large team.
What’s Next?
Take a look at the README for Admission Control, including some of the built-in AdmitFuncs, for how more complex enforcement and object handling can be done.
You can also create an AdmissionServer to simplify the creation of the webhook server, including handling interrupt & termination signals cleanly, startup errors, and timeouts. Good server lifecycle management is important when running applications on top of Kubernetes, let alone ‘control plane’ services like an admission controller.
Contributions to the framework are also welcome. Releases are versioned, and adding to the existing library of built-in AdmitFuncs is an ongoing effort.
•••
Updated September 2020: Now incorporates the matrix functionality supported in CircleCI.
If you follow me on Twitter, you would have noticed I was looking to migrate the Gorilla Toolkit from TravisCI to CircleCI as our build-system-of-choice after they were bought out & fired a bunch of senior engineers. We’d been using TravisCI for a while, appreciated the simple config, but realized it was time to move on.
I also spent some time validating a few options (Semaphore, BuildKite, Cirrus) but landed on CircleCI for its popularity across open-source projects, relatively sane (if a little large) config API, and deep GitHub integration.
Requirements
I had two core requirements I needed to check off:
-
The build system should make it easy to build multiple Go versions from the same config: our packages are widely used by a range of different Go programmers, and have been around since the early Go releases. As a result, we work hard to support older Go versions (where possible) and use build tags to prevent newer Go APIs from getting in the way of that.
-
Figuring out what went wrong should be easy: a sane UI, clear build/error logs, and deep GitHub PR integration so that a contributor can be empowered to debug their own failing builds. Overall build performance falls into this too: faster builds make for a faster feedback loop, so a contributor is more inclined to fix it now.
The Config
Without further ado, here’s what the current (September, 2020) .circleci/config.yml looks like for gorilla/mux - with a ton of comments to step you through it.
version: 2.1
jobs:
"test":
parameters:
version:
type: string
default: "latest"
golint:
type: boolean
default: true
modules:
type: boolean
default: true
goproxy:
type: string
default: ""
docker:
- image: "circleci/golang:<< parameters.version >>"
working_directory: /go/src/github.com/gorilla/mux
environment:
GO111MODULE: "on"
GOPROXY: "<< parameters.goproxy >>"
steps:
- checkout
- run:
name: "Print the Go version"
command: >
go version
- run:
name: "Fetch dependencies"
command: >
if [[ << parameters.modules >> = true ]]; then
go mod download
export GO111MODULE=on
else
go get -v ./...
fi
# Only run gofmt, vet & lint against the latest Go version
- run:
name: "Run golint"
command: >
if [ << parameters.version >> = "latest" ] && [ << parameters.golint >> = true ]; then
go get -u golang.org/x/lint/golint
golint ./...
fi
- run:
name: "Run gofmt"
command: >
if [[ << parameters.version >> = "latest" ]]; then
diff -u <(echo -n) <(gofmt -d -e .)
fi
- run:
name: "Run go vet"
command: >
if [[ << parameters.version >> = "latest" ]]; then
go vet -v ./...
fi
- run:
name: "Run go test (+ race detector)"
command: >
go test -v -race ./...
workflows:
tests:
jobs:
- test:
matrix:
parameters:
version: ["latest", "1.15", "1.14", "1.13", "1.12", "1.11"]
Updated: September 2020:
We now use the matrix parameter to define a list of parameters. Our jobs are then run for each version we define, automtically.
In our case, since we only want to run golint and other tools on the latest version, we check << parameters.version >> = "latest" before running those build steps.
Pretty straightforward, huh? We define a base job configuration, create a reference for it at &test, and then refer to that reference with <<: *test and just override the bits we need to (Docker image URL, env vars) without having to repeat ourselves.
By default, the jobs in our workflows.build list run in parallel, so we don’t need to do anything special there. A workflow with sequential build steps can set a requires value to indicate the jobs that must run before it (docs).
Note: If you’re interested in what the previous TravisCI config looked like vs. the new CircleCI config, see here.
Go Modules?
Updated: September 2020
Works out of the box!
If you’re also vendoring dependencies with go mod vendor, then you’ll want to make sure you pass the -mod=vendor flag to go test or go build as per the Module docs.
Other Tips
A few things I discovered along the way:
- Building from forks is not enabled by default - e.g. when a contributor (normally) submits a PR from their fork. You’ll need to turn it on explicitly.
- Enable GitHub Checks to get deeper GitHub integration and make it easier to see build status from within the Pull Request UI itself (example).
- Updating the CI config on 10+ projects is not fun, and so I wrote a quick Go program that templates the
config.yml and generates it for the given list of repos.
In the end, it took a couple of days to craft a decent CircleCI config (see: large API surface), but thankfully the CircleCI folks were pretty helpful on that front. I’m definitely happy with the move away from Travis, and hopefully our contributors are too!
•••
The Coral Dev Board is a TPU-enabled development board for testing out machine learning models with a requirement for near-real-time inference. For instance, image classification or object detection on video feeds, where a CPU would struggle to keep up.
However, the dev board’s setup instructions only document a native Linux process, but it’s entirely possible to flash the boot image via native Windows (without WSL): here’s how!
Pre-requisites
You’ll need to install a few things: this is mostly a process of clicking “next” a few times (the USB drivers) and unzipping a folder (the Android Platform Tools).
You should also be moderately familiar with serial consoles & have read through the Coral’s setup instructions to familiarize yourself with the process.
Note: It’s important to make sure you’re using a data-capable USB-C cable when connecting to the USB-C data port. Like many things USB-C / USB 3.x, this can be non-obvious at first. You’ll know when the Device Manager shows a “⚠ USB Download Gadget” in the Device Manager. If you use a power-only cable, nothing will show up and it’ll seem as if the OS isn’t seeing the device.
Connecting to the Serial Console
Mostly identical to the Coral setup instructions:
- Connect to the dev board’s micro-USB port, and identify the COM port the device is attached to in the Device Manager by looking under “Ports (COM & LPT)” for the “CP2105 USB to UART (Standard)” device. In my case, it was COM3.
- Power on the board by connecting the USB-C power cable to the power port (furthest from the HDMI port).
- Open PuTTY, select “Serial” as the connection option, set the COM port to the one you identified above, and the data rate to 115200bps. For confirmation, the serial comms settings should be at 8 data bits, no parity bits, 1 stop bit and XON/XOFF flow control.
The serial port on the dev board accepts other settings, but I’m documenting an explicit list for those who don’t have a background in serial comms.
You should now be at the dev board’s uboot prompt, and ready to flash the bootloader & disk image. If not, check that the board is powered on, that the COM port is correct, and that the Device Manager lists the device.
Flashing the Board
Connect the USB-C data cable to the dev board, and the other end to your PC.
In the Device Manager, you’ll see a “USB Download Gadget” appear with a warning symbol. Right click, choose “Update Driver”, select “Browse my computer for driver software” and then “Let me pick from a list of available drivers from my computer”. In the driver browser, choose “WinUsb Device” from the left side, and “ADB Device” (Android Debugger) from the right. Click “Next” and accept the warning. The Device Manager will refresh, and show the device under “Universal Serial Bus devices”.
To confirm it’s configured correctly and visible to the OS, head back to your command prompt and enter:
λ fastboot devices
122041d6ef944da7 fastboot
If you don’t see anything, confirm the device is still showing in the Device Manager, and that you have the latest version of fastboot from the Android Platform Tools (linked above).
From here, you’ll need to download and unzip the bootloader image and the disk image (identical to the official instructions), and confirm you see the contents below:
λ curl -O https://dl.google.com/aiyprojects/mendel/enterprise/mendel-enterprise-beaker-22.zip
λ unzip mendel-enterprise-beaker-22.zip
λ cd mendel-enterprise-beaker-22
λ ls
boot_arm64.img partition-table-16gb.img partition-table-8gb.img rootfs_arm64.img
flash.sh* partition-table-64gb.img recovery.img u-boot.imx
Unfortunately, the flash.sh script is a Bash script, which won’t work for us: but we can easily replicate what it does:
λ tail -n 15 flash.sh
fi
# Flash bootloader
${FASTBOOT_CMD} flash bootloader0 ${PRODUCT_OUT}/u-boot.imx
${FASTBOOT_CMD} reboot-bootloader
# Flash partition table
${FASTBOOT_CMD} flash gpt ${PRODUCT_OUT}/${PART_IMAGE}
${FASTBOOT_CMD} reboot-bootloader
# Flash filesystems
${FASTBOOT_CMD} erase misc
${FASTBOOT_CMD} flash boot ${PRODUCT_OUT}/boot_${USERSPACE_ARCH}.img
${FASTBOOT_CMD} flash rootfs ${PRODUCT_OUT}/rootfs_${USERSPACE_ARCH}.img
${FASTBOOT_CMD} reboot
Where we see “FASTBOOT_CMD” we simply run fastboot - and where we see USERSPACE_ARCH we only have one choice for the dev board: arm64. We can work with this.
In the serial console (e.g. in PuTTY), put the dev board into fastboot mode:
Then, in the command prompt and from within the mendel-enterprise-beaker-22 directory, invoke the following commands. You should leave the serial console connected: you’ll see the progress of each step.
fastboot flash bootloader0 u-boot.imx
fastboot reboot-bootloader
fastboot flash gpt partition-table-8gb.img
fastboot reboot-bootloader
fastboot erase misc
fastboot flash boot boot_arm64.img
fastboot flash rootfs rootfs_arm64.img
fastboot reboot
When the device reboots, you’ll get a more familiar Linux login prompt in the serial console! Enter mendel (username) and mendel (password) to log in, and then follow the steps within the official documentation to set up network connectivity! You’ll then be able to log into the board remotely via SSH, and will only need to connect it to power unless you want to flash it again.
Beyond that: enjoy experimenting & building things on your Coral Dev Board! And if you run into issues, or find something unclear in these instructions, you can reach me on Twitter at @elithrar.
•••
Update (June 2019): kubectl v1.15 now provides a rollout restart sub-command that allows you to restart Pods in a Deployment - taking into account your surge/unavailability config - and thus have them pick up changes to a referenced ConfigMap, Secret or similar. It’s worth noting that you can use this with clusters older than v1.15, as it’s implemented in the client.
Example usage: kubectl rollout restart deploy/admission-control to restart a specific deployment. Easy as that!
One initially non-obvious thing to me about Kubernetes was that changing a ConfigMap (a set of configuration values) is not detected as a change to Deployments (how a Pod, or set of Pods, should be deployed onto the cluster) or Pods that reference that configuration. That expectation can result in unintentionally stale configuration persisting until a change to the Pod spec. This could include freshly created Pods due to an autoscaling event, or even restarts after a crash, resulting in misconfiguration and unexpected behaviour across the cluster.
Note: This doesn’t impact ConfigMaps mounted as volumes, which are periodically synced by the
kubelet running on each node.
Updating the ConfigMap and running kubectl apply -f deployment.yaml results in a no-op, which makes sense if you consider the impacts of an unintended config change and rollout in a larger deployment.
But, there are certainly cases where we want to:
- Update a ConfigMap
- Have our Deployment reference that specific ConfigMap version (in a version-control & CI friendly way)
- Rollout a new revision of our Deployment
So how can we accomplish that? It turns it out to be fairly straightforward, but let’s step through an example.
Example
Our ConfigMap, applied to our Kubernetes cluster:
➜ less demo-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: demo-config
namespace: default
data:
READ_TIMEOUT_SECONDS: "15"
WRITE_TIMEOUT_SECONDS: "15"
NAME: "elithrar"
➜ kubectl apply -f demo-config.yaml
configmap/demo-config created
And here’s our Deployment before we’ve referenced this version of our ConfigMap - notice the spec.template.metadata.annotations.configHash key we’ve added. It’s important to note that modifying a top-level Deployment’s metadata.annotations value is not sufficient: a Deployment will only re-create our Pods when the underlying template.spec (Pod spec) changes.
This is how we’ll couple the Deployment with our ConfigMap, triggering a change in our Deployment only when our ConfigMap actually changes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
labels:
app: config-demo-app
spec:
replicas: 3
selector:
matchLabels:
app: config-demo-app
template:
metadata:
labels:
app: config-demo-app
annotations:
# The field we'll use to couple our ConfigMap and Deployment
configHash: ""
spec:
containers:
- name: config-demo-app
image: gcr.io/optimum-rock-145719/config-demo-app
ports:
- containerPort: 80
envFrom:
# The ConfigMap we want to use
- configMapRef:
name: demo-config
# Extra-curricular: We can make the hash of our ConfigMap available at a
# (e.g.) debug endpoint via a fieldRef
env:
- name: CONFIG_HASH
valueFrom:
fieldRef:
fieldPath: spec.template.metadata.annotations.configHash
With these two pieces in mind, let’s create a SHA-256 hash of our ConfigMap. Because this hash is deterministic (the same input == same output), the hash only changes when we change our configuration: making this a step we can unconditionally run as part of our deployment (CI/CD) pipeline into our Kubernetes cluster.
Note that I’m using yq (a CLI tool for YAML docs, like jq is to JSON) to modify our Deployment YAML at a specific path.
➜ yq w demo-deployment.yaml spec.template.metadata.annotations.configHash \
> $(kubectl get cm/demo-config -oyaml | sha256sum)
...
spec:
...
template:
metadata:
...
annotations:
configHash: 4431f6d28fdf60c8140d28c42cde331a76269ac7a0e6af01d0de0fa8392c1145
We can now re-deploy our Deployment, and because our spec.template changed, Kubernetes will detect it as a change and re-create our Pods.
As a bonus, if we want to make a shortcut for this during development/local iteration, we can wrap this flow in a useful shell function:
# Invoke as hash-deploy-config deployment.yaml configHash myConfigMap
hash-deploy-config() {
yq w $1 spec.template.metadata.annotations.$2 \
$(kubectl get cm/$3 -oyaml | sha256sum)
}