dbt announced “model contracts” in the recent v1.5 release. This looks like a great feature for dbt, but reminded me that I’ve been using contract testing with dbt for a couple of years now, inspired by Pact consumer-driven contracts, but never talked about it. There are some differences, for example: dbt’s new feature is very dbt-centric, the approach I’ve used isn’t - dbt certainly helps, but it isn’t necessary. There’s a GitHub repo to follow along with.[Read More]
With experience in software developnent, data engineering and machine learning, I specialise in data-intensive problems and decentralised data engineering at scale. My experience extends to leading teams, technical architecture and product development. Find out more about my experience and publications in my portfolio.
Contact me to see how I can help at firstname.lastname@example.org.
Checking your Dependencies
Following high-profile incidents like the 2017 Equifax Breach, checking your dependencies for vulnerabilities is a common practice today. We can use great tools like OWASP Dependency Check, Trivy and Snyk in our builds to raise the alarm when vulnerabilities are found.
The question that I find comes up isn’t whether we should check dependencies - but when?[Read More]
So this annoying and trivial little problem catches me out every so often. I am always misled by the error message! You’ll see what I mean shortly. For context, it usually happens when I’m working in Docker containers on a build.[Read More]
Helm Charts for Argo Workflows
Argo is a lightweight, Kubernetes-native workflow solution. Workflows are implemented as Kubernetes manifests, so Helm is a natural choice for packaging them.
Helm also supports templating values which can be really helpful - but that’s where we run into a problem. Helm uses mustache-style string interpolation, and so does Argo.[Read More]