Profile photo Paul Brabban, Lead Consultant at Equal Experts

With experience in software developnent, data engineering and machine learning, I specialise in data-intensive problems and decentralised data engineering at scale. My experience extends to leading teams, technical architecture and product development. Find out more about my experience and publications in my portfolio.

Contact me to see how I can help at paul@tempered.works.

Scala Types in Scio Pipelines

 Posted on July 29, 2018  |  4 minutes  |  Paul Brabban

Data pipelines in Apache Beam have a distinctly functional flavour, whichever language you use. That’s because they can be distributed over a cluster of machines, so careful management of state and side-effects is important. Spotify’s Scio is an excellent Scala API for Beam. Scala’s functional ideas help to cut out much of the boilerplate present in the native Java API. Scio makes good use of Scala’s tuple types, in particular pairs (x, y). [Read More]
scala  apache-beam  scio