Getting the different components in your systems talking nicely to one another relies on a rather mundane but crucial detail: a good data structure in the message payloads. This article will pick out some of the best advice we have for getting your Apache Kafka data payloads well designed from the very beginning of your project.

Use all the features of Apache Kafka records

The events that we stream with Kafka can support headers as well as keys and the main body of the payload. The most scalable systems use all these features appropriately.


There are some excellent articles around about what a Developer Advocate does, but have you ever wondered WHY we do what we do? I tried to articulate this when chatting with another developer lately, and I’m still thinking about it! So I thought I’d write it down to share a little more widely why I love this strange, misshapen career as much as I do.

It’s Alarmingly Varied

Sometimes when I’m doing career development talks or outreach (I’m a STEM Ambassador ) people will ask “Can you describe a typical day?” and the answer is … not really! Advocacy requires a lot of…


Perhaps you recognise Aiven as the multicloud platform for all your open source data needs, the one with the lovely web interface. And perhaps, if you didn’t already, you recognise us that way now :) But Aiven is more than just the Aiven Console, we also have an API for all your integration/automation needs. In this post, we’ll take a look at using the API and what you can do with it.

Presenting — the Aiven API

Aiven offers an HTTP API with token authentication and JSON-formatted data. A great place to start is the API reference documentation which gives you the details of all…


I’ve been using some fun open data sets recently but I’m more familiar with databases than with traditional data science formats (this one is CSV). Importing CSV into Postgres is fairly well-established, but you need a table structure to import into … and I was looking at a CSV with 50 columns (for my post about exoplanets on the Aiven blog). Enter this neat utility: ddlgenerator.

I used it for a CSV file and put the data into PostgreSQL but on closer inspection I realised that this tool can ingest a great selection of different formats, including JSON, YAML and…


The libraries I picked for this post are pgx which is a nice PostgreSQL-specific alternative to database/sql and Squirrel because I love the name (it also has some quite cool features).

Start with a database

It would make sense to already have a PostgreSQL database handy, ideally with some data in it. If you don’t already have an Aiven account, then sign up and start a PostgreSQL service. If you already have some data to use, great! I’m using some open data from the Kepler space mission, you can follow along with the recent blog post to set this up yourself.

In the Aiven…


PostgreSQL is and remains one of our most popular and growing storage platforms; other storage technologies come and go but modern Postgres is a solid choice for so many applications. When you spin up your first Aiven PostgreSQL, you’ll want to take some time to play with the features … but there’s a problem. Your new shiny database is empty.

Finding and using some open datasets is a great way to fill this gap, and one option is go try the Kaggle platform. It’s a place to find open data, advice about data science, and some competitions you can participate…


Choosing Between Kafka and RabbitMQ

There are many roads that can lead to the moment you decide you need a queue. Queues are an excellent way to loosely couple many different components and allow them to exchange data without detailed knowledge about one another. Using a queue is also an excellent way to distribute work between multiple nodes to perform asynchronous tasks.

Queues come in different flavours and success is morely likely when you can use the queue that best fits the shape of your use case. …


I am the proud owner of a Raspberry Pi Pico which is a dinky microcontroller from the lovely people at Raspberry Pi. I also have the RGB Pico Keypad Base, because my husband was buying something else from Pimoroni (3D printer filament, more on that another day) and we love them and … here we are :)

I’ve tried to write down and link to the stuff I found useful along the way with this project, in case it helps you too. Well, no, really so I can come back to it next time I’m using one of these!

Hardware Setup

The…


I enjoy a productivity hack as much as the next person, and I’m often asked about my setup. My background is in engineering (I hold an MEng in Electronic Engineering from York) and recognise the legacy of the lab book in much of what I do. If you’re new to the lab book concept, check out this excellent article on how to maintain one properly. My practice breaks a LOT of the lab book rules and is also heavily influenced by the Bullet Journal trend (if you have time to spare, the instagram #bulletjournal hashtag gives a prettier introduction), which…


I work in Developer Relations, or “DevRel”, and 2020 was a strange year with travel restrictions that meant that the entire industry did everything online, or digitally. I see this as a very positive progression of the way we work as DevRel professionals, and I understand that might be controversial, but I’d like to back up my position so hear me out (and then let me know what you think!).

Digital is an obvious way to do DevRel. Where are the developers? On the internet, on StackOverflow and reading dev.to, chatting on Discord and Twitter. …

Lorna Mitchell

Polyglot programmer, technology addict, open source fanatic and incurable blogger (see http://lornajane.net)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store