Skip to content

Hacking on User Space Networking

As most of you know, I’ve been working at Igalia close to two years now. If you asked me what I do I’d probably say I hack on “networky stuff”, which is a bit opaque. I thought I’d take the time to tell you about the kind of things I work on.

A bit of background

Consider a data centre full of big expensive boxes that have some high speed network cards performing some specific function (e.g. firewalls, VoIP, etc.). These are all well and good but they cost a fortune and are just what they look like: black boxes. If you’d like to modify the functionality beyond what the vendor provides, you’d probably be out of luck. If the market changes and you need to provide other functionality, you’re stuck having to buy yet another big expensive box that only performs the one function, just like the old one.

You might be thinking, just buy regular server hardware and write some program to deal with those packets. Easy, right? Well… not so fast. Your run of the mill linux application written against the kernel is just too slow as it’s not designed to do these zippy networky things. It’s designed to be a general purpose operating system.

For those curious about the kernel’s speeds Cloudflare has a great post demonstrating this.

Enter user space networking

Instead of just relying on the kernel, you can tell linux to leave the network card alone. Linux allows you to poke around directly over PCI, so you can write a driver for the card outside the kernel. The driver will be part of your program allowing you to skip all the kernel faffing and get down to crunching those packets.

Great, right? Job done? Well, kind of…

I’m going to mainly talk about snabb for the rest of this blog post. It’s not the only game in town but I prefer it and it’s what I have the most experience with. Snabb is a framework which implements a bunch of drivers and other neat tools in Luajit (a lightning fast implementation of Lua). Imagine you’re a program using snabb, having all these packets coming at you at breakneck speed from your 10Gb card. You have about 67.2 nanoseconds per packet (my thanks to this helpful person for doing the work for me) to deal with it and move on to the next packet. That’s not much time, right? I’ll try to put it into context; here are some things you may wish to do whilst dealing with the packet:

L2 cache hit  4.3ns
L3 cache hit 7.9ns
Cache miss 32ns
Linux syscall[1] 87.77 ns or 41.85 ns

[1] – If CONFIG_AUDITSYSCALL is enabled it’s the higher one, if not it’s the lower one.

You can imagine that if you’re going full pelt, you don’t really have time to do much at all. Two cache misses and that’s your budget, one syscall and you’ve blown way past the budget. The solution to all this is to deal with a bunch of packets at once. Doing this allows you to go to RAM or whatever once for the whole set rather than doing it for each packet.

Dags att köra Snabb(t)

My colleague Diego has recently written a great set of blogs which are more in-depth than this, so check those out if you’re interested. Snabb works by having a graph of “apps”; each app is usually responsible for doing one thing, for example: being a network driver, reading/writing pcap files, filtering, and so forth. The apps are then connected together to form a graph.

Lets see a real example of a very simple, but potentially useful app. We’re going to take packets in from an Intel 82599 10 Gigabit card and throw them into a pcap file (packet capture file). We’ll take the “output” of the driver app and connect it to the input of the pcap writing app. Here’s the code:

module(..., package.seeall)

local pcap = require("apps.pcap.pcap")
local Intel82599 = require("apps.intel_mp.intel_mp").Intel82599

function run(parameters)
   -- Lua arrays are indexed from 1 not 0.
   local pci_address = parameters[1]
   local output_pcap = parameters[2]

   -- Configure the "apps" we're going to use in the graph.
   -- The "config" object is injected in when we're loaded by snabb.
   local c =, "nic", Intel82599, {pciaddr = pci_address}), "pcap", pcap.PcapWriter, output_pcap)

   -- Link up the apps into a graph., "nic.output -> pcap.input")

   -- Tell the snabb engine our configuration we've just made.

   -- Lets start the apps for 10 seconds!
   engine.main({duration=10, report = {showlinks=true}})

Awesome! 🙏 

Just to give you a taste of what is included in snabb out of the box (by no means an exhaustive list):

  • pcap reading / writing
  • PF filtering (like the openBSD firewall, more about speed and implementation from my colleagues Asumu and Andy)
  • Rate limiting
  • ARP and NDP
  • YANG (network configuration language for specifying, validating and creating parsers for configuration files)

…and the list continues, it also is pretty simple to create your own. Just define a Lua object which has a “pull” (input) and “push” (output) methods and do something with the packet in the middle. I mainly work on the LwAFTR which provides a solution to IPv4 exhaustion, this is done by wrapping the IPv4 packet up in a IPv6 at the user and unwrapping it at the ISP so they can give users a port range of an IPv4 address rather than the whole thing.

Published inIgaliaNetworking