How Devices Create and Consume Packets: The Model Series

Before we dig into devices that create and consume packets, let’s take a moment to remember how we got here. This series started with some observations about the OSI model and how I don’t find it helpful when working on real networks. The problem posed by observing “the OSI model isn’t all that useful” is: “What kind of model is helpful for understanding, designing, and troubleshooting modern networks?” In the second part of this series, I considered the “four things model,” an expansion of the RINA model.

But a data transmission model isn’t enough. We need a model for packet-forwarding devices, considered in the previous post in this series, and a model for packet creation and consumption.

The problem we are solving: a model of the modern network. The solution: a set of interconnected models for creating, carrying, switching, and consuming data in the form of packets.

Host vs. Middlebox

How does a host differ from a middlebox? Hosts are designed to run applications that create and consume packets rather than quickly forwarding traffic through the network. The primary difference is that the source and destination are not two different ports on the same device but rather a virtual interface representing an application and a physical interface. Hosts and middleboxes share many of the same components, but traffic flows differently between these two kinds of devices.

The figure below illustrates.

 

Suppose an application needs to send some data to an application running on some other host someplace “out there on the network.” The application will dump this data into a socket and assume it will just get there without any further fuss or muss.

Once data is dumped into the socket, some application running locally on the host needs to pick the data up, marshal it (put it into packets, encapsulating at least a TCP, UDP, or QUIC header and an IP header), and then push the packet down into a queue to be transmitted. This application is often called a protocol stack or protocol.

The protocol application uses information from local routing and translation tables to build the transport and network-level encapsulations. These tables are primarily built using DNS, ARP, and IPv6 Neighbor Discovery. There is a local routing table, even if it often contains just a default route.

Queueing and Quality of Service (QoS) occur between the protocol stack application and the device driver, just like between the Forwarding Engine and the transmit ring on a middlebox. The transmit ring plays the same role in a host as it does in a middlebox; the PHY chip on the network interface card pulls bits from packet buffers on the transmit ring and serializes them onto a physical medium.

The inbound path travels from the same physical interface (PHY chip) through the transmit ring to the device driver and then to an input queue. Packets are placed into an input queue for sorting and delivery to individual applications, unlike a middlebox, because they are consumed rather than forwarded.

The diagram also has two grey boxes, one labeled user space and the other labeled kernel. What are these about?

A long time ago, in a universe far away, operating systems only had one memory pool. Every application, every driver, and every device was allocated memory from this single pool. Giving every application memory from a single memory pool meant that one application could store something in memory and stop running. Then, some other application could access the stored item directly—everything in the system using a single shared memory pool. The problem with this kind of system is that a mistake or security hole in one application can crash another or the entire system.

Modern applications assign each application its pool of memory, which any other application cannot access (it’s a bit more complex than this, of course, but this is a model rather than a detailed explanation of how things work). These memory pools, or the combination of all these memory pools, are called user space.

A group of applications provides virtual copies of physical devices to each application, maps memory to each application, schedules when applications run, etc. These are called kernel applications, and they run within a single kernel memory space.

An application running in the kernel cannot access a packet stored in memory created by an application running in user space. An application running in user space cannot access a packet stored in memory owned by the kernel, such as a packet sitting in the receive ring.

Packets must be copied from one memory space to another as they traverse the system, so:

• A packet received from the network interface must be deserialized, copied into the receive ring, and copied to the appropriate input queue by copying it from kernel to user space memory so an application can process the received data.
• A packet transmitted by an application must be copied into the correct output queue, where QoS will be applied. The packet must be copied from user to kernel memory before it can be placed on the transmit ring, where the PHY chip can serialize the data onto the physical network medium.

This basic model of a device that creates and consumes packets describes, at a high level, the pieces and processing of hosts, mobile devices, and just about every other device connected to a network.

In the next and last part of this series, we will combine all these models to trace a packet through a network.