{"id":1952,"date":"2016-01-28T00:20:34","date_gmt":"2016-01-28T08:20:34","guid":{"rendered":"https:\/\/www.privateinternetaccess.com\/blog\/?p=1952"},"modified":"2024-02-01T01:21:48","modified_gmt":"2024-02-01T09:21:48","slug":"linux-networking-stack-from-the-ground-up-part-3","status":"publish","type":"post","link":"https:\/\/www.privateinternetaccess.com\/blog\/linux-networking-stack-from-the-ground-up-part-3\/","title":{"rendered":"Linux networking stack from the ground up, part 3"},"content":{"rendered":"\n

part 1<\/a> | part 2<\/a> | part 3<\/a> | part 4<\/a> | part 5<\/a><\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n

This post will pick up where part 2 left off, beginning by describing a packet arriving, examining softirqs, and examining how the e1000e<\/code> driver passes packets up the network stack.<\/p>\n\n\n\n

A packet arrives!<\/h2>\n\n\n\n

So, at long last a packet arrives from the network. Assuming, the rx ring buffer has enough space, the packet is written into the ring buffer via DMA and the device raises the interrupt that is assigned to it (or in the case of MSI-X, the IRQ tied to the rx queue the packet arrived on).<\/p>\n\n\n\n

You can find statistics about hardware interrupts by checking the \/proc\/interrupts<\/code> file.<\/p>\n\n\n\n

In general the interrupt handler that runs when an interrupt is raised should try to defer as much processing as possible to happen outside the interrupt context. This is crucial because while an interrupt is being processed, other interrupts are blocked.<\/p>\n\n\n\n

If we examine the e1000_intr_msi<\/code> function in e1000e, we can see after some device specific code and workarounds for hardware bugs that the interrupt handler signals NAPI (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:1777<\/a>):<\/p>\n\n\n\n

if (napi_schedule_prep(&adapter->napi)) {\n  adapter->total_tx_bytes = 0;\n  adapter->total_tx_packets = 0;\n  adapter->total_rx_bytes = 0;\n  adapter->total_rx_packets = 0;\n  __napi_schedule(&adapter->napi);\n}<\/pre>\n\n\n\n
This code is checking if NAPI is already running and if not, statistics structures are reset and NAPI is scheduled to run to process the packets.<\/p>\n\n\n\n
At a high level, NAPI is scheduled to run from the hardware interrupt handler, but the NAPI code which does the packet processing is run outside of the hardware interrupt context. This is accomplished with softirqs, which will be detailed next.<\/p>\n\n\n\n
The __napi_schedule<\/code> function:<\/p>\n\n\n\n
\/**\n * __napi_schedule - schedule for receive\n * @n: entry to schedule\n *\n * The entry's receive function will be scheduled to run\n *\/\nvoid __napi_schedule(struct napi_struct *n)\n{\n  unsigned long flags;\n\n  local_irq_save(flags);\n  ____napi_schedule(&__get_cpu_var(softnet_data), n);\n  local_irq_restore(flags);\n}\nEXPORT_SYMBOL(__napi_schedule);<\/pre>\n\n\n\n
Schedules the NAPI poll function to be run. It does this by obtaining the current CPU\u2019s softnet_data<\/code> structure and passing that and the driver provided napi_struct<\/code> to ____napi_schedule<\/code>:<\/p>\n\n\n\n
\/* Called with irq disabled *\/                                                            \nstatic inline void ____napi_schedule(struct softnet_data *sd,                             \n                                     struct napi_struct *napi)                            \n{                                                                                         \n  list_add_tail(&napi->poll_list, &sd->poll_list);                                  \n  __raise_softirq_irqoff(NET_RX_SOFTIRQ);                                           \n}<\/pre>\n\n\n\n
Which adds the driver provided NAPI poll structure to the softnet_data<\/code> list for the current CPU.<\/p>\n\n\n\n
Next, the function calls __raise_softirq_irqoff<\/code> (from kernel\/softirq.c<\/code>) which contains this code:<\/p>\n\n\n\n
struct task_struct *tsk = __this_cpu_read(ksoftirqd);                             \n                                                                       \nif (tsk && tsk->state != TASK_RUNNING)                                            \n  wake_up_process(tsk);<\/pre>\n\n\n\n
which brings up an important point: the hardware interrupt handler wakes up the NAPI softirq process on the same CPU as the hardware interrupt handler.<\/p>\n\n\n\n
softirq<\/h2>\n\n\n\n
softirq is one mechanism for executing code outside of the hardware interrupt handler context. As mentioned above, this is important because only minimal work should be done in a hardware interrupt handler; the heavy-lifting should be left for later processing.<\/p>\n\n\n\n
The softirq system is a series of kernel threads, one per CPU, that run handler functions that have been registered for different softirqs.<\/p>\n\n\n\n
The softirq threads are started early in the kernel initialization process (`kernel\/softirq.c`:754):<\/p>\n\n\n\n
static __init int spawn_ksoftirqd(void)                                                   \n{       \n        register_cpu_notifier(&cpu_nfb);                                                  \n        \n        BUG_ON(smpboot_register_percpu_thread(&softirq_threads));                         \n        \n        return 0;                                                                         \n}\nearly_initcall(spawn_ksoftirqd);<\/pre>\n\n\n\n
The softirq_threads<\/code> structure exports a few fields, but the two important ones are thread_should_run<\/code> and thread_fn<\/code> both of which are called from kernel\/smpboot.c<\/code>.<\/p>\n\n\n\n
The thread_should_run<\/code> function checks for any pending softirqs and if there are one or more, the code in kernel\/smpboot.c<\/code> calls thread_fn<\/code>, which for the softirq system happens to be run_ksoftirqd<\/code>.<\/p>\n\n\n\n
The run_ksoftirqd<\/code> function runs the registered handler function for each softirq that is pending and increments stats that are found in \/proc\/softirqs<\/code>.<\/p>\n\n\n\n
NAPI and softirq<\/h3>\n\n\n\n
Recall from earlier we saw that the device driver calls __napi_schedule<\/code> which eventually calls __raise_softirq_irqoff(NET_RX_SOFTIRQ);<\/code>.<\/p>\n\n\n\n
This __raise_softirq_irqoff<\/code> function marks the NET_RX_SOFTIRQ<\/code> softirq as pending and wakes up the softirq thread on the current CPU to execute the NET_RX_SOFTIRQ<\/code> handler.<\/p>\n\n\n\n
NET_RX_SOFTIRQ and NET_TX_SOFTIRQ softirq handlers<\/h3>\n\n\n\n
Early in the initialization code of the networking subsystem (net\/core\/dev.c:7114<\/a>) we find the following code:<\/p>\n\n\n\n
open_softirq(NET_TX_SOFTIRQ, net_tx_action);\nopen_softirq(NET_RX_SOFTIRQ, net_rx_action);<\/pre>\n\n\n\n
These function calls register the functions net_tx_action<\/code> and net_rx_action<\/code> as softirq handlers that will be run when NET_TX_SOFTIRQ<\/code> and NET_RX_SOFTIRQ<\/code> softirqs are pending.<\/p>\n\n\n\n
rx packet processing begins<\/h2>\n\n\n\n
Once the softirq code determines that a softirq is pending, should be processed, and invokes the net_rx_action<\/code> function registered to the NET_RX_SOFTIRQ<\/code>, packet processing begins.<\/p>\n\n\n\n
net_rx_action<\/code> processing loop<\/h3>\n\n\n\n
net_rx_action<\/code> begins the processing of packets from the memory the packets were DMA\u2019d into by the device driver.<\/p>\n\n\n\n
The function iterates through the list of NAPI structures that are queued for the current CPU, dequeuing each structure, one at a time and operating on it.<\/p>\n\n\n\n
The processing loop bounds the amount of work and execution time that can be consumed by poll functions. It does this by keeping track of a work \u201cbudget\u201d (which can be adjusted) and checking the elapsed time (net\/core\/dev.c:4366<\/a>):<\/p>\n\n\n\n
\/* If softirq window is exhuasted then punt.\n * Allow this to run for 2 jiffies since which will allow\n * an average latency of 1.5\/HZ.\n *\/\nif (unlikely(budget <= 0 || time_after_eq(jiffies, time_limit)))\n  goto softnet_break;<\/pre>\n\n\n\n
NAPI also uses a \u201cweight\u201d to prevent individual calls to poll<\/code> from consuming the entire CPU and never terminating. The weight is set on the call to netif_napi_add<\/code> in the device driver initialization; recall that this was hardcoded to 64 in the driver.<\/p>\n\n\n\n
The weight is passed into the poll<\/code> function from the driver that is registered to NAPI. This amount dictates the maximum work poll<\/code> can do before it should return. In this case, it can process up to 64 packets.<\/p>\n\n\n\n
The poll<\/code> function returns the number of packets processed, which may be any value less than or equal to the weight. This work done is then subtracted from the budget:<\/p>\n\n\n\n
weight = n->weight;\n\n\/* This NAPI_STATE_SCHED test is for avoiding a race\n * with netpoll's poll_napi().  Only the entity which\n * obtains the lock and sees NAPI_STATE_SCHED set will\n * actually make the ->poll() call.  Therefore we avoid\n * accidentally calling ->poll() when NAPI is not \n *\/\nwork = 0;\nif (test_bit(NAPI_STATE_SCHED, &n->state)) {\n        work = n->poll(n, weight);\n        trace_napi_poll(n);\n}\n\nWARN_ON_ONCE(work > weight);\n\nbudget -= work;\n\nlocal_irq_disable();<\/pre>\n\n\n\n
If the work done was equal to the weight, the NAPI structure is moved to the end of the queue and will be examined again. The loop then begins anew, with the budget and time check.<\/p>\n\n\n\n
Thus, you can adjust the number of packets processed during the NAPI poll loop by setting the netdev_budget sysctl:<\/p>\n\n\n\n
sysctl net.core.netdev_budget=600<\/pre>\n\n\n\n
You can obtain detailed statistics of the softirq networking system by examining the file \/proc\/net\/softnet_stat<\/code> which outputs information about the number of packets, the number of drops, and the time squeeze counter which tracks the number of times the budget or time limit were consumed, but more work was available.<\/p>\n\n\n\n
NAPI poll == e1000e_poll<\/code><\/h3>\n\n\n\n
It is up to the device driver to clear packets that were DMA\u2019d into the rx ring buffer. This is accomplished by the poll<\/code> method called in the code sample above, which in the case of e1000e, is actually a function pointer to the function e1000e_poll<\/code> (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:2638<\/a>).<\/p>\n\n\n\n
The e1000e driver\u2019s e1000e_poll<\/code> function calls a function via a function pointer named clean_rx<\/code>. It is provided the weight (which was hardcoded to 64 during driver initialization), and a location to write the amount of work done (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:2638<\/a>):<\/p>\n\n\n\n
adapter->clean_rx(adapter->rx_ring, &work_done, weight);<\/pre>\n\n\n\n
This function pointer is set in e1000_open<\/code> when the driver is initialized and the device is brought up. It is set to the an appropriate function based on the MTU.<\/p>\n\n\n\n
For our purposes, this is the e1000_clean_rx_irq<\/code> function.<\/p>\n\n\n\n
e1000_clean_rx_irq<\/code> \u2013 unmap DMA regions and pass data up the stack<\/h3>\n\n\n\n
The e1000_clean_rx_irq<\/code> function runs in a loop, and breaks out when the work done reaches the weight passed into the function.<\/p>\n\n\n\n
The function unmaps memory regions that the device has DMA\u2019d data to. Those memory regions are unmapped so they cannot be written to by the device.<\/p>\n\n\n\n
Stat counters are incremented for total_rx_bytes<\/code> and total_rx_packets<\/code> and some additional memory regions for DMA are added back to the rx ring.<\/p>\n\n\n\n
Finally, e1000_receive_skb<\/code> is called to hand the skb up the network stack.<\/p>\n\n\n\n
e1000_receive_skb<\/code> \u2014 pass data up the stack<\/h3>\n\n\n\n
The function e1000_receive_skb<\/code> starts a chain of function calls that deal with bookkeeping for things like hardware accelerated vlan tagging and generic receive offloading.<\/p>\n\n\n\n
The chain of functions calls is:<\/p>\n\n\n\n