{"id":1938,"date":"2016-01-22T17:51:06","date_gmt":"2016-01-23T01:51:06","guid":{"rendered":"https:\/\/www.privateinternetaccess.com\/blog\/?p=1938"},"modified":"2024-02-01T01:10:57","modified_gmt":"2024-02-01T09:10:57","slug":"linux-networking-stack-from-the-ground-up-part-2","status":"publish","type":"post","link":"https:\/\/www.privateinternetaccess.com\/blog\/linux-networking-stack-from-the-ground-up-part-2\/","title":{"rendered":"Linux networking stack from the ground up, part 2"},"content":{"rendered":"\n

part 1<\/a> | part 2<\/a> | part 3<\/a> | part 4<\/a> | part 5<\/a><\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n

This post will pick up where part 1 left off, beginning by explaining what ethtool<\/code> is, how device drivers register code for ethtool<\/code>, how drivers enable NAPI, and how drivers enable interrupts.<\/p>\n\n\n\n

ethtool setup<\/h2>\n\n\n\n

ethtool<\/code> is a command line program you can use to get and set driver
information. You can install it on Ubuntu by running apt-get install ethtool<\/code>.<\/p>\n\n\n\n

Some ethtool<\/code> settings of interest will be described later in this document.<\/p>\n\n\n\n

The ethtool<\/code> program talks to device drivers by using the ioctl<\/code> system call.
The device drivers register a series of functions that run for the ethtool<\/code> operations and the kernel provides the glue.<\/p>\n\n\n\n

When an ioctl<\/code> call is made from ethtool<\/code>, the kernel finds the ethtool<\/code> structure registered by the appropriate driver and executes the functions registered.<\/p>\n\n\n\n

e1000e<\/h3>\n\n\n\n

The ethtool<\/code> functions are installed in the e1000e<\/code> driver in the PCI probe<\/code> function (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:6627<\/a>):<\/p>\n\n\n\n

e1000e_set_ethtool_ops(netdev);<\/pre>\n\n\n\n
Which registers a structure of function pointers for each of the ethtool functions supported by e1000e<\/code> (accessing stats, changing ring buffer sizes, etc) from drivers\/net\/ethernet\/intel\/e1000e\/ethtool.c:2316<\/a>.<\/p>\n\n\n\n
igb<\/h3>\n\n\n\n
The ethtool<\/code> functions are installed in the igb<\/code> driver in the PCI probe<\/code> function (drivers\/net\/ethernet\/intel\/igb\/igb_main.c:2091<\/a>):<\/p>\n\n\n\n
igb_set_ethtool_ops(netdev);<\/pre>\n\n\n\n
From drivers\/net\/ethernet\/intel\/igb\/igb_ethtool.c:1905<\/a>.<\/p>\n\n\n\n
ixgbe<\/h3>\n\n\n\n
The ethtool<\/code> functions are installed in the ixgbe<\/code> driver in the PCI probe<\/code> function (drivers\/net\/ethernet\/intel\/ixgbe\/ixgbe_main.c:7883<\/a>):<\/p>\n\n\n\n
ixgbe_set_ethtool_ops(netdev);<\/pre>\n\n\n\n
From drivers\/net\/ethernet\/intel\/ixgbe\/ixgbe_ethtool.c:7686<\/a>.<\/p>\n\n\n\n
tg3<\/h3>\n\n\n\n
The ethtool<\/code> functions are installed in the tg3<\/code> driver in the PCI probe<\/code> function (drivers\/net\/ethernet\/broadcom\/tg3.c:17456<\/a>):<\/p>\n\n\n\n
dev->ethtool_ops = &tg3_ethtool_ops;<\/pre>\n\n\n\n
be2net<\/h3>\n\n\n\n
The ethtool<\/code> functions are installed in the be2net<\/code> driver in a function called from the PCI probe<\/code> function (drivers\/net\/ethernet\/emulex\/benet\/be_main.c:4094<\/a>):<\/p>\n\n\n\n
SET_ETHTOOL_OPS(netdev, &be_ethtool_ops);<\/pre>\n\n\n\n
bnx2<\/h3>\n\n\n\n
The ethtool<\/code> functions are installed in the bnx2<\/code> driver in a function called from the PCI probe<\/code> function (drivers\/net\/ethernet\/broadcom\/bnx2.c:8539<\/a>):<\/p>\n\n\n\n
dev->ethtool_ops = &bnx2_ethtool_ops;<\/pre>\n\n\n\n
NAPI poll<\/h3>\n\n\n\n
Prior to the existence of NAPI, NICs would generate an interrupt for each packet received indicating that data is available to be processed by the kernel.<\/p>\n\n\n\n
NAPI<\/a> changes this by allowing a device driver to register a poll<\/code> function that the NAPI subsystem will call to harvest packets. This method of gathering packets has reduced overhead compared to the older method, as many packets can be consumed at a time instead of processing only a single packet per interrupt.<\/p>\n\n\n\n
The device driver implements a poll<\/code> function and registers it with NAPI using netif_napi_add<\/code>. When registering a NAPI poll<\/code> function with netif_napi_add<\/code>, the driver will also specify the \u201cweight\u201d. Most of the drivers hardcode a value of 64<\/code>. this value and its meaning will be described in more detail below.<\/p>\n\n\n\n
Typically, drivers register their NAPI poll<\/code> functions during driver initialization. In the drivers this document examines, the poll<\/code> function is registered in the PCI probe<\/code> function itself or in a helper function called from there.<\/p>\n\n\n\n
e1000e<\/h3>\n\n\n\n
The e1000e<\/code> driver registers its NAPI poll<\/code> function in the e1000_probe<\/code> function (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:6629<\/a>):<\/p>\n\n\n\n
netif_napi_add(netdev, &adapter->napi, e1000e_poll, 64);<\/pre>\n\n\n\n
e1000e<\/code> registers a single NAPI poll<\/code> function because this device supports only a single receive queue. All of the other drivers being examined support multiple receive queues and will call this function to register multiple NAPI poll<\/code> functions.<\/p>\n\n\n\n
igb<\/h3>\n\n\n\n
The igb<\/code> driver registers its NAPI poll<\/code> function in igb_alloc_q_vector<\/code> (drivers\/net\/ethernet\/intel\/igb\/igb_main.c:1180<\/a>):<\/p>\n\n\n\n
netif_napi_add(adapter->netdev, &q_vector->napi, igb_poll, 64);<\/pre>\n\n\n\n
This function is called from igb_alloc_q_vectors<\/code>. igb_alloc_q_vector<\/code> is called multiple times to initialize each of the RX and TX queues. igb_alloc_q_vectors<\/code> is called from igb_init_interrupt_scheme<\/code> which is called from several locations, one of which is igb_probe<\/code>.<\/p>\n\n\n\n
ixgbe<\/h3>\n\n\n\n
The ixgbe<\/code> driver registers its NAPI poll<\/code> function in ixgbe_alloc_q_vector<\/code> (drivers\/net\/ethernet\/intel\/ixgbe\/ixgbe_lib.c:813<\/a>):<\/p>\n\n\n\n
netif_napi_add(adapter->netdev, &q_vector->napi, ixgbe_poll, 64);<\/pre>\n\n\n\n
Similar to igb<\/code>, this function is called from ixgbe_alloc_q_vectors<\/code> and it is called multiple times to initialize each of the RX and TX queues. ixgbe_alloc_q_vectors<\/code> is called from ixgbe_init_interrupt_scheme<\/code> which is called from several locations, once of which is ixgbe_probe<\/code>.<\/p>\n\n\n\n
tg3<\/h3>\n\n\n\n
The tg3<\/code> driver registers its NAPI poll<\/code> function in tg3_napi_init<\/code> (drivers\/net\/ethernet\/broadcom\/tg3.c:7366<\/a>):<\/p>\n\n\n\n
netif_napi_add(tp->dev, &tp->napi[i].napi, tg3_poll_msix, 64);<\/pre>\n\n\n\n
This is called within a loop registering a NAPI poll<\/code> function for each RX and TX queue. It is called from tg3_start<\/code>, which is called from tg3_open<\/code>. Unlike the other drivers, tg3<\/code> registers its NAPI poll<\/code> function in ndo_open<\/code> and not in PCI probe<\/code>.<\/p>\n\n\n\n
be2net<\/h3>\n\n\n\n
The be2net<\/code> driver registers its NAPI poll<\/code> function in be_evt_queues_create<\/code> (drivers\/net\/ethernet\/emulex\/benet\/be_main.c:2053<\/a>):<\/p>\n\n\n\n
netif_napi_add(adapter->netdev, &eqo->napi, be_poll, BE_NAPI_WEIGHT);<\/pre>\n\n\n\n
This is called within a loop registering a NAPI poll<\/code> function for each RX and TX queue. It is called from be_setup_queues<\/code>, which is called from be_setup<\/code>, which is called from be_probe<\/code>.<\/p>\n\n\n\n
bnx2<\/h3>\n\n\n\n
The bnx2<\/code> driver registers its NAPI poll<\/code> function in bnx2_init_napi<\/code> (drivers\/net\/ethernet\/broadcom\/bnx2.c:6322<\/a>):<\/p>\n\n\n\n
netif_napi_add(bp->dev, &bp->bnx2_napi[i].napi, poll, 64);<\/pre>\n\n\n\n
This is called within a loop registering a NAPI poll<\/code> function for each RX and TX queue. Like tg3<\/code>, bnx2<\/code> allocates queue memory in its ndo_open<\/code> function bnx2_open<\/code> and not in PCI probe<\/code>.<\/p>\n\n\n\n
Interrupt number<\/h3>\n\n\n\n
The interrupt number is obtained from the struct pci_dev<\/code> structure and stored on the net_device<\/code> structure:<\/p>\n\n\n\n
netdev->irq = pdev->irq;<\/pre>\n\n\n\n
Later in device initialization the IRQ handlers for this IRQ number will be registered<\/p>\n\n\n\n
Driver initialization<\/h2>\n\n\n\n
When a network device is brought up (for example, with ifconfig eth0 up<\/code>), an open<\/code> function is called in the device driver. A pointer to this function is installed in a net_device_ops<\/code> structure at a field named ndo_open<\/code>. In e1000e, this function is called e1000_open<\/code> (drivers\/net\/ethernet\/intel\/e1000e\/netdev.c:4241<\/a>).<\/p>\n\n\n\n
The open function will typically do things like:<\/p>\n\n\n\n
  1. Allocate RX and TX queue memory<\/li>
  2. Enable NAPI<\/li>
  3. Register an interrupt handler<\/li>
  4. Enable hardware interrupts<\/li><\/ol>\n\n\n\n
    And more.<\/p>\n\n\n\n
    Allocating RX and TX queue memory<\/h3>\n\n\n\n
    For example, the e1000e driver (found in drivers\/net\/ethernet\/intel\/e1000e\/<\/a>) in netdev.c<\/code> around line 4279<\/a>:<\/p>\n\n\n\n
    \/* allocate transmit descriptors *\/\nerr = e1000e_setup_tx_resources(adapter->tx_ring);\nif (err)\n  goto err_setup_tx;\n\n\/* allocate receive descriptors *\/\nerr = e1000e_setup_rx_resources(adapter->rx_ring);\nif (err)\n  goto err_setup_rx;\n<\/pre>\n\n\n\n
    The e1000e_setup_rx_resources<\/code> and e1000e_setup_tx_resources<\/code> allocate receive and transmit queues and initialize associated data structures. It is important to note that these queues are read and written directly from the NIC via DMA. In other words: when data arrives from the network, the data is written directly to the receive queue by the NIC via DMA. The queue size is defaulted to E1000_DEFAULT_RXD<\/code> (256) and the max is E1000_MAX_RXD<\/code> (4096). These values are driver specific.<\/p>\n\n\n\n
    If data arrives faster than it can be processed, it will fill the queue. Once the queue is full, additional data that arrives will be dropped.<\/p>\n\n\n\n
    You can determine if drops are happening and increase the queue size by using the command line tool ethtool<\/code>. ethtool<\/code> communicates with the device driver by using the ioctl<\/code> system call.<\/p>\n\n\n\n
    Most drivers have a file named ethtool.c<\/code> or *_ethtool.c<\/code> implementing this interface. Not all drivers implement every possible ethtool<\/code> method, so you should check the driver code and ethtool<\/code> output to determine if what you are doing is supported by the driver or not.<\/p>\n\n\n\n
    You can get stats from ethtool<\/code> by using the -S flag, for example:
    <\/code><\/p>\n\n\n\n
    ethtool -S eth0<\/pre>\n\n\n\n
    The names of the stats will differ from driver to driver, so you should read the output carefully and grep for things like \u201cdrop\u201d \u201cmiss\u201d and \u201cerror\u201d.<\/p>\n\n\n\n
    As far as e1000e is concerned:<\/p>\n\n\n\n