Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Wiki / Kb75

Kb75

Emulab FAQ: Troubleshooting: Why is the per-packet latency so high (variable) on the pc3000 nodes?

Emulab FAQ: Troubleshooting: Why is the per-packet latency so high (variable) on the pc3000 nodes?

The pc3000 nodes have Intel Pro1000 Gigabit adaptor cards which implement "interrupt throttling" which can add quantity and variability to the latency of packets. Interrupt throttling effectively caps the interrupt rate (either transmit or receive) presented to the CPU. See the README file that comes with the driver (/share/linux/e1000* or /share/freebsd/em*) and the referenced Intel application note http://www.intel.com/design/network/applnots/ap450.htm . Please note that the absolute and packet timer descriptions in the README file are backwards, in particular the descriptions of RxIntDelay and RxAbsIntDelay would lead you to believe that the default values have no effect when in fact they serve to limit receive interrupts to about 8000/sec (RxAbsIntDelay == 128).

To disable interrupt throttling on all interfaces in Linux, put the following as one line in your /etc/modules.conf (/etc/modprobe.conf in Fedora Core images) file:

options e1000 InterruptThrottleRate=0,0,0,0,0,0 \
  TxIntDelay=0,0,0,0,0,0 TxAbsIntDelay=0,0,0,0,0,0 \
  RxIntDelay=0,0,0,0,0,0 RxAbsIntDelay=0,0,0,0,0,0

and then reboot the machine.

For FreeBSD, while you can change the per-interface settings with sysctl (hw.em?.*_int_delay), you cannot overcome the global 8000 ints/sec. that is compiled into the kernel. To override that, you will need to modify the driver and build a new kernel. In /usr/src/sys/dev/em/if_em.c look for:

                /* Set the interrupt throttling rate.  Value is calculated
                 * as DEFAULT_ITR = 1/(MAX_INTS_PER_SEC * 256ns) */
#define MAX_INTS_PER_SEC        8000
#define DEFAULT_ITR             1000000000/(MAX_INTS_PER_SEC * 256)
                E1000_WRITE_REG(&adapter->hw, ITR, DEFAULT_ITR);

and modify to suit your needs.