A Journey to Build an Ethernet Packet Generator with Standard PC Components
When developing Ethernet switches, you need a system to benchmark and test these devices. Currently, I am working on an 8-port Ethernet switch based on OpenWrt, so I needed an 8-port packet generator to test it. This article shows my approach to building such a system using standard PC components.
So how do you get many Ethernet ports? Right—by adding PCIe cards to a motherboard. A good choice is the Intel I350 server network card, which is available with 1, 2, or 4 ports, has excellent Linux support, and can often be found used on eBay. From a retired home server, I had a 12-year-old Intel DH87RL motherboard with an Intel Pentium G3220 CPU. This board provides one PCIe 3.0 x16 slot and three PCIe 2.0 x1 slots. The plan was to install multiple network cards into this board.
I already had one I350-T2 Ethernet card with two ports, and I bought two I350-T4 cards with four ports each. When buying these used, double-check that you’re not getting versions with an ML2 interface. These won’t work because they use a different form factor and are not standard PCIe cards. The I350 cards use PCIe 2.1 with four lanes. Installing the first card in the x16 slot worked well, but to install the other two cards, the x1 slots needed modification.
One of the nice things about PCIe is that cards with more lanes can operate in slots with fewer lanes. So, a x4 card can run in a x1 slot. However, without modification, the cards don’t physically fit into x1 slots because part of the plastic slot blocks them. I used side cutters to carefully remove the obstructing plastic (be careful not to damage the pins or the motherboard). After this, I was able to install the remaining cards. It looks a bit unusual since the full connector isn’t engaged, but it works.
After powering on the PC, I heard three beeps and saw no display output. My first thought was, “Great, I broke the board.” However, three beeps usually indicate a memory issue. That was strange, since I hadn’t touched the memory. After removing the network cards, the PC booted normally again—so the motherboard was fine. I wondered what the connection between Ethernet cards and memory could be.
After some research, I found that some motherboards have issues with the SMBus when certain PCIe devices are installed. This bus is connected to PCIe pins B5 and B6, and a known workaround is to cover these pins with tape (see here). Since I could see the relevant traces on the Ethernet card PCB, I decided to cut the traces instead.
After making this modification, the system booted successfully, and all I350 devices were detected.
albrecht@albrecht-labor:~$ lspci | grep I350
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)The ip command also showed all Ethernet interfaces. In total, I now have a system with 11 physical Ethernet ports (10 from the I350 cards + the from the onboard port).
albrecht@albrecht-labor:~$ ip link | grep enp
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
3: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
4: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
5: enp1s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
6: enp1s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
7: enp3s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
8: enp3s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
9: enp3s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
10: enp3s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
11: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
12: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000The big surprise was that I could run five pairs of iperf3 (each with a server and client) at full Gigabit speed without overloading the 13-year-old CPU. That means the system handled about 10 Gbit/s of throughput in this setup. However, when I tried full-duplex operation, the theoretical 20 Gbit/s wasn’t reached. I suspect this is due to PCIe 2.0 x1 bandwidth limitations, since all x1 slots are connected to the CPU through a PCIe packet switch.
Now I have the technical foundation to run tests on my OpenWrt-based Ethernet switch.
P.S. If you’re wondering how I isolate the Ethernet ports from each other, please read this article.