In the VyOS and Mikrotik OSPF Routing post, I set up a small OSPF lab with a Fitlet2 running VyOS 1.2.8 and two Mikrotik RB450Gx4’s on RouterOS 6.47.10. The performance testing results left some questions so I thought I would modify the OSPF lab a bit and allow not only testing between routers in the OSPF area but also testing on the same router in the area. I happened to have a Ubiquiti EdgeRouter X laying around so I figured I would throw that in the mix as well to check performance. The major two changes to this configuration is that I will not be using a LACP on the Fitlet2 and every thing will be connected to the same Layer 2 broadcast domain (upon which they will receive roughly 150 routes via OSPF). Again the idea is to compare performance to the CPU (and maybe memory utilization if that becomes and issue).
Hardware
Compulab Fitlet2
The Fitlet2 being used here is a model with an Intel Atom E3950 (quad-core @1.6 GHz) with 8 GB of memory and 16 GB of storage. It also has a FC-M2LAN FACET card which provides an additional two Gigabit Ethernet interfaces (for a total of four). The Fitlet2 is running VyOS 1.2.8 (the most recent LTS release at this time).
Mikrotik RB450Gx4
The RB450Gx4 is an updated version of Mikrotik’s popular RB450 line of RouterBoards. This is the 4th iteration (first being the RB450, then the RB450G, the RB850G, and now the RB450G) of the platform that has been released in the last decade. It features an ARMv7 (quad-core @716 MHz) with 1 GB of memory and 512 MB of storage. The RB450Gx4 is running RouterOS 6.47.10 (the most recent LTS release at this time).
Ubiquiti EdgeRouter X
The ER-X features a MIPS (dual-core @880 MHz) with ~250 MB of memory and ~214 MB of storage. It is rumored that as of August 2021, production on the ER-X has ended. The ER-X is running on Edge OS v2.0.9-hotfix.2.
Device Setup
All three devices have basic configuration on them (including static routes) to allow them to be polled via SNMP by my NMS. The static routes will be removed when OSPF is configured.
Fitlet2
Interface: eth0 10.1.10.3/24
eth1 172.16.1.254/24
eth3 172.16.2.254/24
lo 172.16.255.1/32
OSPF: area 0.0.0.9
network
10.1.10.0/24
172.16.1.0/24
172.16.2.0/24
172.16.255.1/32ER-X
Interface: eth0 10.1.10.4/24
eth1 172.16.3.254/24
eth2 172.16.4.254/24
lo 172.16.255.2/32
OSPF: area 0.0.0.9
network
10.1.10.0/24
172.16.3.0/24
172.16.4.0/24
172.16.255.2/32RB450Gx4
Interface: eth0 10.1.10.2/24
eth1 172.16.4.254/24
eth2 172.16.6.254/24
lo 172.16.255.3/32
OSPF: area 0.0.0.9
network
10.1.10.0/24
172.16.5.0/24
172.16.6.0/24
172.16.255.3/32Configuration
Fitlet2
First we will configure the interfaces in configure mode. Eth0 is already configured.
set interfaces ethernet eth1 address 172.16.1.254/24
set interfaces ethernet eth3 address 172.16.2.254/24
set interfaces loopback lo address 172.16.255.1/32Next we will delete the existing static route and configure OSPF.
delete protocols static
set protocols ospf area 0.0.0.9 network 10.1.10.0/24
set protocols ospf area 0.0.0.9 network 172.16.1.0/24
set protocols ospf area 0.0.0.9 network 172.16.2.0/24
set protocols ospf area 0.0.0.9 network 172.16.255.1/32Finally we will configure DHCP on eth1 and eth3.
set service dhcp-server shared-network-name dhcp1 subnet 172.16.1.0/24 default-router '172.16.1.254'
set service dhcp-server shared-network-name dhcp1 subnet 172.16.1.0/24 lease '600'
set service dhcp-server shared-network-name dhcp1 subnet 172.16.1.0/24 range pool1 start '172.16.1.1'
set service dhcp-server shared-network-name dhcp1 subnet 172.16.1.0/24 range pool1 stop '172.16.1.10'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.2.0/24 default-router '172.16.2.254'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.2.0/24 lease '600'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.2.0/24 range pool2 start '172.16.2.1'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.2.0/24 range pool2 stop '172.16.2.10'Finally we can commit the configuration to operational status and then save the running config to config.boot (i.e. the startup config). From the operational mode, executing show ip ospf neighbor returns the neighbor information from the Area Border Gateway and show ip ospf route should return some routes (about 150 in my case). Plugging in a DHCP client on one of the interfaces we should be assigned a DHCP address (first time to setup DHCP on VyOS) and show dhcp server leases returns an active lease. This router is ready.
ER-X
Next we do the same with the ER-X. Now the ER-X has a nice web UI but I’m going to use the command line for reasons which should become obvious by the time the config is saved to the startup config. First, interface configuration (login and enter configuration mode via configure).
set interfaces ethernet eth1 address 172.16.3.254/24
set interfaces ethernet eth2 address 172.16.4.254/24
set interfaces loopback lo address 172.16.255.2/32That was strangely familiar. Anyway, on to OSPF (and removing the existing static route).
delete protocols static route
set protocols ospf area 0.0.0.9 network 10.1.10.0/24
set protocols ospf area 0.0.0.9 network 172.16.3.0/24
set protocols ospf area 0.0.0.9 network 172.16.4.0/24
set protocols ospf area 0.0.0.9 network 172.16.255.2/32Just like with VyOS, the /32 network will automatically become the router-id so no need to manually set it. On to the DHCP servers.
set service dhcp-server shared-network-name dhcp1 subnet 172.16.3.0/24 default-router '172.16.3.254'
set service dhcp-server shared-network-name dhcp1 subnet 172.16.3.0/24 lease '600'
set service dhcp-server shared-network-name dhcp1 subnet 172.16.3.0/24 start '172.16.3.1' stop '172.16.3.10'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.4.0/24 default-router '172.16.4.254'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.4.0/24 lease '600'
set service dhcp-server shared-network-name dhcp2 subnet 172.16.4.0/24 start '172.16.4.1' stop '172.16.4.10'Now commit the config and save it. Except for the the minor differences in the DHCP lease pool, VyOS and EdgeOS configured identically. It’s almost like Ubiquiti used Vyatta to build EdgeOS…
RB450Gx4
We will start off by creating a loopback interface and configuring IP addresses.
/interface bridge add name=lo
/ip address add address=172.16.5.254/24 interface=ether2
/ip address add address=172.16.6.254/24 interface=ether3
/ip address add address=172.16.255.3/32 interface=loNow we configure OSPF.
/routing ospf instance set 0 router-id=172.16.255.3
/routing ospf area add name=area9 area-id=0.0.0.9
/routing ospf network add area=area9 network=172.16.5.0/24
/routing ospf network add area=area9 network=172.16.6.0/24
/routing ospf network add area=area9 network=10.1.10.0/24
/routing ospf network add area=area9 network=172.16.255.3/32Finally DHCP.
/ip pool add name=pool0 ranges=172.16.5.1-172.16.5.10
/ip pool add name=pool1 ranges=172.16.6.1-172.16.6.10
/ip dhcp-server add name=dhcp0 interface=ether2 address-pool=pool0 disabled=no
/ip dhcp-server add name=dhcp1 interface=ether3 address-pool=pool1 disabled=no
/ip dhcp-server network add address=172.16.5.0/24 gateway=172.16.5.254 dns-none=yes
/ip dhcp-server network add address=172.16.6.0/24 gateway=172.16.6.254 dns-none=yesPerformance Testing
The performance testing here is going to be very similar to what was done in the last post. Two Minix Z83’s will be plugged into independent interfaces on the router and iPerf3 will be used to conduct throughput tests while monitoring CPU performance. If any of these results come back as less than the benchmark (testing between the two Z83’s on the same Layer 2 segment) then we know that the router is struggling to keep up (which should also be evident with a high CPU load).
Benchmark
| Packet Size (Bytes) | Average Throughput (Mbps) | Streams |
| 128 | 185.5 | 8 |
| 256 | 419 | 8 |
| 512 | 693.5 | 8 |
| 1024 | 868 | 8 |
| 1500 | 903.5 | 8 |
Testing Results
Fitlet2
| Packet Size (Bytes) | Average Throughput (Mbps) | Packet Throughput (kpps) | Average CPU Utilization (%) |
| 128 | 178 | 173.8 | 29 |
| 256 | 437.5 | 213.6 | 22 |
| 512 | 727.5 | 177.6 | 19 |
| 1024 | 875 | 106.8 | 3 |
| 1500 | 909 | 75.7 | 3 |
EdgeRouter X
| Packet Size (Bytes) | Average Throughput (Mbps) | Packet Throughput (kpps) | Average CPU Utilization (%) |
| 128 | 36.3 | 35.4 | 78 |
| 256 | 103 | 50.3 | 78 |
| 512 | 228.5 | 55.8 | 86 |
| 1024 | 484.5 | 59.1 | 84 |
| 1500 | 709 | 59.1 | 88 |
RB450Gx4
| Packet Size (Bytes) | Average Throughput (Mbps) | Packet Throughput (kpps) | Average CPU Utilization (%) |
| 128 | 179.5 | 175.3 | 3 |
| 256 | 430.5 | 210.2 | 1 |
| 512 | 707 | 172.6 | 1 |
| 1024 | 883 | 107.8 | 1 |
| 1500 | 919 | 76.6 | 1 |
Conclusion
Both the RB450Gx4 and the Fitlet2 seem to perform adequately and the real limitations seems to be the Minix Z83’s being used for speed testing as the throughput testing is all very close to the benchmarks between the two Z83s. While the ER-X did perform significantly worse, Ubiquiti only spec’d the device at 80 kpps, which I feel is fairly accurate considering the ER-X was running the web UI, accepting SNMP polling, had two independent DHCPs servers running in addition to OSPF running. Those numbers are likely ideal with everything disabled minus static routes. Now comparing an RB450Gx4 to a ER-X isn’t exactly fair but it is simply the equipment I had on hand and either a hEX or RBM33 would not only be at a similar price point but with similarly spec’d processors MMIPS and MIPSBE processors would be more in-line for comparison.
Now the RB450Gx4 and Fitlet2 are better for comparison as we don’t really know the capabilities of the Fitlet2 and Mikrotik is generally really good about publishing throughput testing results of their products. Throughout the the throughput testing, the RB450Gx4 and Fitlet2 perform very similarly. The Mikrotik hardly seems to break a sweat while the Fitlet2’s CPU on the is peaking close to 30% for the smaller bandwidth testing. This could be related to the design of the RB450Gx4 (all ports are interfaced to a switch chip and then passed to the CPU via a 2 Gbps interface) and while I do not believe Router OS v6 supports any form of routing offload onto the switch chip, that is supposed to be something being introduced in Router OS v7 for certain hardware setups. The main reason for me wanting to compare these routers was to obtain a rough estimation of the routing capabilities of the Fitlet2 due to the Fitlet2’s ability to handle a much broader temperature range compared to what Mikrotik rates and tests their devices for. At ~$120 USD once the enclosure is purchased, the RB450Gx4 has a hardware limitation (well documented) of just under 2 Gbps of routing capability. At closer to $220 USD a Fitlet2 has the potential to route enough packets to support up to 4 Gbps throughput while using standard sized packets without overly taxing the CPU.
