In a compact system where the CPU handles both routing (IRQ processing) and storage (I/O wait), physical separation is almost always superior to VLANs when the hardware supports it.
1. Physical NICs vs. VLANs
Latency Under Load: Physical NICs reduce latency because each interface has its own hardware queues and interrupt (IRQ) lines. With VLANs, all traffic shares a single pipe; a massive file transfer can "saturate" the bus, causing packet queuing (bufferbloat) for your WAN/LAN traffic.
Stability: Large NVMe transfers generate high CPU overhead. Physical separation prevents a "broadcast storm" or heavy congestion on one link from dropping your management or routing sessions.
Hardware Offloading: Using separate NICs allows the OS to better distribute network processing across multiple CPU cores (RSS/Receive Side Scaling), which is critical for 2.5G speeds.
2. Best Practices for NAS Traffic
Most pros use Isolated Storage Networks:
Non-Routed Subnet: Keep the storage interface on a completely separate subnet (e.g., 10.0.50.x) that does not have a gateway. This ensures storage traffic never touches your routing engine, saving CPU cycles.
MTU Tuning: If your switch and clients support it, enable Jumbo Frames (MTU 9000) on the storage-only link. This reduces the number of interrupts the CPU has to process during large NVMe writes.
3. NVMe Performance Under Mixed Workload
NVMe drives are exceptionally fast, but in a "Router + NAS" box, the bottleneck isn't the NAND—it’s the PCIe lanes and Interrupts:
I/O Wait: If your Linux kernel is busy writing to NVMe, network throughput may jitter. Use an I/O scheduler like none or mq-deadline for NVMe to minimize CPU overhead.
Thermal Throttling: Compact systems trap heat. Heavy 2.5G routing + NVMe writes simultaneously can trigger thermal throttling, which will tank your network speeds. Ensure you have active cooling or a massive heatsink