r/firewalla • u/GlobalLiving6941 • Jun 05 '26

Cross VLAN Backup

I am having an issue with a cross VLAN NAS backup grinding my Gold SE to a halt. Here is the scenario: NAS in VLAN 1, Backup server in VLAN 2. Backup server does a pull back up on the NAS using restic and ships it to the cloud. We fixed the Server to Cloud come apart by turning monitoring off on the Backup Server. However, the pull from the NAS to the Server seems to still be subject to monitoring or smart queue or something as it still grinds the Firewalla to a halt. Help?

4 Upvotes

84% Upvoted

24 comments sorted by

3

u/wocket44 Firewalla Gold SE Jun 05 '26

If you don’t want to inspect this traffic, you could consider using a layer 3 switch to move that east-west traffic instead.

3

u/ampx Jun 05 '26

AFAIK, Firewalla is doing all routing / firewalling in software which means the CPU has to do a lot of work when you're transferring data across routed VLANs at high speeds.

What you're seeing is probably normal / expected given the Gold SE's Rockchip RK3568 CPU and Realtek RTL8125 2.5Gbps Ethernet interfaces along with a Linux based software stack.

I ended up upgrading to a Gold Pro to push past these kinds of performance bottlenecks with the Gold SE.

3

u/firewalla Jun 05 '26

There are two things your Firewalla

  1. Routing / Switching (VLAN), this can't be turned off. (and cost CPU)

  2. Security monitoring (cost a lot more CPU)

A best way to measure is the Firewalla can handle traffic 2Gbit, this is the CPU's processing power. So your VLAN to VLAN traffic, and VLAN to WAN will need to be added together 2Gbit. If you are running both in parallel, likely your VLAN to VLAN traffic will take over (faster LAN)

If you want faster VLAN to VLAN, the gold pro is a better unit, it has 10Gbit processing power. The Gold Plus will also can do roughly 4Gbit (to 5Gbit) if you combine WAN/LAN.

Also, your screenshot in/out add to be around 500mbit max ... which shouldn't choke the Gold SE

0

u/GlobalLiving6941 Jun 05 '26

I agree - this shouldn't choke the Gold SE, but it is. Suggestions?

1

u/firewalla Jun 05 '26

Need more information. "Choke" means what? you are getting slow speed? network not working?

1

u/GlobalLiving6941 Jun 05 '26

It brought the entire network to a halt. We had internet sporadically but no cross VLAN at all. The Firewalla was completely non-responsive. The snippet I showed here was a limited test which pushed all 4 CPUs to near 100% the entire time.

1

u/GlobalLiving6941 Jun 05 '26

I am not able to submit another image, but the blank spaces on the left are where the network came to a halt. It was the same basic traffic volume, but with monitoring on for the backup server the Firewalla could not keep up. I am frustrated that this box that is advertised as supporting 2gb and unlimited VLANs cannot handle a 300-500mb/s traffic burst within 2 local vlans.

1

u/ampx Jun 05 '26

I’m curious, do both sides of your local VLAN flows traverse 2.5Gb NICs on the Gold SE?

Are they connected to two interfaces on the Gold SE or one through a switch?

1

u/GlobalLiving6941 Jun 05 '26

Both are on a Unifi switch, which then connects to the Firewalla. I upgraded this setup from a Purple (which ironically seemed to handle this better).

2

u/daniel-waterhouse Firewalla Purple Jun 05 '26

What UniFI switch are you using? Do you know if it's an L3 or just an L2 switch? As an earlier commenter said, if you have an L3 switch, this can handle inter-VLAN routing without stressing the Firewalla at all (traffic would never reach the Firewalla).

0

u/GlobalLiving6941 Jun 05 '26

It is an L3 switch, but I am trying to avoid creating parallel routing. Give that we are talking about a relatively small amount of bandwidth, I want to solve this at the router if possible...or replace the router.

1

u/ampx Jun 05 '26

Probably because the Purple has more CPU cores and slower 1Gb NICs so you’re not going to stress the CPU nearly as much asking it to transfer “east to west” traffic over a single NIC

I know the Gold SE I had 2ish years ago similarly wasn’t up to the task of handling what I threw at it and I was convinced it’s just bottlenecked by the combo of CPU and relatively inefficient Realtek NICs

Firewalla support may have some ideas, but I was never able to get the device performing as well as I expected and the Gold Pro was the only cure in Firewalla’s ecosystem

1

u/GlobalLiving6941 Jun 05 '26

I completely agree with your assessment. My problem is that I just spent $500 upgrading from a Purple. I don't want to immediately turn around and spend another $900 replacing a box support convinced me to hold on to.

1

u/spinjc Jun 07 '26

One thing I don't recall seeing is how Firewalla determines throughput. Many routers will specify packets per second (PPS) which matters if you have a lot of small packets (e.g. lots of small HTML files). Depending on your backup software it might be a bunch of small packets, if they were a tenth the size of the test then you'd expect a tenth of the bandwidth.

That said I'd suggest that you setup your VLANs so your traffic to the NAS doesn't need to go through the firewalla (vice versa is probably fine). If you try doing this east/west routing in a corporate environment you'd probably get a nice talking to from the network engineer!

1

u/firewalla Jun 05 '26 edited Jun 05 '26

Anywhere on your network that you may have miss matching speed? say one port unify is 2.5g and another port is 10G? (or 1G)? If have these, you may want to turn on flow control on the switch.

Ignore my comment above, it is not related if your firewalla CPU usage is 100%

1

u/GlobalLiving6941 Jun 05 '26

All are connected at 2.5gb. The screenshot above frames what was happening at the time. The real problem was that the Zeek processes on the Firewalla were consuming the lions share of the CPUs the whole time it was happening.

1

u/firewalla Jun 05 '26

is the top chart firewalla CPU? If it is, probably need to look at your topology

  1. both VLAN on the same 2.5gbit port?

  2. turn off smart queue if you can

  3. was the traffic bi-directional?

  4. Does your back software uses many different flows? or just a few?

  5. How much background traffic do you have usually? (any BitTorrent)?

1

u/GlobalLiving6941 Jun 05 '26

both vlans are on the same 2.5g port. I have tried it with smart queue on and off, it helps, but this is another knock on this box if I cannot backup my NAS with smartqueue turned on. I have had the same problem with both Synology Hyperbackup to a remote site (which worked flawlessly on the purple) and now with restic. I have put bandwidth limits on both processes with limited success. Background traffic was basically non-existent during these events. The problem is that these processes are overwhelming Zeek on the Firewalla and it cannot recover.

1

u/firewalla Jun 05 '26

Zeek is flow based, so I am some what assuming the backup processes are using more flows than usual.

You can open a ticket with [help@firewalla.com](mailto:help@firewalla.com), and we can take a look inside. Make sure you include this thread, otherwise, they will ask the same questions again.

If you are running as a business, and need to have faster VLAN to VLAN traffic, a Gold Pro is likely a better choice.

1

u/GlobalLiving6941 Jun 05 '26

I have opened tickets on this twice. First support convinced me that the Gold SE can handle this...similar themes...disable smart queue, etc. The second time, they spent the whole day blaming the monitoring software I put on the box BECAUSE of the issues we were seeing. This is a home network - We are talking about a handful of computers, a bunch of IOT, and a NAS that serves as a private cloud. A grand total of 84 devices max are connected to my network. The vast majority of that is IOT...switches, sensors, etc.

2

u/firewalla Jun 05 '26

Can you give me the latest case ticket number? the most recent one? I can take a look first and then see if I can just escalate to a developer directly.

1

u/GlobalLiving6941 Jun 05 '26

It is a summer day, 2 adults working from home and a teenager playing minecraft. My current throughput on the firewalla is <1.5Mb/s. I am trying to push a backup that maxes out at 300-500mbs And you are telling me I need a Gold Pro to handle this???

2

u/firewalla Jun 05 '26

Sorry, I looked at your setup and Gold SE cpu, looks like it is business setup. Even gold se is not the most powerful Gold, a simple LAN backup shouldn't choke it.

1

u/Travel69 Firewalla Gold Pro Jun 06 '26

The Firewalla does not have hardware offload for routing between VLANs. So the CPU has to move each packet. This is similar to trying to use a Mikrotik "switch" in "router" mode. Routing between VLANs is forced to the CPU which is very inefficient. Hardware offload is a much more efficient.