Occasional Loss Of Connectivity

[Zema Bus]

Lately I've been randomly losing connectivity for a couple of minutes. This is different from the issue I had months ago where a YT video would knock out connectivity for all machines temporarily. In this case it does affect all machines at the same time - if one has an existing audio or video stream going it will keep going uninterrupted while on another machine I can't establish a connection to anything, so maybe a DNS issue. Tonight I thought I'd try my backup pfSense machine to see if it happens there, but it turns out it was borked by my prior attempt to upgrade pfSense on it, so it'll need to be reinstalled. I thought this might be a good time to checkout OPNSense, I've been considering moving to OPNSense for years since pfSense has become very corporate. So I got it installed, it uses the FreeBSD installer. Here's the web GUI. It needs to be configured and I don't feel like doing that now on a Sunday night so I'm back on my primary pfSense router for now. If I switch to OPNSense I won't be able to import my backup configuration from pfSense so I'll be configuring it from scratch.

opnsense1.jpg

opnsense2.jpg

[Grogan]

It's probably not your router OS, it's probably something to do with your connection. Have you tried power cycling whatever transceiver box you have ("modem?") when this happens?

Yes, it's possible for an established connection to stay working, especially if it's UDP (which video streams often are). When I was having modem problems, if I had something like a wireguard connection (UDP) going, it would stay working while nothing else on the network had connectivity until I'd power cycle the cable modem.

It's easy enough to troubleshoot DNS issues, simply change your DNS server (can do it in your own /etc/resolv.conf).

[Zema Bus]

That's another thing I'm going to explore, I want to verify how to properly reboot the tiny fiber optic box, it's very different from the old Verizon ONT box I was used to for years. My go-to place for stuff like this has always been broadbandreports.com / dslreports.com, going back over 20 years, but I was sad to see that they've shutdown as of January 2025. Only the front page remains up. I read a thread where someone said the owner "checked out". Guess that explains why their speed test had been broken for years.

[Zema Bus]

I cut power to the fiber box, it gets its power from the little switch mounted to the wall below it. It seems to be powered via POE, the other Ethernet cable then goes to my router via a long cable run to where the router is in another room. The fiber box is located in the living room behind the couch, to me that's kind of a silly location but that's where the previous owners had it wired. I'd rather have it in a dedicated "server room" along with my router, file server etc, like what I had at the other house. Maybe once I can get a metal building installed. Anyway, now to see if that's the last I see of that problem.

Airebeam_FIOS_box2.jpg

[Grogan]

Now that's a weird looking little thing to bet the farm on. I can't say I've ever seen anything like that.

[Zema Bus]

That's what I thought when I first saw it, after years of having a big industrial looking Verizon FIOS ONT box in the garage.

[Zema Bus]

Well it happened again today, only for 2 mins like all the other times, then everything was back to normal. Another possibility is my main switch, a 16 port TP-Link (unmanaged) I've had for 5 years. So, I power cycled it. Hope that was it.

[Grogan]

That sounds more like an upstream problem, there. A switch like that could need power cycling I guess, but there really isn't much to go wrong with it.

[Zema Bus]

It wasn't the switch either, and from the pfSense logs it would have to indeed be from upstream. I previously didn't think it was the same as the issue I was having months back, but now I'm not so sure, I'm thinking it in fact may be a return of the same issue. Here's what my logs look like immediately after I started a video stream on one computer and lost connectivity on everything else:

Code: Select all

Oct 10 12:02:03 	rc.gateway_alarm 	54853 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:1 RTT:508.529ms RTTsd:877.638ms Loss:11%)
Oct 10 12:02:03 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:02:03 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:02:03 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:02:03 	check_reload_status 	429 	Reloading filter
Oct 10 12:02:05 	php-fpm 	37095 	/rc.openvpn: MONITOR: WAN2_DHCP has high latency, omitting from routing group GWGroup
Oct 10 12:02:05 	php-fpm 	37095 	192.168.10.1|192.168.10.10|WAN2_DHCP|520.491ms|882.254ms|12%|down|highdelay
Oct 10 12:02:05 	php-fpm 	37095 	/rc.openvpn: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:02:05 	php-fpm 	37095 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:02:05 	php-fpm 	29618 	/rc.filter_configure_sync: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:02:05 	php-fpm 	29618 	/rc.filter_configure_sync: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:03:39 	rc.gateway_alarm 	47459 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:0 RTT:397.039ms RTTsd:958.696ms Loss:0%)
Oct 10 12:03:39 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:03:39 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:03:39 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:03:39 	check_reload_status 	429 	Reloading filter
Oct 10 12:03:40 	php-fpm 	753 	/rc.openvpn: MONITOR: WAN2_DHCP is available now, adding to routing group GWGroup
Oct 10 12:03:40 	php-fpm 	753 	192.168.10.1|192.168.10.10|WAN2_DHCP|397.015ms|958.705ms|0.0%|online|delay
Oct 10 12:03:40 	php-fpm 	753 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:04:08 	rc.gateway_alarm 	23521 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:1 RTT:516.339ms RTTsd:1260.932ms Loss:0%)
Oct 10 12:04:08 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:04:08 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:04:08 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:04:08 	check_reload_status 	429 	Reloading filter
Oct 10 12:04:09 	php-fpm 	29618 	/rc.filter_configure_sync: MONITOR: WAN2_DHCP has high latency, omitting from routing group GWGroup
Oct 10 12:04:09 	php-fpm 	29618 	192.168.10.1|192.168.10.10|WAN2_DHCP|517.193ms|1260.617ms|0.0%|down|highdelay
Oct 10 12:04:09 	php-fpm 	29618 	/rc.filter_configure_sync: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:04:09 	php-fpm 	37095 	/rc.openvpn: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:04:09 	php-fpm 	37095 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:04:09 	php-fpm 	29618 	/rc.filter_configure_sync: GATEWAYS: Group GWGroup did not have any gateways up on tier 1!
Oct 10 12:04:45 	rc.gateway_alarm 	31192 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:0 RTT:464.193ms RTTsd:1245.998ms Loss:0%)
Oct 10 12:04:45 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:04:45 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:04:45 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:04:45 	check_reload_status 	429 	Reloading filter
Oct 10 12:04:46 	php-fpm 	753 	/rc.openvpn: MONITOR: WAN2_DHCP is available now, adding to routing group GWGroup
Oct 10 12:04:46 	php-fpm 	753 	192.168.10.1|192.168.10.10|WAN2_DHCP|461.606ms|1241.078ms|0.0%|online|delay
Oct 10 12:04:46 	php-fpm 	753 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:05:01 	rc.gateway_alarm 	91701 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:1 RTT:510.930ms RTTsd:1248.209ms Loss:0%)
Oct 10 12:05:01 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:05:01 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:05:01 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:05:01 	check_reload_status 	429 	Reloading filter
Oct 10 12:05:02 	php-fpm 	37095 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:05:11 	rc.gateway_alarm 	44875 	>>> Gateway alarm: WAN2_DHCP (Addr:192.168.10.1 Alarm:0 RTT:118.926ms RTTsd:331.139ms Loss:0%)
Oct 10 12:05:11 	check_reload_status 	429 	updating dyndns WAN2_DHCP
Oct 10 12:05:11 	check_reload_status 	429 	Restarting IPsec tunnels
Oct 10 12:05:11 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Oct 10 12:05:11 	check_reload_status 	429 	Reloading filter
Oct 10 12:05:12 	php-fpm 	753 	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Oct 10 12:16:00 	sshguard 	64306 	Exiting on signal.
Oct 10 12:16:00 	sshguard 	66475 	Now monitoring attacks.
Oct 10 12:25:00 	sshguard 	66475 	Exiting on signal.
Oct 10 12:25:00 	sshguard 	77629 	Now monitoring attacks.
Oct 10 12:30:57 	php-fpm 	37095 	/status_logs.php: Session timed out for user 'admin' from: 192.168.1.146 (Local Database)
Oct 10 12:30:59 	php-fpm 	37095 	/status_logs.php: Successful login for user 'admin' from: 192.168.1.146 (Local Database) 

[Grogan]

So you're getting severe latency and packet loss.

Time to get in touch with your ISP's tech support? (The first call likely won't be very useful)

[Zema Bus]

Yeah I'm going to see what the best way is, I've heard that it's hard to get anywhere with their phone tech support. I'd like to get my logs to a tech who's not a script reader. One thing I used to like about the Broadband Reports / DSL Reports site is they had a section with techs from a lot of the ISPs, you could privately submit requests and issues and they'd take care of it. I used that service many times over the years. But that's gone now.

[Zema Bus]

I had an email composed including my router logs, I thought I'd try that first. But I realized there had been no occurrences all day that day, so I held off to see if it would happen the next day, but it didn't. It's now been a full week without a single occurrence. Apparently they fixed something, maybe they already received complaints. Or maybe this is just something that's going to come and go since it wasn't the first time.

[Grogan]

Yes, upstream problems like that may solve themselves, by the simple act of them changing equipment and/or topology. Mine has pretty much solved itself. The new coax line they insisted I get (took a while because they wouldn't bring it into the house and we had to get someone to drill and snake) didn't, but the problem hasn't happened to me in a long time. It's even been a long time since I've had to power cycle, but last time I did it wasn't the old symptoms.