Occasional tunnel break - CGNAT is the culprit?

[mcfly9]

Hi,

I have the following setup:

device c --> 5G (CGNAT) --> internet <--> pppoe fiber <--> device F

• device F: using pppoe fiber internet, changes public IP's once every week, has a dynamic dns name registered
• device C: using a 5G connection, behind CGNAT, has no DNS name registered

These are 2 pfsense installs with a site-to-site link in between.
Peer config representing device C in device F is configured as dynamic (IP not known / behind CGNAT).
To handle public IP address changes of device F, I am running a watchdog script on device C that restarts the wireguard service in pfsense on device C when the IP of device F changes (using periodic DNS lookups to detect a change in IP).

I have noticed though that in some cases, after the wireguard service restart on device C the endpoint address shown for device F is its not the new IP, but its previous IP address.
In this case, I even tried doing wg set tun_wg0 peer <key> endpoint <newip>:port, which is briefly reflected in the output of wg show but then the peer's IP switches back to the previous (now not valid) IP in the wg show output.
Obvisouly, in this state, the link is not operable. As expected, neither end shows any recent handshakes.
Curiously though, on device C, both the sent and received (!!!) size keeps increasing.

Looks almost like if the wireguard client on device C continued to receive valid wireguard traffic from the old IP address and updated the peer endpoint address automagically.

No matter if I restart the wireguard service on either end, does not help. The only remedy seems to be restarting the 5G modem itself - possibly triggering some CGNAT state resets.

Anyone experienced similar behavior? Seems like a broken CGNAT implementation? T-Mobile HU is the carrier.