What the symptom looks like
A cellular link drops or a fibre circuit goes down, and the plant expects the router to move traffic onto the backup path within seconds. Instead, the operators see one of these patterns:
- Session drops and does not come back — the primary WAN is dead, but the tunnel, MQTT session or SCADA poll only returns after a manual reboot.
- Slow recovery — traffic comes back, but several polls time out and the HMI shows stale values for 30–90 seconds after the link change.
- Flapping — the router switches to the backup and back again repeatedly, and every switch costs another outage window.
- Stuck on backup — the primary link has been restored for hours and the router is still routed over cellular.
"Recovery time" is not a single router specification. It is the sum of detection, decision, interface bring-up, address acquisition and application re-connection. Diagnosing it means measuring which of those stages is consuming the seconds, not simply replacing the router.
Safety and stop conditions
- Do not open the router housing or probe the terminal block or antenna connectors while power is applied. Work on an isolated supply and follow the site's lock-out procedure.
- Do not change failover settings on a router that is the only path for a safety function, a medical device link or an active remote-control session without a planned window and local fallback.
- Do not bypass, jumper or disable an isolation barrier, surge protector or any site safety device in order to "prove" a link is good.
- Stop and escalate if the fault involves a hot or discoloured power supply, an antenna installation on a mast (leave that to the rigging contractor), or any circuit that operates at mains potential.
- If the site has a contractual or regulatory reporting obligation for the outage, keep the test log rather than repeatedly resetting the device.
Likely cause groups
- Health-check design — the probe target, ping interval and failure threshold decide how quickly the router even notices the primary link is gone. A probe that still answers (a cached DNS reply, an endpoint on the local side of the outage, or a public address reachable through the operator's proxy) will hold the router on a dead path.
- Interface bring-up and addressing — cellular attach, PDP/PDU session setup, DHCP on the secondary WAN, and PPP negotiation all add time; a SIM that needs a full network registration is far slower than one that reconnects to a still-registered cell.
- Routing and NAT convergence — policy routes, static routes with high metric, source-NAT pools bound to the old interface, and tunnels bound to a literal WAN IP rather than a tracked interface.
- Tunnel and application behaviour — an IPsec or WireGuard tunnel that must renegotiate, an MQTT keep-alive of 60 seconds, a Modbus/TCP poll with a long timeout: the network may be up long before the application notices.
- Link-layer instability rather than true failure — marginal signal, a loose external antenna connector, a saturated uplink at the cell edge, or a congested primary circuit produce intermittent loss that triggers repeated decisions.
Safe checks, in order
1. Confirm which interface is active and for how long
Read the router's status page and event log before changing anything. Note the active WAN, the time of the last switch, and the reason recorded for it. Many industrial firmwares log events such as "WAN failover: mobile1 up" with a timestamp — that timestamp is your measurement baseline.
2. Measure the outage window end to end
Use two independent clocks: a poller on the far side of the tunnel and the router's own log. A simple and repeatable method is to record the moment a monitored interface goes down (visible from the router log) and the moment the far-end application resumes receiving data (visible from the SCADA or broker log). The gap between them is the real recovery time the site experiences. Do this during a planned test window, not during production.
3. Review the health-check configuration
Record the monitored IP or hostname, the check interval, the timeout, the retry count and the action taken on failure. Typical industrial routers ship with a check interval in the region of ten seconds and a small retry count; the effective detection time is roughly interval × retries, so a ten-second interval with three retries can take half a minute to decide — before any interface even starts to come up. Confirm that the probe target sits beyond the primary link and is not reachable via the backup path, or the check proves nothing.
4. Check for secondary-path dependencies
Verify that the backup interface can attach without manual intervention: SIM status, PIN lock, APN profile, roaming permission, and whether the modem is in a low-power or scheduled-off state. A backup that is administratively disabled or parked in a power-saving mode will only look like a slow failover.
5. Check what is bound to the old address
List every configuration object that references the primary WAN by address rather than by interface: IPsec peers, VPN gateways, port forwards, NAT rules, remote-management allow-lists and DDNS entries. Each of these can keep the site offline after the routing has already switched.
6. Test with a controlled interruption
With the site informed, disconnect the primary link at a safe point (unplug the WAN patch lead at the router, or administratively disable the interface in the configuration, never by touching live conductors) and time the recovery twice: once with a warm modem and once after a cold start, if the site permits. Repeating the test reveals whether the delay is deterministic or random.
Reading the results
- Detection dominates the delay (router log shows the failover decided long after the physical link died): shorten the check interval and retry count cautiously, and choose a probe target that truly reflects the path. Aggressive probing costs cellular data and can cause flapping on marginal links.
- Bring-up dominates (the decision is logged promptly but the mobile interface stays down): investigate SIM registration, APN and signal quality; check whether the modem is configured to fall back to a slower radio mode.
- Application dominates (routing is up quickly, the tunnel or broker session is not): shorten keep-alives where the application's design allows it, use interface-tracked tunnels rather than address-hardcoded peers, and consider a secondary management path for the far end.
- Flapping: the primary link is unstable rather than down. Treat it as a link-quality problem — signal, cabling, antenna alignment, carrier congestion — not as a failover tuning problem.
- No switch at all: confirm the failover feature is licensed and enabled, that the monitored interface is the one you think it is, and that no higher-priority static route pins traffic to the dead path.
When to escalate
- Recovery times remain unpredictable after configuration has been verified and repeated tests disagree by more than the site's tolerance.
- The modem never attaches, or attaches and drops repeatedly, on a SIM that is known to be provisioned and within coverage — raise it with the mobile operator with the attach logs.
- The router reboots, the log resets, or the device reports hardware or temperature alarms during the test.
- The recovery depends on a tunnel whose cryptographic configuration you are not authorised to change.
- The site's outage window has a documented service level; once measured recovery exceeds it, escalate to the system integrator before attempting further tuning.
Prevention
- Write the recovery expectation into the commissioning record: probe target, interval, retry count, expected total time, and how it was measured. "Failover enabled" is not a specification.
- Re-test failover at every firmware update and after any change to the carrier profile or APN.
- Keep the backup path genuinely independent: separate carrier where possible, separate physical route, and a SIM that is monitored for credit or data-cap exhaustion.
- Bind tunnels, NAT and management access to interfaces rather than literal addresses so a path change does not require reconfiguration.
- Log every failover event centrally; a rising count is an early warning of a degrading primary link.
Sources
- Teltonika Networks Wiki, RUT955 Failover and related RUT series failover pages — health-check parameters (monitored host, ping interval, ping timeout, retry count) as documented by the manufacturer.
- Mobile operator and router vendor configuration guides for APN, PDP context and interface tracking — consult the documentation for your specific model, since parameter names and defaults differ by firmware version.
Specifications and default values vary by model and firmware release. Verify the parameter names and defaults in the documentation for your exact device before changing settings on a live site.