Cell modem troubleshooting
Step 1 is to ensure you have aspirin or similar handy: you are likely to need it 
Step 2 is to set-up internet access or connect after a troublesome session so log files get uploaded and you can get remote help that way. Please post screen shots or CLI excerpts when asking for help!.
The first place to look for information is the cellular panel in the web UI, the Info portion shows some error information and while the modem is searching it typ. shows the set of carriers heard by the modem (scrolling down may be necessary).
Web UI panel
The cell modem stat field is the first place to look, it has a small help pop-up for the meaning of the various values:
- no modem: no hardware detected, is the USB cable plugged in (even if it’s a HAT)?
- disabled: SG software has not enabled the modem yet
- enabled: modem is trying to register (“connect with”) cellular network, this may take time and never change if the SIM does not provide service
- searching: modem is trying to locate a carrier to connect to, or in the process of connecting to a carrier; this may be transient if the modem is going from one carrier to the next or it may be more persistent if no carrier is found, the info panel should have a list of carriers seen by the modem
- registering: transient state attempting to register with cellular network, at this point the modem is talking to a tower and trying to get the SIM accepted
- registered: connected to the cellular network but no internet access; this could be due to an incorrect APN, a SIM that is not fully enabled or out of data allowance, or a SIM that does not allow roaming on the carrier chosen by the modem; my modem sometimes reverts to this state due to an error which requires a modem reset (ugh)
- connected: connected to the cellular network and internet access according to the modem
Command line tools
For in-depth info it is necessary to use command line tools: SSH into the Sensorgnome using your favorite terminal emulator. Username is gnome and the password is the same as the Web UI.
Listing current interfaces and their IP addresses: ip addr
The cell modem is generally wwan0 although it can also show up as ethernet device (confusingly):
8: wwan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/none
inet 10.13.101.239/27 brd 10.13.101.255 scope global dynamic noprefixroute wwan0
valid_lft 4997sec preferred_lft 4097sec
This shows that it is enabled (LOWER_UP) and that it has IP address 10.13.101.239
Listing internet routes: ip route
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.94 metric 1002
default via 10.13.101.240 dev wwan0 proto dhcp src 10.13.101.239 metric 10001 mtu 1500
10.13.101.224/27 dev wwan0 proto dhcp scope link src 10.13.101.239 metric 10001 mtu 1500
192.168.0.0/24 dev eth0 proto dhcp scope link src 192.168.0.94 metric 1002
Focus on the default routes, these are used to reach the internet at large, the other ones are for local traffic. The routes are shown in order of priority, so in this case the route via eth0 is used for all internet traffic, i.e., via the rPi’s ethernet interface. The cellular route (second line, using wwan0) is there just as a backup. To use it the first route needs to be removed, the easiest way being to unplug ethernet assuming you’re not connected via that…
Forcing traffic to use cellular
For testing purposes it is possible to force internet traffic to use the cell modem by adding a route with higher priority than other routes. Continuing the ip route example shown above, one could add a route as follows:
sudo ip route add default via 10.13.101.240 dev wwan0 metric 900
with the result:
> ip route
default via 10.13.101.240 dev wwan0 metric 900
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.94 metric 1002
default via 10.13.101.240 dev wwan0 proto dhcp src 10.13.101.239 metric 10001 mtu 1500
10.13.101.224/27 dev wwan0 proto dhcp scope link src 10.13.101.239 metric 10001 mtu 1500
192.168.0.0/24 dev eth0 proto dhcp scope link src 192.168.0.94 metric 1002
The route addition used the same address and device as the existing route (second line in the original example) but specified a metric lower than the eth0 route (metric specifies cost, so lower is better).
Checking packet routes
You can test internet connectivity by pinging 1.1.1.1 (or 8.8.8.8): ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=51 time=485 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=51 time=286 ms
Note that Motus servers do not appear to respond to pings (a very poor practice IMHO), so don’t test those…
You can try to verify that packets indeed flow through the cell modem using sudo traceroute 1.1.1.1 which in my example yielded:
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 212.2.102.1 (212.2.102.1) 442.836 ms 443.153 ms 442.936 ms
4 wawbal-i1-BE11.plusnet.pl (212.2.102.189) 443.111 ms 443.018 ms 442.926 ms
5 * * *
6 162.158.100.7 (162.158.100.7) 442.521 ms 319.940 ms 320.008 ms
7 one.one.one.one (1.1.1.1) 319.572 ms 319.416 ms 319.285 ms
This lists the network hops with * * * being unknown hops. Generally the ethernet or wifi interface will show the local router as first hop (see the via in the routes, e.g. 192.168.0.1 in my case) so its absence is good, and often you have to guess a little bit what the first or second shown hop belongs to. In my case, I use an APN called plus and the second hop is plusnet.pl (a low cost SIM going through Poland it seems 
) so I’m 99.9% sure that my traceroute went though cellular.
Checking DNS
A final check is that DNS resolutions work (mapping host name to IP addresses): curl sgdata.motus.org
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://sgdata.motus.org">here</a>.</p>
<hr>
<address>Apache/2.4.10 (Debian) Server at sgdata.motus.org Port 80</address>
</body></html>
The key here is that no name resolution error occurred. The result is a web page telling the browser that it should retry using https, which means we connected to the server.
ModemManager commands
The modem is managed by software called ModemManager. To list the modems it knows about: mmcli -L
/org/freedesktop/ModemManager1/Modem/3 [Telit] LE910C4-NF
This shows that it’s seeing the modem and that it’s modem #3, which is needed to get more details.
To get details about the modem: mmcli -m 3 where 3 is the modem number from above
Output
----------------------------------
General | path: /org/freedesktop/ModemManager1/Modem/3
| device id: 69b580f5cb29c45e584c21118098565c831ab3a6
----------------------------------
Hardware | manufacturer: Telit
| model: LE910C4-NF
| firmware revision: 25.21.660 1 [Mar 04 2021 12:00:00]
| carrier config: default
| h/w revision: 1.10
| supported: gsm-umts, lte
| current: gsm-umts, lte
| equipment id: 356000000020091
----------------------------------
System | device: /sys/devices/platform/scb/fd500000.pcie/pci0000:
00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4
| drivers: qmi_wwan, option
| plugin: telit
| primary port: cdc-wdm0
| ports: cdc-wdm0 (qmi), ttyUSB1 (ignored), ttyUSB3 (gps)
,
| ttyUSB7 (ignored), wwan0 (net)
----------------------------------
Status | lock: sim-pin2
| unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (3), sim-puk
2 (10)
| state: registered
| power state: on
| access tech: lte
| signal quality: 89% (recent)
----------------------------------
Modes | supported: allowed: 3g; preferred: none
| allowed: 4g; preferred: none
| allowed: 3g, 4g; preferred: 4g
| allowed: 3g, 4g; preferred: 3g
| current: allowed: 3g, 4g; preferred: 4g
----------------------------------
Bands | supported: utran-4, utran-5, utran-2, eutran-2, eutran-4, e
utran-5,
| eutran-12, eutran-13, eutran-14, eutran-66, eutr
an-71
| current: utran-4, utran-5, utran-2, eutran-2, eutran-4, e
utran-5,
| eutran-12, eutran-13, eutran-14, eutran-66, eutr
an-71
----------------------------------
IP | supported: ipv4, ipv6, ipv4v6
----------------------------------
3GPP | imei: 356000000020001
| enabled locks: fixed-dialing
| operator id: 310260
| operator name: T-Mobile
| registration: roaming
| packet service state: attached
----------------------------------
3GPP EPS | initial bearer path: /org/freedesktop/ModemManager1/Bearer/6
| initial bearer apn: plus
| initial bearer ip type: ipv4
----------------------------------
SIM | primary sim path: /org/freedesktop/ModemManager1/SIM/3
| sim slot paths: slot 1: /org/freedesktop/ModemManager1/SIM/3 (active)
| slot 2: none
----------------------------------
Bearer | paths: /org/freedesktop/ModemManager1/Bearer/7
Some key lines:
- Status > state is the field shown in the UI
- Status > access tech shows that I have an LTE connection
- Status > Signal quality may be of interest
- 3GPP > operator name says that the modem is connected to T-Mobile
- 3GPP > packet service state says that a packet (data) connection is established, so data ought to flow
- SIM > primary sim path shows that the SIM is known as #3 to modem manager
- Bearer shows that a connection (bearer) for the data is established and is known to modem managers as bearer #7
To get details about the SIM (useful if some PIN is required, which the SG software doesn’t support): mmcli -i 3 where 3 is the SIM number from above.
To get details about the bearer (data connection): mmcli -b 7 where 7 is from above
--------------------------------------
General | path: /org/freedesktop/ModemManager1/Bearer/7
| type: default
--------------------------------------
Status | connected: no
| connection error name: org.freedesktop.ModemManager1.Error.MobileEquipment.Unknown
| connection error message: Call failed: cm error: emm-detached
| suspended: no
| multiplexed: no
| ip timeout: 20
--------------------------------------
Properties | apn: plus
| roaming: allowed
| ip type: ipv4
--------------------------------------
Statistics | start date: 2024-05-23T15:43:03Z
| duration: 5731
| bytes rx: 72027
| bytes tx: 424526
| attempts: 1
| total-duration: 5731
| total-bytes rx: 72027
| total-bytes tx: 424526
Of interest here is that an error is shown, which matches the fact that my modem dropped down from being connected to just being registered. It’s possible to look up emm-detached but the likely result is more technical mumbo jumbo… Also of interest are the stats because they show that quite some data was transmitted in both directions, so the connection did actually work at one point.
The big gun to either fix issues (like my modem that fell back to registered from connected) or to mess thing up completely is to reset the modem: sudo mmcli -m 3 --reset
It’s best to watch the web GUI as the state changes. It will take a minute or so to settle back into a new state. This is a good place to mention that modems can be very slow. For starters the SIM can contain an application that determines which towers the modem should or should not connect to. The Twilio global SIMs can cycle through multiple configurations if they don’t know which part of the world they’re in, and it takes minutes to try each one (an APN of super means you have one of those SIMs). The modem can also cycle through multiple carriers because the home carrier may not work but roaming through another carrier may. Again, this takes minutes. The feedback from the modem is minimal although you should see the state cycle in the web UI. Information on why it gave up on a particular attempt is not available or indecipherable… The only thing handy is that aspirin 
