FUNcube sample rate dropping and disconnecting - FIXED

TL;DR: Two issues affecting FUNcube stability have been identified and are corrected in the latest SensorStation update. Any V2 and V3 SensorStation running more than 3 FUNcubes should be updated to increase stability.

Dear Researchers,

CTT has been working with Birds Canada after identifying some cases where FUNcube sample rates were dropping below the expected 48kHz. Additionally, there are some cases where FUNcubes drop out entirely, and do not restart. These are not limited to SensorStations, but does appear affect newer SensorStation V3 boards more often.

The root cause is a limitation in the Raspberry Pi chipset that limits the isochronous data transfers that are used by FUNcubes, which are USB1.2 devices. Various combinations of USB hubs and devices (and possibly detection order) can affect the bandwidth ultimately available. Shutting down unused devices, such as the cell modem or WiFi frees up bandwidth for the FUNcubes. Note that RTL-SDRs are USB2 devices and do not use isochronous transfers, as a result they are not affected by these bandwidth limitations however they are likely to affect the bandwidth available to FUNcubes in a mixed system. After lots of testing, and various updates, we have pushed a public update that addresses both issues.

We recommend anyone with undeployed V3 stations (or V2 stations where you have flashed the Pi with the latest disk image), or visiting stations with internet connectivity, run a system update from the SensorStation interface. This will pull the latest updates which address two known issues:

  1. When more than 3 Funcube Dongles (FCD) are installed, and both the Cellular Modem and WiFi module are installed, the frame rate on all FCDs drops.

  2. The patch adds new user settings in the SensorStation Interface GUI that allow the user to shut off either or both of the Cell Modem and WiFi module. Users running 4 or more FCDs choose one of the two features to shut off in order to maintain full sample rates. Regardless of whether you are actively using the Cell Modem or WiFi module, it must be toggled OFF via the interface in order to restore the sample rate.

  3. FCDs drop out and do not restart without a system reboot.

  4. This patch introduces some new code under the hood which restarts FCDs that are dropped or where frame rates drop too low. No user input is required for this to happen.

If you have any questions please don’t hesitate to reach out to support@celltracktech.com. A special thanks to Birds Canada and Thorsten von Eicken for helping with Sensorgnome code and testing each patch.

1 Like

FYI for Sensorgnome V2 users: the same issue affects Raspberry Pi based Sensorgnomes. The SensorGnome V2 software has similar patches (more below). Due to the diverse variety of HW it’s a bit more difficult to characterize which combinations cause issues. I now have 4 FCDs to test with thanks to Birds Canada (!) and hope to do some more testing soon. Some known issues:

  • the raspberry Pi USB subsystem can provide at most 1.2A of power (total for all 4 ports), thus any configuration that exceeds this needs an externally powered hub (using a non-HAT cell modem certainly does), a lack of power can cause FCDs to “drop”
  • the raspberry Pi root hub (which is what provides the 4 physical ports on the device itself) is “single-TT” and not “multi-TT” (more below) and can only support one directly plugged in FCD, so in general, all FCDs should be plugged into an external multi-TT hub

Sensorgnome V2 releases

Currently: Sensorgnome V2 RC14 (2024-) detects and logs dropped and low-rate FCDs, and also attempts to restart them, but the restart is not reliable. I still need to implement the more robust restart we worked out for the SS (that’s an oversight of mine).

Release candidates RC13 and RC14 both measure the FCD and RTL-SDR sample rates and the data can be viewed in a chart on the radio tab of the Web UI as well as on SGhub (www.sensorgnome.net).

Single-TT vs Multi-TT hubs

FCDs are USB1 devices, they run very slow. Once USB2 came out hubs started to translate USB1 transactions to USB2 speeds so they don’t hog the bus for so long. A single-TT hub can do this for one device only. A multi-TT hub can do this for multiple devices (typ all its ports). Thus connecting 4x FCDs to a single-TT hub accelerates one and the other three continue to hog the bus at USB1 speeds. A multi-TT hub would accelerate all 4.

Raspberry Pi4B

The rPi 4 root hub is a Single-TT hub, so plugging FCDs directly into the Pi is not a good idea. When an external hub is plugged in it is possible to determine whether it is multi-TT by using the command-line (sorry) and issuing the command

sudo lsusb -v | grep TT

look for the bInterfaceProtocol lines (and ignore errors and other stuff). For example, my SG has one 4-port multi-TT hub:

      bInterfaceProtocol      1 Single TT
      bInterfaceProtocol      2 TT per port

The Single-TT is the root hub and the TT per port is the multi-TT hub. Since the rPi itself doesn’t have a multi-TT hub the presence of TT per port indicates that the added hub is indeed multi-TT.

Raspberry Pi3B+

Undetermined…

Raspberry Pi3B (V1.2)

The hub of the rPi3B is a Multi-TT hub, thus plugging FCDs directly into it is a good idea. The full story is that the root hub is Single-TT and has only one device on it, which is a fast-ethernet plus 4-port hub combo chip. This combo chip has a Multi-TT hub. All this makes sorting out whether an external hub is also Multi-TT a bit more difficult. When plugging in an external hub I recommend to verify that it is Multi-TT using the following command:

sudo lsusb -v | grep -E 'TT|Bus'

On an rPi3B without any external hub this shows:

Bus 001 Device 028: ID 04d8:fb31 Microchip Technology, Inc. FUNcube Dongle V2.034
      (Bus Powered)
Bus 001 Device 003: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub
  bDeviceProtocol         2 TT per port
      bInterfaceProtocol      1 Single TT
      bInterfaceProtocol      2 TT per port
    TT think time 8 FS bits
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  bDeviceProtocol         1 Single TT
    TT think time 8 FS bits

The “Bus” lines show the devices found and the bDeviceProtocol the Transaction Translators. The SMC9514 hub is what matters (and I don’t know why it also lists a Single-TT something). An external hub should show in addition to the above lines.

Notes

  • USB3 is completely irrelevant: a USB3 hub internally consists of a USB3 hub plus a USB2 hub, the transfers in question here do not touch the USB3 side, it’s as if it didn’t exist (and the standard prohibits translation from USB1/2 to USB3).
  • the majority of 7-port hubs consist of 2 stacked 4-port hubs, which is fine, but useful to know if you’re looking at lsusb output
  • Multi-TT is not something that is normally listed in hub specs and more recent hubs don’t seem to have a higher chance of being multi-TT than older ones, at least not in my sampling of cheap hubs from Amazon

Acknowledgments and caveat

A lot of people contributed to chasing down the issue, in particular Josh Sayers, Adam Smith, Matt Webb, Garret Rhyne, Nick Haddad, Bob Fogg, David La Puma and Kylle Lamoree (I hope I’m not forgetting someone). The symptoms depend heavily on how the hardware is configured and it feels like there are still some ingredients that elude us. The team set-up more than a dozen systems, some with up to 6 FCDs, and swapped hardware and software pieces to try and narrow down the issue. Sometimes everything looks fine but then errors show up a couple of days later: it has been quite frustrating at times! I still have the feeling that while the fixes are real improvements we haven’t quite seen the end of this. Thank you for all the hard work and to CTT for jumping in!

NB: I also want to add that despite a multi-page write-up with details of the issues and examples we got basically zero support from the creator of the FUNcube, even though Motus has resulted in hundreds of FCD sales. I found that quite shameful and it has contributed to me wanting to figure out how to make RTL-SDRs as effective as FCDs.

Recommendations for Raspberry Pi

This post is preliminary, I have not tested all the rPi4 permutations yet and have not tested a 3B+. My initial version of this post confused 3B and 3B+ :roll_eyes: :grimacing:

This post summarizes how to connect multiple FUNcube dongles to Raspberry Pi’s. If you will deploy many FCDs a Pi3B is recommended over a Pi4B because it has a Multi-TT hub for its 4 ports while the Pi4B has a Single-TT hub (see previous post for details).

The bottom line: each FCD needs a TT (Transaction Translator, i.e., USB1.1->USB2 accelerator). A Single-TT hub can thus support one FCD, a Multi-TT hub can support one on each of its ports.

Recommendations

  • the total number of FCDs probably shouldn’t exceed 4, at least it’s untested territory
  • rPi3B only: connect 4x FCDs to all four ports (but then no ports are left for CTT radios, GPS, cell modem, etc)
  • rPi3B only: connect 3x FCDs to three ports and a hub to the 4th port
  • rPi4B: connect one FCD to one of its ports, that will use the rPi’s single TT
  • if a Multi-TT hub is connected each of its ports can have an FCD, this the best especially for rPi4B, but Multi-TT hubs are not easy to find anymore
  • if a 4-port Single-TT hub is connected then it can have one FCD (plus other devices)
  • if a 7-port Single-TT hub is connected it most likely can have two FCDs (plus other devices) if the hub consists of two stacked 4-port hubs and the two FCDs are plugged one into the “upper” hub and the other into the “lower” hub
  • the USB ports on the rPi can support 1.2A total, 4x FCDs use ~700mA, 1x RTL-SDR appears to use ~200mA so a powered hub may be required depending on configuration

I have successfully plugged 4x FCDs + 1x RTL-SDR + 1x CTT radio into an rPi3B, including 2xFCDs + RTL-SDR + CTT radio into a 7-port Single-TT hub.

Tools

  • the Sensorgnome V2 software’s radio tab shows the sample rate: it needs to remain pegged to 48kHz for all FCDs & RTL-SDRs
  • the port mapping pane shows the “port path” where each level corresponds to a hub, so 1.2 is port 2 of the root hub (1), 1.5.3 is port 3 of a hub in port 5 of the root hub, and 1.5.4.2 means there are two stacked hubs (e.g. 7-port hub)

I made a mistake: my rPi 3B is an older “3B (V1.2)” and not a “3B+”. Thank you to Josh for pointing it out. I updated the posts above and am ordering a 3B+ for testing… :grimacing: