It looks like Motus is waking up for the 2024 season and several things are happening on the Sensorgnome V2 software front so I thought I’d send a first status update around.
First some embarrassing not so good news: I made the rookie mistake of letting the encryption keys used to sign software packages for the Sensorgnomes expire. Long story, but the upshot is that existing SGs that run the V2 software can no longer be simply upgraded by pressing a button in the Web UI. What is needed is to first SSH into the SG and run the command (all one line):
sudo curl -L -o /etc/apt/trusted.gpg.d/sensorgnome.gpg https://sensorgnome.s3.amazonaws.com/sensorgnome.gpg
After that the upgrade via the UI will work as before…
The other more humorous not-so-good news is that I started to prepare a “V2.0” release just before the holidays but got side-tracked by other stuff, and at this point new feature requests have landed plus the above signature issue has come up and all this means that I’m going to do yet another “release candidate” version. Maybe, if the stars align, a V2 can follow “soon” thereafter… In any case, I’m not aware of any show-stopper issues with the software so from my point of view it’s safe to use.
What I currently have on the feature list for the next version is:
update to latest version of Raspberry Pi OS ‘bullseye’ (2023-12-05) and updated packages
get it out ASAP 'cause Adam wants to install a couple of SGs
add some troubleshooting code for the Adafruit GPS (it’s not working on some SGs and we don’t know whether it’s a software issue or power issue)
monitor the rate at which the FUNcubes are producing data and alert if that changes/drops
keep system logs for 60-120 days to facilitate troubleshooting SDcards recovered from non-connected SGs
report cellular info into sg-hub-agent so it can be monitored on sghub
I’ll send an update once the new version is ready. Feel free to reply here or contact me any time if you have comments, questions or suggestions.
Thanks for the update Thorsten. I ran the sudo command on our local SG and nothing evil happened or so far as i can tell. :) Unfortunately all the SG software was up to date so can’'t quite verify all is working as it should with the UI.
Looking forward to giving the new release a go. Will it support the CTT homebrew Adafruit feathers? Finally have three to try out, with the wire mod!
Mark, after the sudo command you still need to go through the web ui to upgrade. E.g., on the software tab, enable buttons, run check, then run sg-upgrade. Then when it’s done (or after 5 minutes) do a reboot.
Thorsten, indeed there was an update ready to go when i checked this morning. Update to 2024-035 worked perfect (as per the UI instructions and yours above), thanks! Lol, the hardest part was trying to remember my password for the Sensorgnome.
We have just bought new RPIs with the Sensorgnome-v2 installed. We do not have an Internet or cell modem installed on those. We have tested them with the Adafruit GPS (coax cable). We are able to connect to the web interface but the device is not recognized and we cannot get the time/date and GPS position (see screenshot attached).
Does this have to do with your previous message about the update of the software?
Thank you,
Camille
(I apologize if this message has been sent twice).
Hi Camille, you’re running a fairly old version of the Sensorgnome V2 software, please see https://www.sensorgnome.net/download and use the development build 2024-039 which should solve your issues. If it doesn’t, please connect the SG to the internet for ~24 hours so it uploads logs and let me know.
Regards,
Thorsten
Thanks a lot! We’ll try this and let you know how it goes.
Does CompuData has the latest version of SG-V2? We ordered/received the RPIs just a few months/weeks ago. I am surprised that a more recent version was not pre-installed considering that it does not recognize the Adafruit GPS.
Please do let me know if you continue to have issues!
CompuData is aware of the V2 firmware. They tried to test last year but ran into some issue and I suspect stopped working on it. Please ask them to load the most recent V2 image when you order ;-).
We have updated to V2 and we are now able to connect to the web interface and download the data.
Question - can we still download the data using Filezilla with SG-V2?
Hi Vincent, with V2 you can download the data directly from the web interface, once connected to the SG’s hotspot. Just go to the “files” tab. No need for FileZilla or any other software.
I’m attempting to upgrade many of my sensorgnomes to V2 software (2024-053), but I am having similar GPS issues. I’m hoping to deploy these receivers in a remote area without cell or internet connectivity and will need the GPS to work.
I’ve tried multiple power sources and running different GPS hats (Adafruit versions) but no changes. I receive the same no-dev. I’ve tried multiple reboots as well. Some of my other RPIs seem to have a much easier time working with the GPS. I’ve been conducting my testing at the field house with a clear line of sight to the sky. I have tested the archived version of the software (V1), and these units work in the same location and with the same power sources.
I will leave the SG-573ARPI3EDB9 running and connected to the internet for the next 24 hours or so to collect the software logs. Let me know if you have any thoughts.
Hi Bryant, I’m sorry you’re having issues with the GPS. I took a look at the logs (the SG seems to have “fallen off the network” ~15 hours ago?) and they’re perplexing. The operating system does not recognize that a HAT is plugged in so the SG software never tries to set it up. I haven’t seen this before.
The last boot I see in the logs was at 2pm UTC on March 11th, which is ~25 hours ago and ~24 hours before you sent your email to this group: was the Adafruit HAT plugged in at that time? Maybe that’s not the most recent boot and it just didn’t get time to upload everything (it goes slowly to save bandwidth) and thus my above comment is meaningless.
If the HAT was indeed plugged in, please confirm that it has the small EEPROM chip, which is a small 8-pin black device ~8mm behind the “adafruit” logo text. If it’s there and you can SSH into the SG the following command shows whether it has been recognized or not, this is from my rPi 3B+ with the Adafruit HAT:
gnome@SG-D011RPI31F2B ~> cat /proc/device-tree/hat/product
Ultimate GPS HAT
Yes, the Adafruit hat was plugged in for the duration of that time. I’ve set the unit back up with a new GPS hat and running into the same issues. The EEPROM chip is there and appear to be on all of the GPS hats I currently have. The GPS hat seems to be getting power as the red LED blinks occasionally.
I’ve attached the full logs in case it’s helpful.
I SSH’d into the gnome and ran that command and received:
Thanks for the confirmation. Dunno… The std operating system discovery of the HAT is clearly not doing what it should be doing.
I will load the exact same image on my rPi3 to see whether I can reproduce this here and I’ll search the rpi forums for similar symptoms.
The other thing I can do is prep a new version for you where you can manually override the HAT discovery, that ought to work. It’ll take me a day to produce that.
Thorsten
Update: I loaded the RC14 image onto my test rpi3B+ with Adafruit HAT and everything works as expected, so it’s not something readily reproducible. Looking at forums only brings up issues with defective HATs or users not fully seating the HAT or plugging the HAT in misaligned (1 pin off).
I saw in the code that I already prepared for forcing the HAT detection, this will get us one step further but the thing may just not work for other reasons. The blinking red LED only means that the GPS is getting power. Only trying will tell… To force the HAT detection please SSH into the SG and run:
sudo tee /etc/sensorgnome/force-hat <<<“Ultimate GPS HAT”
(Yes, that’s three “<” ).
Then reboot (e.g. “sudo reboot”).
Just a note: this is not some universal fix for GPS “no-dev” problems. This is the first time I see this particular issue. I implemented the force-hat feature a long time ago because some users were stacking two HATs and that is not supported by the rPi HAT detection so we needed a work-around.
Please let me know how this goes. Again, if you leave the SG connected I can check in the logs what happens…
Regards,
Thorsten
I’ve seen similar problems (Adadfruit GPS Hat not being recognised) on some of my test units. You should be able to see this in the logs of my receivers which have been on-line for a couple of week.
The fault is puzzling because the HAT is recognised when running the old V1 software on the same hardware.
A couple of observations:
I think most SG’s running the V1 software run on the Pi3A not the A+ that you used for testing. I recall a comment in the V1 development notes from John that due to some significant differences between the 3A and 3A+ on the Pi3A could be used with V1 SGs.
The Red LED on the AdaFruit HAT blinks at different rates that depend on the state of the GPS FIX. A fast blink indicating power ON but no GPS Fix, a slow blink indicating a GPS Fix. The HAT will get a fix irrespective of the state of the Pi., see https://www.adafruit.com/product/2324
I have observed that when my test V2 SG’s have not recognised the HAT, the LED is showing the HAT does have a Fix.
If it helps on one of my test devices I can try forcing the recognition of the HAT using the command above, or install the new software.All the best.
EAP
Not following the thread, but Ewan’s comment about 3A/A+ applied to 3B/B+. I never used the 3As so I’m not sure on that front, but the V1 SG software could not be used on Pi 3B+
Ewan, thanks for chiming in! The observation that you see the issue primarily with stations that have a GPS fix is an interesting one 'cause my test rpi sits indoors where it does not get a fix. I just moved it outside and am crossing fingers that I can reproduce the problem. I do not believe the 3B/3B+ makes a difference here, unless there is some subtle timing difference. At this point, who knows… I know that the problem is fairly wide-spread, sadly, or maybe that’s good 'cause it increases the chances of eventually narrowing it down.
The info I have so far:
when the GPS unit powers up it is reset to default settings and starts emitting NMEA stanzas at 9600 baud and starts acquiring a fix, this behavior is completely autonomous and cannot be altered by talking to the GPS (only by flashing a different firmware)
Adafruit left the device’s reset pin unconnected, only the serial RX/TX are connected, this means it’s impossible to reset/restart the GPS other than by power cycling, e.g. a reboot has exactly zero effect on the GPS
the non-recognized GPS units emit binary gibberish, switching through the supported baud rates doesn’t fix that and looking at the patterns I’m 99% sure that it continues at 9600 baud (if I can reproduce it here I can attach a scope and look at the timing to confirm)
at boot-up the old SG software listens to the serial port input at 9600 baud for up to a few minutes and if it sees what looks like GPS stanzas it starts the standard Linux GPS daemon (gpsd)
at boot-up the new SG software checks whether the operating system detected the Adafruit HAT (this happens over I2C, querying the little eeprom chip on the HAT) and if so it starts gpsd, there’s really no difference WRT what the old SG software does from the GPS unit’s point of view, neither sends any commands to the GPS, however, the version of gpsd is different
when gpsd starts it auto-detects the GPS unit and cycles through a handful of options before detecting the specific model in the Adafruit HAT, this does send commands to the GPS (which it mostly doesn’t recognize), I would be surprised if this auto-detection hadn’t changed since the version of gpsd used by the old firmware
there is no way to tell gpsd what type of GPS is attached so it skips the auto-detection
I’ve tried to send a malfunctioning GPS multiple reset commands at all supported baud rates to no avail
Something that might be interesting is to switch from the new SG software on a malfunctioning unit back to the old one without power cycling to see whether the old version manages to recover the GPS. I’m not sure how to do that (the rpi connector doesn’t have a reset pin) but I believe that having a second sd card ready, then issuing a reboot command and quickly switching sd cards will do it. I believe the bootloader sits there for a couple of seconds giving time to do the switch if prepared. It may also be possible to have an SSH session open and the reboot command typed out, then switch SD cards and quickly hit enter in SSH… This all hinges on having a unit that malfunctions in the first place.
Update on the GPS issues, there may be light at the end of the tunnel…
If you have a unit that typically malfunctions, i.e., has an Adafruit GPS HAT, shows “no-dev” using the Sensorgnome V2 software, and you are comfortable SSH-ing into the device, can you please run the following command:
sudo tee -a /boot/config.txt <<<“dtoverlay=disable-bt”
then reboot (e.g. “sudo reboot”). Once it comes up, the web UI should show the GPS functioning, e.g. showing “3d-fix”, or for units indoors “no-sat”. If that doesn’t do the trick, please power off the SG, wait a minute, and power on again, then check again. In any case, please email me the outcome and the sensorgnome ID. You can also just email me the ID after the reboot and I’ll check up on it (assuming it’s online).
Another update on the GPS issues. So far two users have reported that adding “dtoverlay=disable-bt” fixes the issues with the adafruit GPS. Both have asked “why?”. Good question…
First I thought that the GPS is the culprit and just outputs garbage every now and then. I attached an FTDI serial adapter, a logic analyzer, and a scope to the GPS output pin. All of them see perfectly fine stanzas when the rPi sees garbage.
Then I thought that some other process must mess with the serial port periodically, but I can’t see what it is. The bluetooth adapter is the main suspect 'cause it uses the other serial port and there are various settings to change which uart device is used by gpio vs bt: maybe I messed something up. But I can use bluetooth just fine while the GPS runs, if the two were interfering I’d see constant garbage while using BT and I don’t.
Then I noticed this in the kernel log:
[ +18.720172] hwmon hwmon1: Undervoltage detected!
[ +0.088741] ttyS ttyS0: 1 input overrun(s)
[Mar17 20:47] hwmon hwmon1: Voltage normalised
Ahem. I was using a dubious power adapter. So I hooked the Pi to a better bench power supply, feeding power into the GPIO connector instead of micro-USB and I adjusted to 5.1V. No errors! Turning the power supply down to 4.9V immediately brought a good stream of GPS serial input garbage and it kept going. The moment I turned the power supply back to 5.1V all the garbage input stops. I checked the GPS TX pin and it’s not outputting the garbage, it’s the Pi messing up internally. 5.0V still has errors but less frequently. (I use 50cm long AWG22 wires to feed the Pi and the GPIO power path differs slightly from the USB power path so I would take the absolute voltages with a grain of salt.)
What I don’t understand: the CPU chip is powered at 1.8V with 3.3V power for the I/O pads. Why would the voltage regulator care between 5.1V and 4.9V? It has an internal voltage drop of up to 0.5V at 1A, that’s a 3x headroom even at 4.9V. Go figure.
Something I noticed is that the GPS daemon (gpsd) changes state to “unknown device” when it encounters the garbage, and then it starts device detection from scratch. So if there is garbage every few seconds it never detects the device with the “no-dev” outcome and even if it does it looses it again.
Where does this leave us?
the “dtoverlay=disable-bt” fix swaps uarts, the other uart has more buffering and perhaps simply behaves better in undervoltage situations
the rPi’s are just completely intolerant of power that is not 100% up to spec (and I understand neither why is is so nor why is has to be so)
I need to add some code to detect the undervoltage messages and surface them in the web ui
I need to make yet another RC build with the fixes
