Repository navigation
Feature Request - FT8 logging to remote curl #25
Description
Activity
I think rsyslog is a good idea. I will go to add this. And expert can configure this easily.
I also receive a similar request to ask to pump the spots to a remote network URL. I want to check if this is a better option or syslog gives more flexibility?
- pinned this issue
on Nov 14, 2024 web-888 is a great product. Any of the solution would be perfect.
The syslog have benefit, due the fact that logging facility level could be adjusted.
So I can log "PSKReporter spot" to IP_A and logs related to "ADMIN connection closed" to sec. log server IP_B.
One integration place for all plugins, if they use system log feature.On the other hand sending it as http curl or something, will be better for public post processing using IOT sw like node-red, etc. Just simple processing of http get/post request could be parsed quickly.
The report for pskreporter send currently batches of reports in one bundle to report.pskreporter.info using ip flow information exchange. The IP address of pskreporter is hard encoded, but I have seen some test address in source code for debug purpose. But for many people it would be hard to program just udp packet receiver for batch packets processing.
But probably dirty solution could be also make IP redirect on the router :).
Tcpdump/wireshark is also able to process this data, but it requires additional processing, since data are not nice to read. IMHOIn any case great product which really work nice.
Ps: Thx really for nice manual how to build firmware.Juraj
It would be great to have the ft8 data sent in the wsjt-x udp protocol as this is what numerous logging and support applications such as jtalert or gridtracker use for alerting on wanted spots.
There is a kiwisdr implementation of exactly this documented on this webpage
https://robertresearchradios.com/p/wsjt-x-and-digiskimmer-and-gridtracker-oh-my/
All the code is linked at the bottom of the page and is hosted on github.
A python library to generate or interpret wsjt udp packets is here:
- I looked at wsjt-x udp protocol. The protocol is designed for a single band. Every decoded message only carries the freq diff information. It will not be an idle protocol for web-888 to pump up all decoded messages out.…On Sat, Dec 14, 2024 at 6:24 PM mikerbts ***@***.***> wrote: It would be great to have the ft8 data sent in the wsjt-x udp protocol as this is what numerous logging and support applications such as jtalert or gridtracker use. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF3GRG5SPCEMKYNURASRZT2FQBMRAVCNFSM6AAAAABP6PTRT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBTGA2DINBVG4> . You are receiving this because you were assigned.Message ID: ***@***.***>-- -Howard
Hmmm that doesn't seem right. The protocol carries the full frequency they call "dial frequency" in the protocol.
The full protocol spec is outlined here. The info captured in the pskreporter output should carry enough info to pipe out over the protocol to satisfy the apps.
https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/Network/NetworkMessage.hpp
And here are some client encoder and server decoding examples bundled with the wsjt software
https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/UDPExamples/
There is also a python wsjt udp message encoder decoder located here :
https://github.com/kholia/py-wsjtx
This suggests the packet content could be as simple as:
(self.dial_frequency, self.rx_df, self.tx_df, self.dx_call, self.dx_grid, self.report)
WSJT-X supports output the frequency to udp server via message code=1. But the decode message is output via message code=2, which doesn't include a freq but only Delta frequency.
* Decode Out 2 quint32 * Id (unique key) utf8 * New bool * Time QTime * snr qint32 * Delta time (S) float (serialized as double) * Delta frequency (Hz) quint32 * Mode utf8 * Message utf8 * Low confidence bool * Off air boolI see what you mean. Could both packets just be sent for each decoded station or would that cause too much overhead?
Or not knowing how the decoder is called, could the udp code be called for each ft8 channel being decoded?
Does it decode one channel at a time? If so would that provide opportunity to get the dial frequency and add the freqeuncy delta for each decoded station as each channel is processed?
Appreciate you having looked into it. If it can't work i understand. I wish I was better at coding / programming.
- overhead is not a problem. but the client should handle this correctly. I may need to look at the tools you mentioned to see how they deal with the packets.…On Mon, Dec 16, 2024 at 3:07 AM mikerbts ***@***.***> wrote: I see what you mean. Could both packets just be sent for each decoded station or would that cause too much overhead? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF3GRCZLUTBDCPKAYJA7ND2FXHNPAVCNFSM6AAAAABP6PTRT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBUGAYDMNZWGQ> . You are receiving this because you were assigned.Message ID: ***@***.***>-- -HowardReacted by Stefan Junger
How about using MQTT to stream out the data as openwebrx?
I have a look on openwebrx implementation.
Base on inside, it would be a great solution.OpenWebRX can report received data to an MQTT broker. The data for each mode is sent under the "openwebrx/" topic. For example, FT8 data is reported as "openwebrx/FT8". A typical report will look as follows:
{ "mode": "FT8", "timestamp": 1722482895000, "freq": 7074985, "msg": "CQ YV6BXN FK60", "callsign": "YV6BXN", "locator": "FK60" }OpenWebRX+ uses MQTT to report SDR device changes and other internal events. For example, when an OpenWebRX+ server starts up or shuts down, it sends the following event under the "openwebrx/RX" topic:
{ "mode": "RX", "timestamp": 1722737506393, "version": "v1.2.64", "state": "ServerStarted" }OpenWebRX+ reports SDR device changes under the "openwebrx/RX" topic that can be used by various automation scripts. Thus, an SDR profile change will be reported as follows:
{ "mode": "RX", "timestamp": 1722483751451, "source_id": "rspdx", "source": "RSPdx", "profile_id": "air-1", "profile": "Air Band 118-120", "freq": 118900000, "samplerate": 2000000 }OpenWebRX+ reports user activities under the "openwebrx/CLIENT" topic. Thus, a user connecting or disconnecting from the server will be reported as follows:
{ "mode": "CLIENT", "timestamp": 1722738356255, "ip": "::ffff:101.23.45.67", "state": "Connected", "banned": false }I partially use http://mqtt.pskreporter.info for getting data about ft8, but I am not sure, how long it will work.
As we have new year in 2 days.
I want to wishing you a wonderful year ahead, filled with joy, success, and good health.
May 2025 bring you new adventures and unforgettable moments!Juraj/George
I just want to ask, if it would be possible to increase time reporting in ft8 logs to mili seconds ?
I want to play with TDOA and search some data point, where to start. FT8 seems as good field, since each signal is marked with time and location. I would like to be able to verify that locator provided in FT8 log match the TDOA.
Mon Feb 3 07:51:45 17d:23:53:42.845 ..234567..... 3 PSKReporter spot FT8 14076.838 PE1CPJ JO21 -10 1184km Mon Feb 3 07:51:30 2025 Mon Feb 3 07:51:45 17d:23:53:42.847 ..234567..... 3 PSKReporter spot FT8 14076.575 DL1YCK JN49 -10 888km Mon Feb 3 07:51:30 2025Current time stamp contains just seconds:
Feb 3 07:51:30 2025Also the information from PSK reporter is just in UNIX time:
{"sq":53332502268,"f":28076271,"md":"FT8","rp":-6,"t":1736607071,"sc":"DF1OLS","sl":"JO51en46sj","rc":"YO6SSW","rl":"KN25TP","sa":230,"ra":275,"b":"10m"} {"sq":53332506340,"f":28075744,"md":"FT8","rp":-4,"t":1736607071,"sc":"DF1OLS","sl":"JO51en46sj","rc":"HA4BF","rl":"JN97si38","sa":230,"ra":239,"b":"10m"} {"sq":53332514562,"f":28075763,"md":"FT8","rp":7,"t":1736607071,"sc":"DF1OLS","sl":"JO51en46sj","rc":"UT7UV","rl":"KO40vk77lk","sa":230,"ra":288,"b":"10m"}The radio signal would probably travel around the world in time less then 0.13 sec.
Maybe additional topic for me, could be to have a look on Doppler shift.
Similar when we receive signal from GEO satellite, programs try to calculate the right frequency shift in forecast.
I would like to have a deeper look on FT8 Doppler shift in HF transmissions.Thx a lot.
Thx a lot for adding syslog support.
Base on commit 1b0c70a
I have seen that logging is already in Master, so I compile it and give it a try. Works wonderfully.Additional changes on my side in ssh:
apk-add rsyslog vi /etc/rsyslog.conf #### Rules #### *.* @10.10.10.10:514 & ~To avoid to store logs locally and just enable remote udp logging.
/etc/init.d/rsyslog restartThx
Example from syslog server:
Date Time Level Host Name Category Program Messages 02/23/2025 22:13:16 Info web-888 user 00:18:10.608 ..23456789... 2 FT8 DECODE: 7074.616 G6ORT IO83 -14 1749km Sun Feb 23 22:13:00 2025 02/23/2025 22:13:02 Info web-888 user 00:17:56.383 ..23456789... 8 FT8 DECODE: 10137.934 JH0JLP PM85 -14 8689km Sun Feb 23 22:12:45 2025 02/23/2025 22:12:54 Info web-888 syslog -- MARK -- 02/23/2025 22:12:31 Info web-888 user 00:17:25.368 ..23456789... 4 FT8 DECODE: 18101.519 CT7AIU IM67 -13 2596km Sun Feb 23 22:12:15 2025 02/23/2025 22:12:01 Info web-888 user 00:16:55.282 ..23456789... 2 FT8 DECODE: 7075.119 M0XLR IO83 -11 1749km Sun Feb 23 22:11:45 2025 02/23/2025 22:11:46 Info web-888 user 00:16:40.912 ..23456789... 8 FT8 DECODE: 10137.731 EB7CFU IM77 -15 2459km Sun Feb 23 22:11:30 2025I just play a little and create an simplex app for visualize of ft8logs. Logs are send over syslog, then parsed and visualized using javascript. It is just POC, to have somethign on side:
https://github.com/jurajbujdoso/playWith-web888-ft8-logs
The nice think on syslog is, that I have 2 lines for sharing syslog data. One which send data to syslog server and the second line which send data just to the upper application. As I use UDP, there is no need that application is app all the time.
In latest update, I added feature to log ft8 spots into syslog. and it is possible to remote pump syslog data into other machine. partially, the requirement in this thread can be met. Love to hear the feedback.
I've installed the rsyslog and redirected it to other machine where the playWith-web888-ft8-logs runs and it's working great. Can I somehow make rsyslog and its config persist?
Not sure if that's only me, but my idea on using web-888 was to have it working 24/7 and maybe somehow notify me when there's a new DXCC received etc. Of course the user would have to provide ADIF logs. If that would be possible within web-888 then the dream would come true (-:
Thank you for great work
73
Jakub SQ2WB- 'lbu commit' Check this on Alpine Linux website for more details.…On Mon, Mar 31, 2025 at 12:57 AM gorzynsk ***@***.***> wrote: I've installed the rsyslog and redirected it to other machine where the playWith-web888-ft8-logs <https://github.com/jurajbujdoso/playWith-web888-ft8-logs> runs and it's working great. Can I somehow make rsyslog and its config persist? Not sure if that's only me, but my idea on using web-888 was to have it working 24/7 and maybe somehow notify me when there's a new DXCC received etc. Of course the user would have to provide ADIF logs. If that would be possible within web-888 then the dream would come true (-: Thank you for great work 73 Jakub SQ2WB — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF3GRAHPMEDAV7MALYW7QD2XAO6TAVCNFSM6AAAAABP6PTRT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGY2DMNRWG4> . You are receiving this because you were assigned.Message ID: ***@***.***> [image: gorzynsk]*gorzynsk* left a comment (RaspSDR/server#25) <#25 (comment)> I've installed the rsyslog and redirected it to other machine where the playWith-web888-ft8-logs <https://github.com/jurajbujdoso/playWith-web888-ft8-logs> runs and it's working great. Can I somehow make rsyslog and its config persist? Not sure if that's only me, but my idea on using web-888 was to have it working 24/7 and maybe somehow notify me when there's a new DXCC received etc. Of course the user would have to provide ADIF logs. If that would be possible within web-888 then the dream would come true (-: Thank you for great work 73 Jakub SQ2WB — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF3GRAHPMEDAV7MALYW7QD2XAO6TAVCNFSM6AAAAABP6PTRT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGY2DMNRWG4> . You are receiving this because you were assigned.Message ID: ***@***.***>-- -Howard
I have add adif support for received rsyslog logs. Hope it helps to interconnect with your apps.
Just change the location of the directory "$(pwd)/logs" where you want to create wsjtx_log.adi.#remove old container #create dir: mkdir logs docker run -d -p 5000:5000 -p 5140:5140/udp \ -e LimitTime=900 \ -e STATION_CALLSIGN=N0CALL \ -e MY_GRIDSQUARE=AA00aa \ -e ADIF_LOGS=Yes \ --name ft8-visual \ --restart unless-stopped \ -v $(pwd)/logs:/app/logs \ bujdo/ft8_visualisation_syslog_web888:010425Juraj om0ace
Wow. That's great, but... I proposed kind of reverse of that...
I have an ADIF log in WSJTX and would like to upload this log to Web-888 or play-with-web888-ft-logs so it knows which counties are already on my list. Then compare newly received stations with my log and notify me on new country being received so I switch on a PC and TRX to make a QSO.
I even found https://pyhamtools.readthedocs.io/en/latest/index.html
Wow. That's great, but... I proposed kind of reverse of that...
I have an ADIF log in WSJTX and would like to upload this log to Web-888 or play-with-web888-ft-logs so it knows which counties are already on my list. Then compare newly received stations with my log and notify me on new country being received so I switch on a PC and TRX to make a QSO.
This is what I was wanting. The WSJT UDP packets can feed into an app called GridTracker which shows currently heard stations you need via a criteria you set such as need states or dxcc, etc.
That protocol is well documented but I don't have the tech skills to implement having the syslog data converted and broadcast via that format over UDP.
Just FYI, I plan to add MQTT support to web-888 like openwebrx is doing. I believe it should also solve some usecases discussed in this thread.
It would be great to have it.
I have look on the internet, but I was not able to found some standards for MQTT sdr data format.Probably tools like gridtracker.org could benefits, when there would be some standard in the field.
Even aggregate data from multiple sdr using similar interface for all.
At the moment they have integration for wsjtx/jtdx but utilizing udp packets with some init section + data content. I try to create similar packet after transforming data from rsyslog, but I have not succeed yet.On psk-reporter there is already mqtt
http://mqtt.pskreporter.info/ and they have about 2 application which actively consume that format.Beside that, I have search for description of PINs for EXTio and I found out "lbu commit -d" in the wiki in hostname section.
[You have mention that rsyslog stream].So now I know, that is much more nicer way to install packages and save time. And not to search for own solution :)
It would may be good for others, to mention it explicitly somewhere, because it is little bit hard to find.
And I guess the linux lowers would like to have such option visible.Understanding how to integrate your own APK package to run persistently at boot would be a major step for anyone looking to develop a lightweight background daemon.
Juraj
Thank you for your suggestion. I will put together some information regarding apk and lbu stuffs into a document "Advanced Linux User Guide".
besides gridtracker, openwebrx+ has some definition as well. https://fms.komkon.org/OWRX/#HOW-MQTT
This is my current plan on mqtt messages. I am optimize for application which can filter topics via MQTT without parse all the events. Welcome suggestions and feedbacks.
- SDR events
SDR Events are mainly used for the important events from application. The current planned events are
Topic: websdr//
Payload: Json payload for the details.deviceid is uniqe for every device.
events: start/ stop / temp/ login/ logout
payload: timestamp, and additional fields for each event- FT8/WSPR events
Topic: websdr/deviceid/{ft8|wspr|ft4}/prefix/callsign/
Payload; JSON format, detailed about event like timestamp, callsign, target callsign, grid, etc. (To be finalized)Sorry for the late update.
Everything looks really good so far from your suggestions.In the meantime, I try to reach out on the Gridtracker2 team using Discord to ask if they could consider adding MQTT support, but I haven’t received a response for quite a while, so I’m not expecting much to change.
I still believe MQTT could be a great solution, especially since it allows hams to handle their data more independently. It would also be great if the data visualization could support multiple streams from several FT8 SDR devices at the same time. Currently, most visualization tools seem to only handle a single stream, which is limiting.
Receiving multiple streams could be very beneficial, as it would allow us to observe frequency shifts between different receiving stations in real time, as well as compare signal strength across various locations. I believe this could provide some really interesting insights.I also attempted 2. time to create a translation engine that grabs logs/JSON data from web-888 and converts it into a format that GridTracker2 can read via WSJT-X UDP, but unfortunately, I wasn’t successful—again.
Juraj
At the moment, I am not able to store or process FT8 received data on my own.
The file system is in RO mode. (So far this is good solution since sd card could gone with many writes.)
The other problem is that logs from FT8 does not have flag set to "L", which would mean, that they are stored on locall FS or rsyslog.
Currently reporting on the screen is done:
It would be great if, the same message from PSKReporte.cpp can be send to as remote [syslog /or https post request] to remote url.
Url or rsyslog IP could be defined in web config.
Thx a lot
Ps: Somewhere in kiwi forum , I have seen message that rsyslog redirect is available to wspr, but not ft8. But it should be placed in latest release. Probably having the solution to forward systems logs fully to remote rsyslog server would be best option for myself. Since it is most transparent solution.
Something like:
Or rsyslog: