In my previous post I showed a GNU Radio demodulator for the QO-100 multimedia beacon, which AMSAT-DL has recently started to broadcast through the QO-100 NB transponder, using a downlink frequency of 10489.995 MHz. This demodulator flowgraph could receive and save to disk the files transmitted by the beacon using the file receiver from gr-satellites. However, the performance was not so good, because it had a couple of ad-hoc Python blocks. Also, the real-time streaming data (which uses WebSockets) was not handled.
I have continued working in the decoder and solved these problems. Now we have a decoder with good performance that uses new C++ blocks that I have added to gr-satellites, and the streaming data is supported. I think that the only feature that isn’t supported yet is displaying the AMSAT bulletins in the qo100info.html web page (but the bulletins are received and saved to disk).
I have added the decoder and related tools to the examples folder of gr-satellites, so that other people can set this up more easily. In this post I summarise this work.
8APSK Costas loop
I have added a C++ block called 8APSK Costas loop to gr-satellites. This follows the structure of the GNU Radio Costas loop, but uses the phase detector that I introduced in the previous post. The immediate advantage of this, besides the performance increase, is that it uses the control_loop class from GNU Radio, which figures out the appropriate coefficients for critical damping.
As expected, the performance of the C++ block is much better than the Python block I had earlier, specially because the loop (both in Python and C++) needs to work sample by sample in a for-loop, which in Python has significant overhead.
Fixed Length to PDU block
Another point of the decoder where the performance could be improved was how PDUs of fixed length are extracted from the stream of symbols starting at the location of each syncword. This was done with the gr-satellites “Sync and create packed PDU” block.
Under the scenes, “Sync and create packed PDU” is a hierarchical flowgraph that is essentially composed of the following blocks:
- The in-tree Correlate Access Code – Tag block, which finds the location of the syncwords and inserts a tag at the end of each syncword.
- The gr-satellites Fixed Length Packet Tagger block. This transforms the output of Correlate Access Code – Tag into a tagged stream that only contains the packets of a certain fixed length that start at each of the tags. Overlapping packets are allowed (which is useful, because syncword detections are never 100% certain). When packets overlap, their items are duplicated in the output of this block.
- The in-tree Tagged Stream to PDU block.
The Fixed Length Packet Tagger block, which is one of the oldest blocks in gr-satellites is implemented in Python, so its performance is not so good. Moreover, the implementation is not so straightforward, since handling tagged streams and arbitrary input to output ratios is tricky. In 2021 I tried to do a direct C++ translation of this block. However, I quickly reverted this, since the performance of the C++ version was even worse than that of the Python block. I believe that my main mistake was using std::deque to replace a Python deque.
The performance of Sync and Create packed PDU is quite important for the QO-100 decoder because there are 7 of these blocks running in parallel at 7200 bits per second each. Now I had an idea about how to improve Sync and Create packed PDU by getting rid of Fixed Length Packet Tagger.
At some point I had studied the code of the (now in-tree) Tags to PDU block, back when it was only in the gr-pdu_utils OOT module. The one thing I didn’t like about this block is that it doesn’t support overlapping packets. When it starts processing a packet, it will ignore syncword tags until the packet has finished. For me, allowing overlapping packets is important, because I often have false syncword detections due to setting the detection thresholds relatively low. If overlapping packets are not allowed, the false syncwords could cause a good packet to be missed.
Still, the Tags to PDU block gave me the idea that outputting to a PDU was easier than outputting to a tagged stream. Therefore, I decided to do a new C++ block, which I have called Fixed Length to PDU. This outputs PDUs of a certain fixed length whose data starts at the location of each tag in the input stream. In a sense, it is like the combination of Fixed Length Packet Tagger and Tagged Stream to PDU, but doesn’t deal with tagged streams at all.
The block works in the following way. Let us denote by N the fixed packet size. The block owns a buffer of N-1 items that is used as a circular buffer to store history (in the sense of the history() of a GNU Radio block). This is done instead of using the block history to avoid limitations regarding the GNU Radio buffer sizes (which are problematic for large packet sizes).
In each call to the work function, all the locations of the tags that are placed in input items of this work function are saved by appending them to a std::list. Then, this list of locations (which potentially contains locations saved in earlier calls to the work function) is scanned to find for which locations (which mark the start of a packet) the packet ends at some item inside the current input buffer. For each of these locations, the corresponding packet is reconstructed into a buffer by copying the data from the history buffer (at most two disjoint pieces need to be memcpy()ed), and from the input buffer (a single piece needs to be copied). Then the packet is sent out as a PDU.
The reason for storing the locations of the tags in a std::list is that I am not certain if we can get tags that we have seen many work function calls ago by calling get_tags_in_range() (i.e., I don’t know when and how tags are destroyed). Hence, as a precaution, I’m only asking for tags corresponding to the current input buffer and saving them for later. The block could be optimized further by ensuring that the std::list is ordered. This could be easy if get_tags_in_range() already returns the tags ordered by location, but I don’t know if this is true (the documentation doesn’t mention this).
Additionally, since often we want to pack 8 bits per byte in the output PDU, the Fixed Length to PDU block supports this as an option.
The new Fixed Length to PDU block is working quite well in terms of performance, and it also seems that it doesn’t fail in corner cases. Therefore, I have modified all the Sync and create PDU blocks (there are 3 such related blocks) in gr-satellites. I think this is an important performance improvement for gr-satellites, because these blocks are used in most of the decoders.
As a collateral effect of adding this C++ block, I have now decided to maintain different code bases of gr-satellites for GNU Radio 3.10 and 3.9. Until now, I used the same code base to simplify the maintenance, because the changes from 3.9 to 3.10 were small enough that could be handled with some Python code (a few things have been moved to other modules).
However, the Fixed Length to PDU block uses the vector_type type to define the item type (byte, short, int, float, complex), in the same way as Tagged Stream to PDU does. This type has been moved from gr-blocks to gr-runtime in GNU Radio 3.10 (and its namespace has changed as a consequence). I don’t see how to add some code to get around this, since the version of GNU Radio doesn’t seem to be directly available to the C++ preprocessor (say, as a #define in some header file).
Splitting the code bases for 3.9 and 3.10 also has some advantages. For instance, I have been able to merge Clayton Smith’s pull request that modernizes the logging and removes boost (these changes weren’t backwards compatible with GNU Radio 3.9).
Scrambler
As I mentioned in the previous post, the scrambler used by the QO-100 multimedia beacon is a synchronous (additive) scrambler that is defined by a sequence given in scrambler.cpp. I considered the idea that this sequence was the output of a suitable LFSR, in which case the scrambler could be implemented with the GNU Radio Additive Scrambler block.
However, it seems that the sequence is not the output of an LFSR. Indeed, if we look at the output sequence \(s\) of an LFSR with an \(n\)-bit register, then the subsequences of \(n\) consecutive elements of \(s\) will be all the \(2^n-1\) possible non-zero vectors of \(n\) bits.
We can take advantage of this fact to find the value of \(n\) given \(s\). For each \(k = 1,2,\ldots\) we form the vectors of \(k\) adjacent elements of \(s\) and count how many distinct vectors appear. For \(k < n\) we will have \(2^k\) distinct vectors, while for \(k = n\) we will only get \(2^k – 1\) vectors.
In the case of the QO-100 multimedia beacon scrambler, we get \(2^k\) vectors for \(k \leq 7\), but for \(k = 8\) the count drops to 248, which is much less than \(2^8-1 = 255\). Therefore, the scrambler sequence is not the output of an LFSR. I don’t know if the sequence was generated using another algorithm or if it was simply chosen at random.
I have implemented a GNU Radio C++ block that gets a list of bytes as a parameter and uses that list as a scrambling sequence. The block is called “PDU Scrambler”, since it works on PDUs. Using this block, the appropriate scrambler sequence is simply hardcoded in the flowgraph.
WebSocket server
Probably the most interesting feature of the multimedia beacon is the streaming data, which is displayed in real-time in an HTML page in a web browser. As I mentioned in the previous post, this uses WebSockets. One of the files sent by the beacon is the page qo100info.html. This has some JavaScript that connects to a WebSocket server to get the streaming data and update the page. The beacon decoder should spawn a WebSocket server to feed the data.
There is no documentation about how the system works, but the implementation is easy to follow. The frames that carry streaming data use frame type 8 (see the previous post for a description of the frame structure). The first byte of the 219 byte payload identifies the type of data, as follows:
0. DX cluster messages1. NB transponder waterfall data2. CW skimmer3. WB transponder waterfall data
The payload of DX cluster messages and CW skimmer data is sent as is to the HTML page through the WebSocket server. The waterfall data is processed using a function like this before being sent to the server. At first glance, it might not be obvious what the function does, but on a closer look it is clear that it is repacking the 218 bytes of the payload (excluding the first byte) from 8 bits per byte to 6 bits per byte.
A pitfall that I found while implementing this is that there are 3 bytes prepended to the payload data when it arrives to the HTML page. Below I show how the beginning of the JavaScript code that gets the frame and checks to which data type it corresponds. The first byte of the payload (which is 0 for DX cluster) is now at arr[3].
websocket.onmessage = function (message)
{
var arr = new Uint8Array(message.data);
if(arr[3] == 0)
{
// ....
I don’t know where these 3 extra bytes come from, since they don’t seem to be produced in extdata.cpp. Probably they are inserted in the WebSocket server implementation. I have found that if I send these 3 bytes filled with zeros the HTML page works fine, as the JavaScript doesn’t seem to use them.
I have implemented a simple Python script using the websockets library that spawns a WebSocket server to which the HTML page can connect to, and in turn connects to a TCP server in the GNU Radio flowgraph to obtain the decoded PDUs. Since the websockets library uses async, I have decided to make a small script that is fully async (and uses asyncio for the TCP connection to the flowgraph) rather than trying to make a GNU Radio Python block with some async code.
The Python script is quite simple. For each connection, three async coroutines are run concurrently with asyncio.gather:
ws_recv_alive() reads continuously from the WebSocket to receive the “alive” messages that the HTML page sends. It does nothing with these messages, other than extracting them from the socket.ws_send_data()awaits on aQueueto get the messages that need to be sent to the HTML page, and sends them to the WebSocket after prepending three zeros.tcp_client()connects to a TCP client run by the GNU Radio flowgraph and awaits reading 221 byte PDUs. It checks the frame type and the first byte of the payload, repacks as 6 bits per byte the payloads corresponding to waterfall data, and puts the resulting payloads of the streaming data in theQueue.
Running the decoder
I have written a README file with some instructions about how to run the decoder and the WebSocket server. Once everything is running correctly, we should see a clean 8APSK constellation in the GUI of the decoder, and the HTML page updating with real-time data.
The only feature that isn’t implemented yet is displaying the AMSAT bulletins in the lower right part of the HTML page. These seem to be handled here and involve a WebSocket message that has the value 16 in its first byte (or actually in the fourth, if we count the three extra bytes), but I haven’t looked at the details.
Besides this, there could be some corner cases (such as starting the reception of a file mid-way or with severe packet loss) that might make the file receiver thread die. I’ll try to be more careful about these and fix them. Any reports of people using the decoder (whether it works well or not) are welcome.