Reverse engineering Outernet: modulation, coding and framing

Outernet is a company whose goal is to ease worldwide access to internet content. They aim to provide a downlink of selected internet content via geostationary satellites. Currently, they provide data streams from three Inmarsat satellites on the L-band (roughly around 1.5GHz). This gives them almost worldwide coverage. The downlink bitrate is about 2kbps or 20MB of content per day.

The downlink is used to stream files, mostly of educational or informational content, and recently it also streams some APRS data. As this is a new radio technology to play with, it is starting to get the attention of some Amateur Radio operators and other tech-savvy people.

Most of the Outernet software is open-source, except for some key parts of the receiver, which are closed-source and distributed as freeware binaries only. The details of the format of the signal are not publicly known, so the only way to receive the content is to use the Outernet closed-source binaries. Why Outernet has decided to do this escapes me. I find that this is contrary to the principles of broadcasting internet content. The protocol specifications should be public. Also, as an Amateur Radio operator, I find that it is not acceptable to work with a black box receiver of which I can’t know what kind of signal receives and how it does it. Indeed, the Amateur Radio spirit is quite related in some aspects to the Free Software movement philosophy.

For this reason, I have decided to reverse engineer the Outernet signal and protocol with the goal of publishing the details and building an open-source receiver. During the last few days, I’ve managed to reverse engineer all the specifications of the modulation, coding and framing. I’ve being posting all the development updates to my Twitter account. I’ve built a GNUradio Outernet receiver that is able to get Outernet frames from the L-band signal. The protocols used in these frames is still unknown, so there is still much reverse engineering work to do.

The only two closed-source pieces of the Outernet software are called sdr100 and ondd. sdr100 is the receiver. It uses an SDR dongle to receive the L-band signal and decode Outernet frames. Then it passes the Outernet frames to ondd, which is the daemon in charge of doing something useful with the frames. Its main job is to reconstruct and decompress the files that are being streamed.

In particular, sdr100 is a bit polemical because it seems to me that it violates the GPL licence, as it links librltsdr and libmirisdr, which are GPL (not LGPL) software. I’ve tried to write to the Outernet developers, but they don’t seem to care.

In any case, my GNUradio Outernet receiver is now able to substitute sdr100 and send Outernet frames to ondd. I’m starting to do this to reverse engineer the protocols used in the frames, as the goal is to replace ondd as well (or at least come up with some open-source software that does something useful with the Outernet frames).

In my reverse engineering effort, the help of Scott Chapman K4KDR and Balint Seeber has being invaluable. Scott has being making SDR recordings for me of the Outernet signal, as I don’t have an Outernet receiver. The work of Balint has being very inspirational for me, in particular, his slides about blind signal analysis and his Auto FEC GNUradio block from gr-baz.

The first thing we note when reverse engineering the signal is that it is a bit more than 4kHz wide. We know that it is probably a PSK signal of some sort, but BPSK and QPSK are both good candidates. To see which type of PSK it is, we study the powers of the signal. We raise the complex baseband PSK signal to the power 2 and observe if there is a large DC component in the signal we get. In this case, there is, so the signal is BPSK. If there wasn’t, we would raise the signal to the power 4, where a DC component would indicate QPSK, and so on in order to detect higher order PSK. This trick works because of the symmetry of the PSK constellations. The BPSK symbols are 180º apart (i.e., they are oposite) while the QPSK symbols are 90º apart (i.e., they are related by the 4 roots of unity of order 4). Thus, when raising to the power 2 the BPSK symbols they become the same symbol, so the resulting signal has DC. Similarly, when raising to the power 4 a QPSK signal, the four symbols collapse.

Next, we use cyclostationary analysis to deduce the symbol rate. We multiply the signal by the complex conjugate of the signal delayed one sample. The resulting signal will have a strong DC component. The next strong frequency component will be at a frequency equal to the symbol rate. This works because the signal and the delayed signal are more or less equal most of the time, since both samples are still in the same symbol, thus we get 1 most of time. However, when the signal just jumps from one symbol to a different symbol, the signal and the delayed signal will be very different and we get something that is not 1. Some sort of spike. Thus, we get a spike every time a symbol change happens, so the resulting signal has a strong frequency component at the symbol rate. In the cyclostationary analysis for the Outernet signal I got a symbol rate of 4200baud, which is a nice round number and hence seemed to be right.

In hindsight, it was already known that the signal is 4200baud BPSK, but running these sort of tests allows us to confirm that the modulation has not changed recently.

Since we know that the signal is 4200baud BPSK but the data stream is only quoted to be around 2kbps by Outernet, we suspect that an r=1/2 FEC is in use, probably the usual r=1/2, k=7 convolutional code with CCSDS polynomials. Since this code admits some variations, the Auto FEC block from gr-baz is very useful, as it tries many combinations until a low bit error rate is achieved, to try to detect automatically the variation used. This block needs a patched version of GNUradio, because it is necessary to modify the Viterbi decoder to make it output bit error statistics. The patch given by Balint is for an older version of GNUradio, so I had to modify it. Here is a patch that works with GNUradio 3.7.10.1, in case anyone needs it.

The Auto FEC block works only with QPSK. I modified it to make it work with BPSK (only). Probably the best thing to do would be to make it support several PSK constellations. Here is the BPSK patch for Auto FEC.

The Auto FEC block found that the convolutional code used is the same that the “Decode CCSDS 27” GNUradio block expects but with the polynomial order swapped (first POLYB and then POLYA). Therefore, each pair of soft symbols needs to be swapped before the Viterbi decoder. As with any BPSK signal coded with an r=1/2 convolutional code, there is the ambiguity of how to make the pairs in the soft symbol stream. Thus, we run one swap + Viterbi chain on the soft symbol stream and another chain on the soft symbol stream delayed one symbol.

Using the patched Viterbi decoder, we can see that our Viterbi decoder is working because of the low bit error (which corresponds to a positive and almost constant output in the statistics output of the modified “Decode CCSDS 27” block). We suspect that we need to use a descrambler after the Viterbi decoder and a raster plot of the bitstream confirms it. We can try popular asynchronous descramblers to see if there is any luck and we get some structure in the raster plot. For instance, the polynomial used in G3RUH 9k6 packet radio is a good first choice. In this case, there wasn’t any luck.

Since I have the binaries for the Outernet closed-source receiver, there is another way to attack this problem: to reverse engineer the assembler code. I’m using the Linux x86_64 L-band receiver binaries. These are not the latest version, but they seem to work. The latest version is only available for Linux on ARM, since Outernet targets single board computers such as the Raspberry Pi 3 and the CHIP to be used as the receiver. They now advise to run their ARM software in a virtual machine if one wants to use a desktop computer. I disassembled sdr100 (this is done with objdump -D) and quickly found that the scrambler is implemented in a function called scrambler308, which is not very long. I translated the assembler code back to C. This is a slow process which requires concentration. The code I got was pretty close to the code that has finally made it into gr-outernet.

As you can see in the code, the descrambler has some sort of counter that gets reset sometimes. The counter also influences the output bit sometimes. This is something I hadn’t seen before, as I’m used to the multiplicative scramblers I’ve described in a previous post. Using “308” as a keyword for my search, I found out that this scrambler is called IESS-308 scrambler. It is described in an Intelsat document which is not publicly available. However, I managed to find a description of the scrambler in another document (see page 28). As you can see, the diagram in this document matches the C code obtained from descrambler308, except for the fact that descrambler308 inverts the output bit.

At this point, we have to worry about signal polarity. As you may know, when receiving a BPSK signal there is a phase ambiguity of 180º which translates to the fact that we have ambiguity on the signal polarity. We don’t know if we are receiving the original bitstream or an inverted version of it. Generally, differential coding is used to resolve this ambiguity, but when using a Viterbi soft decoder, the differential decoding is done after Viterbi decoding, just because the Viterbi decoder works on soft symbols, and the differential decoder can’t provide soft symbols.

The question now is whether differential decoding should come before or after the descrambler or whether it is used at all in the Outernet signal (there are other ways to resolve the polarity ambiguity). When thinking about this, it is good to know what happens with the various processing blocks if we feed in an inverted version of the signal we expect. It is well known that for a Viterbi decoder with the usual CCSDS polynomials we just get an inverted output (except for a few bits at the beginning of the stream). This is just because the CCSDS polynomials have an odd number of nonzero coefficients.

The IESS-308 descrambler has the same property, because the reset line for the counter is obtained by XORing an even number of bits in the stream, while the output depends on an odd number of bits in the stream. Thus, if we hook up the descrambler directly after the Viterbi decoder we still have a polarity ambiguity on the signal.

Figuring out the differential coding issue was probably the hardest part. It was a matter of blind trial and error. Most tries will yield some kind of noticeable structure on the raster plot of the output. This means that the descrambler is working and we’re on the right track.

By examining the assembler code of sdr100 we know that HDLC is used in some way for framing, as several of the names of the functions refer to HDLC. However, I didn’t manage to get any valid HDLC frames with my deframer from gr-kiss. I also reverse engineered the checksum functions of sdr100 just in case a different checksum was used. It turned out to be a table-based implementation of CRC16-CCITT, which is the checksum specified for HDLC, and bit-endianness was handled correctly. So, nothing unexpected here.

The solution turned out to be something really simple: no differential coding is used. Since there is an ambiguity on the polarity of the bitstream, an HDLC deframer is run both on the bitstream and on the inverted bitstream. One of these two deframers will successfully get the frames. Thus, there is a total of 4 HDLC deframers, since we also have 2 Viterbi decoders. With this decoder setup, I started to get correct HDLC frames from the Outernet signal.

To sum up, the specifications for modulation, coding and framing of the Outernet L-band signal are:

• 4200baud BPSK
• r=1/2, k=7 convolutional code with CCSDS polynomials (polynomials swapped)
• IESS-308 scrambler
• No differential coding
• HDLC framing

With these specifications, the decoder in gr-outernet is able to get Outernet frames from the L-band signal. I still don’t know much about the protocols used in the Outernet frames, since I need lots of them to try to detect some patterns, scan for plain text contents and so on. However, I have already noted a few things.

This is a typical Outernet frame:

pdu_length = 276
contents = 
0000: ff ff ff ff ff ff 00 30 18 c1 dc a8 8f ff 01 04 
0010: 3c 02 00 00 18 00 01 00 00 00 08 11 10 e5 21 4b 
0020: 48 2c e0 77 00 86 4d 14 06 3c 24 f7 30 e7 19 4c 
0030: ed 60 d4 44 94 6a 4a 18 34 ad b2 b5 92 01 b7 87 
0040: 06 ba 80 61 a5 87 06 80 f6 04 12 f6 d9 12 13 02 
0050: 64 0b 68 94 21 36 01 ab af 01 50 d0 13 4b dc b6 
0060: 92 90 6b f4 76 27 73 3d 91 f5 84 3d 75 d9 77 90 
0070: d2 74 15 49 66 e5 9a 57 df df 72 28 32 48 97 ed 
0080: 9a 46 6e 68 8e 72 b3 54 5f 52 ce f6 f5 de c1 fd 
0090: e4 e6 f8 a2 bd bb bb 65 cf 9e d0 ed 80 1e ad 8c 
00a0: 0c b8 59 28 41 cf 27 d3 cf a9 9e 28 06 8e c0 c8 
00b0: 42 7a bd ea da ae 7e 41 ee 24 c2 f9 28 b7 35 f6 
00c0: 8b 12 13 23 1f fb 0d 3e 32 49 b9 75 4b 31 d3 29 
00d0: 11 c1 48 a2 3b d4 8b 40 e6 2c 69 02 59 f2 f8 c8 
00e0: d2 ea aa ce 63 57 ed f7 25 42 8e 9b 21 d4 64 07 
00f0: 89 59 d0 47 d6 7b c7 3c c7 11 2c 91 d3 ca b1 52 
0100: ea ba be e3 00 39 fb be 6a 02 52 e3 8f ac ba 30 
0110: b7 d1 c2 3f

I expect that this contains a chunk of a compressed file, since this is most of the Outernet traffic. Almost all the frames are 276 bytes long. At a rate of almost 2100bps (you would have to take bit-stuffing into account), a frame takes about 1.05 seconds to transmit. This is pretty good. Each second you get a new packet if the signal is good enough for the Viterbi decoder to do its job. If the lock on the signal is lost momentarily, you only loose one second of data.

The thing that intrigues me most about the frames is that they look very much like Ethernet L2 frames. The destination MAC would be the broadcast MAC ff:ff:ff:ff:ff:ff,
which makes sense, and the source MAC would be 00:30:18:c1:dc:a8, which turns out to be a valid universally administered MAC address with an OUI assigned to Jetway Information Co., Ltd. There is some company called Jetway Computer which makes embedded computers for industrial applications, so perhaps this makes sense. However, the ethertype would be 0x8fff, which is not used for any standard protocol. I think that it is likely that this is the case: the frames are actual ethernet frames and the contents are some lightweight UDP-like protocol that rides just on top of ethernet (without an IP layer). I haven’t found a standard protocol that does such a thing and matches the structure of these packets. Probably Outernet has come up with some simple ad-hoc protocol. I don’t know why would anyone waste 5% of the bandwidth of an already low bitrate signal to send Ethernet headers (which are useless to the receiver), but it’s fun to see Ethernet frames being downlinked from geostationary orbit.

I’ve also being playing with injecting the few frames I have (Scott’s recordings were only around 1 minute long) into ondd to see if it does anything useful with them. ondd listens on a SOCK_SEQPACKET Unix socket at /tmp/run/ondd.data and sdr100 writes the frames it decodes to this socket. Unfortunately, socat doesn’t have good support for SOCK_SEQPACKET sockets, but it’s easy to write a little program that listens for the frames from the GNUradio decoder by UDP and writes them to the ondd socket. Here you have such a program. I’ll probably publish it in a better manner in the future. strace is a great tool to do this kind of research. That’s what I used to discover the type of socket that ondd uses and I also use it to keep an eye over ondd to see if it does something.

Using this technique I managed to spot two special packets:

pdu_length = 60
contents = 
0000: ff ff ff ff ff ff 00 30 18 c1 dc a8 8f ff 00 1c 
0010: 3c 00 00 00 81 00 00 18 01 04 6f 64 63 32 02 08 
0020: 00 00 00 00 57 f6 94 20 48 3a ca 8d 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00

pdu_length = 60
contents = 
0000: ff ff ff ff ff ff 00 30 18 c1 dc a8 8f ff 00 1c 
0010: 3c 00 00 00 81 00 00 18 01 04 6f 64 63 32 02 08 
0020: 00 00 00 00 57 fa a0 b0 11 56 ab ab 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00

As you can see, these packets are shorter than the regular packets. When ondd receives one of them, it tries to set the system clock (of course it fails, since I don’t run it as root). The timestamp it uses is 0x57f69420 for the first packet and 0x57faa0b0 for the second packet, which are at position 0x24 inside the packets. Moreover, these timestamps correspond to “Thu, 06 Oct 2016 18:12:48 GMT” and “Sun, 09 Oct 2016 19:55:28 GMT”, which match well the time that Scott’s recordings where made.

Therefore, it’s pretty clear what these packets are: they are just a time packet that is used to set the clock on the receivers. This is a very good idea, since the single board computers used as receivers do not have a real-time clock and of course using NTP is not an option. I expect that a time packet is transmitted every minute more or less.

By now, I don’t know what are the other 4 nonzero bytes that follow the timestamp. I don’t know why there are so many zero bytes either. The beginning of the packet is the same for both, but I expect that this is some sort of header that identifies the service, probably using some concept of ports.

So there you have it, someone could use all this information to build a fully functional Outernet clock. I hope that in the future we will know how to pull out more useful data from the frames.