Simulating JT modes: how low can they get?

In this post I’ll show how one can use the signal generation tools in WSJT-X to do decoding simulations. This is nothing new, since the performance of the modes that WSJT-X offers has being thoroughly studied both with simulations and real off-air signals. However, these tools seem not very widely known amongst WSJT-X operators. Here I’ll give some examples of simulations for several JT modes. These can give the operators a hands-on experience of what the different modes can and cannot achieve.

Please note that when doing any sort of experiments, you should be careful before jumping to conclusions hastily. You should make sure that the tools you’re using are working as they should and also as you intend to (did you enter correctly all the parameters and settings?). Also, you should check that your results are reproducible and agree with the theory and other experiments.

Another warning: some of the software that I’ll be showing here, in particular the Franke-Taylor soft decoder for JT65 and the QRA64 mode, is still under development. The results that I show here may not reflect the optimal performance that the WSJT-X team aims to achieve in the final release version.

After all these warnings, let’s jump to study the modes. We’ll be considering the following modes: WSPR, JT9A, JT65A, JT65B and QRA64B. To give our tests some purpose, we want to find the decoding threshold for these different modes. This is the signal to noise ratio (SNR) below which the probability of a successful decode is too small to be useful (say, lower than 20%). For each mode, we will generate 100 test files containing a single signal with a fixed SNR. We will then see how many files can be successfully decoded for each SNR.

In these experiments I’m using the latest trunk build of WSJT-X, r7159. You shouldn’t use a development build unless you know what you’re doing. Unless you’re an experience user, it’s probably better that to do these experiments you use WSJT-X 1.7.0-rc1, or the final 1.7.0 version when it gets released. You can also use WSJT-X 1.6.0, but note that it doesn’t include the Franke-Taylor JT65 decoder (only the Berlekamp-Massey hard decoder, which is not suitable for EME or very weak signal work) nor the QRA64 mode.

For some reason, some of the tools don’t get installed when doing make install. Therefore I do the following, which is handy later. This particular path is where my build of WSJT-X is located. You should modify it for your own location.

export BUILD=~/wsjt/branches/wsjtx/build/

WSPR

The WSPR mode is a beacon mode for the HF bands. It is also somewhat usable on the VHF and UHF bands, but requires equipment with stable frequency references. The T/R period is 2 minutes, the FEC used is an \(r=1/2, k=32\) convolutional code and the modulation is 1.4684baud 4-FSK with a tone separation of 1.4684Hz. In each tone, the LSB is used is used for synchronization and the MSB is used for data. A WSPR message contains 50 data bits.

The tool to generate WSPR signals is called wsprsim. Unlike the other tools, it doesn’t generate .wav files, but rather .c2 files, which seem to be specific to WSPR signal processing. Here I generate 100 test files at a SNR of -30dB containing the WSPR message “M0HXM IO94 20”.

for file in {0..99}; do $BUILD/wsprsim -ds -30 -o wspr_$file.c2 "M0HXM IO94 20"; done

The tool used to decode WSPR signals is called wsprd. Here I run the tool on the 100 test files and collect the output of the decoder in the text file decodes for later analysis.

cat /dev/null > decodes; for file in wspr_*.c2; do wsprd  $file >> decodes; done

Note: When reading the signal reports in dB from wsprd or any other decoder, you should take them with a pinch of salt. Estimating the SNR of a weak signal is rather difficult. The SNR entered in the signal generator tools is the true SNR of the signal, but the SNR that the decoders report is at best an approximation.

The number of correct decodes can be calculated as follows. You should also examine the decodes for any possible false decodes. In all my experiments, no false decodes where produced, but it’s certainly possible to get them with any mode if you try with enough sample files. Operators should always be aware for false decodes when working in any of these modes.

grep "M0HXM IO94 20" decodes | wc -l

At -30dB SNR a total of 34 files where decoded successfully. Repeating the experiment for -31dB SNR only yielded 6 decodes. Therefore, we see that the decoding threshold of WSPR is around -30dB.

JT9A

The JT9A mode is designed to make minimal weak signal QSOs on the LF, MF and HF bands. It also works well under usual weak sporadic-E propagation conditions in the 6m band. With stable equipment it may also give good results for terrestrial work in the VHF and UHF bands, in the same manner as WSPR, although it’s definitely not the best mode for these bands.

The T/R period is 1 minute, as in the other JT modes except WSPR. The FEC is also a \(r=1/2, k=32\) convolutional code and the modulation is 1.736baud 9-FSK with a tone separation of 1.736Hz. One of the tones is used for synchronization and the other 8 tones are used for data (3 FEC bits are transmitted on each data tone). The synchronization tone appears in 16 symbol intervals (i.e., 19% of the time). A JT9 message contains 72 data bits.

The tool used to generate JT9 signals is called jt9sim. Here I generate a 100 test files at a SNR of -27dB containing the message “EA4GPZ M0HXM IO94”.

$BUILD/jt9sim "EA4GPZ M0HXM IO94" 0 1 1 -27 100

The decoder for JT9 (and also for JT65 and JT4) is called jt9. Here I try to decode the 100 files using a decoder depth 3. The decoder depth sets the timeout for the soft FEC decoder, which uses the Fano algorithm and can take exponentially long time. A depth of 3 sets the longest timeout possible and it is the setting that should generally be used, except in slow machines. Depth 3 is set in wsjtx by using the menu “Decode > Deep”.

jt9 -9 -d 3  *.wav > decodes

At -27dB SNR a total of 27 files where decoded. At -28dB only 4 successful decodes where produced. We see that the threshold for JT9A is around -27dB.

JT65A

The JT65A mode was originally designed to make minimal EME QSOs in the VHF and UHF bands. However, now its use for EME has being replaced by JT65B, which uses twice the tone spacing. JT65A is routinely used for minimal weak signal QSOs in the HF bands and under ionospheric openings in the 6m band. I find a bit stupid the popularity of JT65A in HF. As we will see, JT9A performs 2dB better and uses much less bandwidth. When the bands are open, the JT65 frequencies are crowded with overlapping JT65A signals. JT9A is a much better choice, as many non overlapping signals can fit in a 2.5kHz bandwidth. For 6m, JT9A also provides better performance than JT65A under most circumstances.

The FEC used by JT65A is a (63,12) Reed-Solomon code over GF(64). The modulation is 2.692baud 65-FSK, with a tone separation of 2.692Hz (the separation of the lowest tone is 5.383Hz). The lowest tone is used for synchronization and the remaining 64 tones are used for data (each tone transmits one FEC symbol). The synchronization tone appears in 63 symbol intervals (i.e., 50% of the time). A JT65 message contains 72 data bits.

The tool used to generate JT65 signals is called jt65sim. Here I generate 100 files at a SNR of -25dB. It is not possible to set the message. It is fixed to “K1ABC W9XYZ EN37”.

$BUILD/jt65sim -m A -n 1 -f 100 -s \\-25

WSJT-X 1.7.0 is the first release that will include the Franke-Taylor soft decoder. This decoder is fully implemented in the software I’m using, but it’s still under development and perhaps some details will be tweaked. This coder replaces the patented and closed-source Kötter-Vardy algorithm that was used in previous releases of WSJT-X and WSJT, and which was regularly used with the polemical Deep Search function.

Although it may be a bit off-topic for this post, I can’t resist to tell a bit of the story behind the Franke-Taylor decoder. The Reed-Solomon codes are normally decoded with a hard algebraic decoder, usually the Berlekamp-Massey algorithm. Because of its interpretation using polynomials over finite fields, the Reed-Solomon codes lend naturally to these sort of algebraic decoders (all the algebraic decoders have the same performance, they just differ in how the computations are arranged). What “hard decoder” means is that the decoder must be given the list of the symbols received. Thus, the FSK receiver just chooses the strongest tone as the symbol received and passes that information to the decoder, throwing away all the information about which tones are stronger, which other tones where the second most likely and so on. A hard decoder can’t simply perform as well as a soft decoder, which uses all this information about the probability that different tones where received. On the other hand, convolutional codes have an interpretation in terms of hidden Markov models, and so, they lend naturally to soft decoders, such as the Viterbi decoder and the Fano algorithm. This explains their use in many other JT modes.

Until recently, the only good soft decoder for Reed-Solomon codes was the Kötter-Vardy algorithm, which is patented. Joe Taylor managed to include it in his programs under a closed-source implementation, but that was not really an acceptable solution. However, the Kötter-Vardy algorithm performed much better than the hard Berlekamp-Massey algorithm. Using Berlekamp-Massey, EME was simply not possible except for the largest stations. Thus, Joe Taylor used the Kötter-Vardy algorithm as a sort of temporary solution to increase the popularity of JT65 and digital modes in general for EME.

Recently, Steve Franke K9AN and Joe Talyor K1JT have published a new soft decoder for the RS(63,12) code used in JT65. The thing which I like the most about this new algorithm is that its key idea is rather simple. Algebraic hard decoding of a RS(63,12) code can always correct up to 25 errors. However, if we are certain that some of the symbols are errors, then we can consider those symbols as erasures and pass that information to the algebraic decoder. The algebraic decoder can correct \(e\) errors and \(s\) erasures as long as \(s + 2e \leq 51\) (the erasures don’t count as errors). Thus, we see that if we guess correctly some of the erasures, the algebraic decoder can correct much more than 25 errors. In the extreme case where we know beforehand the positions of all the errors, by considering all of them as erasures the decoder can correct up to 51 errors. The key of the Franke-Taylor decoder is to do an educated guess of which symbols to consider erasures and then to run the Berlekamp-Massey algorithm with this erasure information. Many different guesses (10000 is usual) are tried until the decoder succeeds or the try limit is reached.

Of course, the ingenious part of the Franke-Taylor algorithm lies in the details. A statistical analysis is done in their paper to justify which symbols are most likely to be received with errors. Also, there are some statistical considerations after a successful decode is achieved, to try to decide whether that decode is good or a false positive. In its current version, some of this parameters are tunable in wsjtx.

The decoder for JT65 is also jt9. Here I try to decode the 100 test files using the new Franke-Taylor algorithm.

jt9 -b A -6  *.wav > decodes

It is also possible to use $BUILD/jt65 instead of jt9. This allows tuning some parameters.

At -25dB SNR a total of 39 files where decoded. At -26dB SNR only 5 decodes where obtained. Thus, we see that the decoding threshold for JT65A using the new Franke-Taylor algorithm is around -25dB. This agrees nicely with the graphs in the paper. In comparison, the threshold for the Berlekamp-Massey decoder is only -23dB and the threshold of the patented Kötter-Vardy decoder is -25dB. In fact, the Franke-Taylor decoder outperforms the Kötter-Vardy decoder slightly when using 10000 or more trials.

JT65B

JT65B is identical to JT65A except for the fact that it uses twice the tone separation as JT65A. This makes it more tolerant to Doppler spread and frequency instabilities, making it the mode of choice for EME in the 2m and 70cm bands. Sometimes it’s also used in the 23cm band when libration is low.

So far, we’ve only being simulating additive white Gaussian noise. Many of the tools also allow to simulate Doppler spread. Here we will simulate 4Hz of Doppler spread, which corresponds to bad EME conditions on the 2m band. Note that this is only a limited simulation of the EME propagation channel. In particular, we don’t account for any type of fading. In fact, it’s difficult to come up with a good mathematical model for the EME channel. More complex channels could be simulated with GNUradio, which offers several processing blocks. To do this, one would have to find a way to process .wav files in batches with GNUradio.

Here we generate 100 JT65B files with a SNR of -24.5 and a Doppler spread of 4Hz.

$BUILD/jt65sim -m B -d 4 -n 1 -f 100 -s \\-24.5

Decoding is done using jt9.

jt9 -b B -6  *.wav > decodes

At -24.5dB SNR a total of 34 decodes where successful. At -25dB only 16 decodes where obtained. Thus, the threshold for JT65B in these conditions is around -24.5dB. At lower Doppler spreads it can decode down to around -25dB. To compare with JT65A, we also did simulations for JT65A with 4Hz of Doppler spread. At -24.5dB SNR, 24 decodes were obtained and at -25dB SNR only 8 decodes where obtained. We see that JT65B performs better that JT65A in these channel conditions.

QRA64B

The QRA codes first appeared some months ago in Nico Palermo IV3NWV’s paper. There, he proposes to replace the Reed-Solomon code in JT65 with some rather novel codes called Q-ary repeat accumulate codes. These are a particular class of LDPC codes. The encoding process of a QRA code is rather simple and it almost seems a bit silly in comparison with Reed-Solomon. However, the decoding process uses a procedure called Maximum A Posteriori Probability. This is really serious statistical machinery. In its heart, it eventually boils down to Bayes’ Theorem, but I find it rather complex and interesting.

What I find most intriguing about QRA codes in comparison with Reed-Solomon codes is that they are not very well understood from a mathematical point of view. The Reed-Solomon codes are perfectly understood in terms of polynomials over finite fields. However, the QRA codes depend on several parameters that have to be selected experimentally for the best performance and nobody really understands why these codes work so well.

The best part of the Maximum A Posteriori Probability decoder is that it allows one to introduce in a natural way some a priori knowledge about the structure or content of the message that it’s expected to be decoded. This is very useful for minimal weak signal QSOs. For example, if we are calling CQ, we expect that the messages that we are going to receive are composed by our callsign (which is known), other callsign (unknown) and either a grid locator or a signal report (unknown). We may also want to listen out for other stations calling CQ. In that case we know something about the structure of the message: the message is a CQ call, but the station and grid are unknown. The Maximum A Posteriori Probability decoder uses all the a priori information available to improve decoding performance. The amount of a priori information known increases during a minimal QSO. At the final phase of the QSO both calls and signal reports are known and we expect to receive the rogers, whose structure and content we already know at this point. The only thing that the decoder has to do at this stage is to check whether it is plausible (with very high probability) that the received message is in fact the roger we expected. Thus, during a QSO the decoding threshold for the QRA code improves. From Nico’s paper we see that the threshold changes from -26.5dB with no a priori information to -30.5dB when the message we expect is completely known a priori.

This is a much more elegant and better solution than the polemical Deep Search decoder. What Deep Search does is to construct a list of all the possible messages that one would expect to see on air. Given the fact that the number of stations active on EME is not very high, this list of messages is of a manageable size. Then it tries to match blindly every possible message with the message that has being received. The database of stations active in EME is expected to be downloaded from the internet and/or maintained manually by the operator. With a typical database, the decoding threshold is around -29.5dB. The two more common complaints among the people that consider that Deep Search is cheating are the following. First, it wouldn’t work if the database of stations was much larger. For instance, it is of no use with the database of stations active on JT65 on the HF bands. Second, it uses information which is not obtained from the radio channel. In fact, it allows QSOs at SNRs low enough that the Shannon limit doesn’t permit to exchange all the information needed. For instance, at -27dB SNR it’s only possible to copy one callsign in 60 seconds. Deep Search pretends to copy two callsigns and a report in 48 seconds by filling in the missing information from the database.

When using QRA for a random QSO, all the information that is used as a priori information for the decoder has being obtained from the radio channel in previous successfully decoded messages. If using QRA for a sked, we already know the callsign and grid square of the other station. Still one piece of unknown information needs to be copied from the other station: the signal report. QRA doesn’t pretend to copy both callsigns and the report in this case. Since both callsigns are already known in advance, the decoder only needs to check if it is plausible that the callsigns in the received message match (with very high probability) those we expect. The signal report is unknown and still needs to be copied completely. This is enough for a valid QSO. For instance, according to the IARU R1 rules, both stations need to identify themselves, exchange some piece of unknown information (the report) and acknowledge the receipt of said piece of information. Note that it’s not necessary to copy both callsigns completely to be sure that both stations have being identified. It’s only needed that we check that the callsigns we receive matches the callsigns we expect, which we know in advance for a sked QSO.

QRA64B is the mode that is supposed to replace JT65B for EME and other weak signal work in VHF and UHF. As QRA64 is supposed to perform better than JT65 in any situation, there are also submodes QRA64A and QRA64C designed to replace JT65A and JT65C. Moreover, there are currently also modes D and E which use a larger tone separation and which may be useful on the higher microwave bands. The QRA64 modes are still in development, so their technical details are not well documented and could still change in the future. I known that 64-FSK modulation is used. Each tone encodes a FEC symbol (QRA64 works over the finite field GF(64), as the RS(63,12) code used in JT65). The synchronization tone used in JT65 is not present in QRA64 and instead a Costas array is used for synchronization.

The tool to generate QRA64 signals is called qra64sim. Again, note that QRA64 is still in development, so the performance of the version I’m using may not match the performance that the development team plans to achieve. Here, I generate 100 test files with the message “CQ M0HXM IO94” at a SNR of -27dB and 4Hz of Doppler spread.

$BUILD/qra64sim "CQ M0HXM IO94" B 1 4 0 100 -27

There is no command line tool to decode QRA64 yet. This is not a problem, because wsjtx can be used. To do so, we first go to “File > Open” and select our first .wav file. wsjtx attempts to decode that file. Then we can go to “File > Decode remaining files in directory” and wsjtx will attempt to decode the rest of the files, one by one. This method could also be used for any of the other modes.

First we are simulating a random QSO. We don’t know that M0HXM is calling CQ. Thus, the only a priori information that the decoder has is that we are mostly interested in messages which contain “CQ” or our own call (EA4GPZ, in this case). At -27dB SNR a total of 25 decodes where obtained and at -28dB SNR only 2 decodes where successful. Thus, the threshold for this situation is -27dB.

Now we simulate a sked QSO with M0HXM. We know his callsign and grid square in advance, so we set those in the “DX Call” and “DX Grid” fields in wsjtx. The decoder uses this a priori information, since it expects to receive the messages “CQ M0HXM IO94” or “EA4GPZ M0HXM IO94” or perhaps “EA4GPZ M0HXM ??”, where ?? is a signal report. The decoder doesn’t really copy M0HXM’s callsign and square over the air. It just checks that the message we receive matches plausibly (with high probability) the message “CQ M0HXM IO94” that we already expect to receive. In this situation, at -30dB SNR a total of 22 decodes where obtained. At -31dB I only got 9 decodes. The threshold in this situation is around -30dB SNR. This shows the improvement in performance by using a priori information. An advantage of 3dB is obtained in this phase of the QSO for a sked QSO compared to a random QSO.