So, about that Apple Belle footer...

Yeah, I did some refactoring of that one and must have accidentally yanked down the wrong line. It should be

You'll note that now the total entropy no longer peaks at 5-6 bits per symbol, but just decreases monotonically. This is basically what you'd expect from random noise, which hints that looking at the entropy like this isn't going to help solve anything. (but maybe it will, you never know til you solve it!)

The biggest issue with all of these numbers is that the input is so short.

Indeed. Essentially all the patterns you've seen in the entropy for this and the previous one with the 5x2 key in the lower right have been spurious artifacts of the fact that the message is finite length. I'll explain a bit more. The equation for the entropy involves the probability of each symbol occurring. With a finite length message, you estimate this probability numerically, which means for shorter messages your estimate is increasingly inaccurate. In fact, it's also inaccurate in semi-predictable ways. For instance, if your choose your symbol length to be equal to the message length, you'll always get an entropy of 0 (which is related to the fact that you see your total entropies decreasing as you increase symbol length).

I'll reiterate that when you do these entropy calculations, it's a wise idea to always double-check your work by running you code on a pure-random input. If you had done so in this case I think you'd also see the entropy for a pure-random 80-bit message matching the same patterns which led you to conclude a 5-bit symbol length, which indicates that your math is wrong somewhere.

A longer string with the same Kolmogorov complexity as this data would have lower Shannon entropy.

This depends on how you calculate things. Really, shannon entropy and Kolmogorov complexity are measuring different types of things. Shannon entropy is measuring the information content of a message drawn from a possibly noisy source, while Kolmogorov complexity measures the inherent complexity within a single message. This makes them hard to compare.

For the way you're calculating entropy, what you've said is false. For example if I have a string of the form "01010101010...", it has very low Kolmogorov complexity, but if I calculate the 1-bit Shannon entropy it will be high no matter how long the string. I'm sure it's intuitively obvious to you whi 1-bit Shannon entropy is a terrible way to analyze that string, but this points to some of the difficulties of making a direct comparison. Really, Kolmogorov complexity deals with single strings, and Shannon entropy deals with randomized string generators, so care is needed in translating from one context to the other. At least that's my impression. I'm not an expert in this.

As you say, Kolmogorov complexity is provably impossible to calculate in general. So it's a tool of primarily theoretical interest.

The remainder bits get clipped.

Odd, I did a quick calculation of the theoretical maximum per-symbol entropy you could expect to see in this case, and your numbers are larger, which either indicates a careless error in your code or some careless error in mine.

/r/MLPLounge Thread