The Principle of Maximum Entropy Explained

Hello Youtubers!

If you watched my very first video on this channel, I described the Principle of Maximum Entropy as it pertains to Climate Systems. However, the Principle of Maximum Entropy doesn’t apply to just climate systems. It applies to all systems. It’s a foundational understanding of Information Theory and is derived from the Second Law of Thermodynamics. Today I’ll go into greater detail about the Principle of Maximum Entropy. In the near future, I’ll relate it to political systems and systems of government.

Let’s first start with the concept of entropy. There are several definitions for entropy, but in general, entropy is thought of as the amount of disorder in a system. In information theory, entropy can be thought of as the amount of uncertainty as to the possible outcomes of a variable; that is, in some ways, it could be thought of as the amount of chaos or randomness associated with a variable’s outcome. In thermodynamics, entropy can be thought of as the amount of possible combinations in which the atoms in the system can mix around.

Now let’s consider the 2nd Law of Thermodynamics. The 2nd Law teaches that, in a closed system, entropy increases in the system until it reaches thermodynamic equilibrium. So, in a closed system, the state with the maximum amount of entropy in it is thermodynamic equilibrium. That is, that’s when the system is the most disordered because that’s when the atoms are mixed up the most. So, the system is seeking the state with the maximum amount of entropy in it.

While the 2nd Law only applies to closed systems, both open and closed systems seek a state of maximum entropy. The difference is, that, in a closed system, thermodynamic equilibrium is the maximum entropy state. Open systems, however, never reach thermodynamic equilibrium, because energy is added to the system. Instead, the system will seek the maximum entropy state given the constraints of the system.

Let’s consider a simple open system. Open systems will reach what’s called a steady-state, which is when the inputs and the output are equal. Open system can have only one output, but multiple inputs. So, a simple open system could have two inputs.

So for example, let’s say that the output is ten…something. That means the system reaches a state where the inputs can result in ten being produced as the output. So, to keep things simple, considering two inputs, and an output of 10, the inputs could be 5 and 5, or 4 and 6, or 7.5 and 2.5 or a trillion other combinations. In fact, an infinite amount. What’s important is that an open system can take many possible steady states.

The principle of Maximum entropy is that, the system will seek the state with the maximum amount of entropy in it. Rather than seeking a steady state, a closed system will seek thermodynamic equilibrium but it’s still the state with the maximum amount of entropy in it, given its constraints. So the 2nd Law can be seen as a specific application of the Principle of Maximum Entropy for closed systems. Again, the point is that systems seek as much disorder as possible.

This principle has been successfully applied to a number of scientific fields because, remember, the principle applies to all systems, not just thermodynamic systems. Information systems, astronomical system, biological systems, climate systems. All systems.

So naturally, this concept applies to political systems. There’s been quite a bit written about entropy in political systems already, but I imagine I can add something new to the discussion. In a series of videos in the near future, I’ll evaluate and summarize several of these articles first, then do an experiment of my own, and detail the process on either my blog site or on this channel. For now, I’ll simply say this.

When considering an open system, perhaps the most important consideration is the constraints of the system. The better knowledge you have about the constraints of the system, the more accurate the model output when compared to real world observations. That is, the constraints of the systems represent the known information about the system, and the model output is the unknown information. So, in short, as any data analyst or data scientist would know, good data in = good data out.

That’s all the time I have for now. If you liked this video, please show your support by subscribing! Thanks for your time.