Information is Physical: Landauer’s Principle and Information Soaking Capacity of Physical Systems

There is a beautiful description in information theoretic thermodynamics to account for the Maxwell Daemon and the Szilard Engine that makes these two thought experiments comply with the second law of thermodynamics in the long term and that is through Landauer’s Principle: the idea that the merging of two computational paths or the erasing of one bit of information always costs useful work, an amount given by $k_B\,T\,\log 2$, where $k_B$ is Boltzmann’s constant and $T$ the temperature of the system doing the computation.

The Szilard engine is a three-stage, two-stroke idealised heat engine drawn below:

Szilard Engine

Figure 1: The Szilard Engine

The engine comprises a two piston cylinder containing one molecule as the working fluid, kitted with a sideways sliding door, and steeped in an infinite heat bath together with a Maxwell Daemon: some measurement system which can measure the molecule’s position and work the sliding door and pistons given that information. It is important to take heed that the Maxwell Daemon can be non biological – so thatr consciousness and brains don’t enter this argument – and indeed the Maxwell Daemon is a simple three-state machine that can be realised in very simple computer logic. The hard part of this system is actually the position measurement. This crazy system has actually been built and successfully tested in the laboratory! See the following Nature paper:

Shoichi Toyabe; Takahiro Sagawa; Masahito Ueda; Eiro Muneyuki; Masaki Sano (2010-09-29). “Information heat engine: converting information to energy by feedback control”. Nature Physics 6 (12): 988–992. arXiv:1009.5287. Bibcode:2011NatPh…6..988T. doi:10.1038/nphys1821.  The abstract:”We demonstrated that free energy is obtained by a feedback control using the information about the system; information is converted to free energy, as the first realization of Szilard-type Maxwell’s demon.”

In the laboratory system, the molecule was replaced by a pollen grain undergoing Brownian motion in a fluid, so that (i) there was a goodly sized target for the laser interferometric position measuring system to track readily and (ii) the “molecule” was slow enough for the Maxwell Daemon hardware to track it. That the Szilard engine can be upsized and slowed down in this way so to make its realisation in molecular biological systems is interesting; a point I shall return to below.

In the “set door” stage (1 and 2), the Daemon shuts the door sundering the cylinder into two separate chambers. In principle, this operation needs no work: it could be opened or shut reversibly against a spring, for example, which could return the energy needed to compress ready for using again in the next spring compression. In the “insight” stage (3 and 4), the Daemon measures which side of the door the molecule is on. In principle this too requires an arbitrarily small amount of work and even quantum limits do not hinder us. For we can use bigger and bigger molecules and slow the system down: the molecule’s rms momentum is $p = \sqrt{\alpha\,k_B\,T\,m}$, where $\alpha$ depends on how many degrees of freedom (from the standpoint of applying equipartition of energy ideas) the molecule has. We can make this momentum arbitrarily big at a given temperature if we are willing to slow the engine down: so we simply make  $p$ big enough that any uncertainty in it that needfully arises from the measurement of the molecule’s position does not affect the system’s working (or at least it only does so with fleetingly small probability, so that the “failed” engine cycles do not affect the overall, long term engine performance). In stage (3), the piston on the cylinder’s “empty” half is thrust home against the central sliding door. Because the molecule is not inside this cylinder half, the work needed for this task is again in principle nought. In stage (4) the central door is withdrawn.

Now in the “Induction / Power” stage (1′), the molecule bounces back and forward against the pistons and also bounces against the thermalised walls of the cylinder. As it does so, it can do work against the piston rammed home in stage (3), whilst the energy lost from the molecule in doing this work is replenished when the molecule bounces against and interacts with the cylinder walls. Put simply: the working fluid expands to its beginning volume isothermally, drawing heat from the reservoir into the molecule in so doing as shown in the drawing. If we assume that the  the ideal gas law, the work done against the piston is $k_B\,T\,\log 2$, where $T$ is the heat bath’s thermodynamic temperature. At last when the power stroke is over, the system is ready to begin its cycle again.

The point about this system is that it has used the one bit of information that tells us which half of the cylinder the molecule is in to convert $k_B\,T\,\log 2$ units of heat in a heat bath at thermodynamic equilibrium to useful work. Given the assumption that the second law of thermodynamics holds, this must mean that somehow the workings of the system above must call for an input of $k_B\,T\,\log 2$ units of work. Szilard thought that what this meant was that the measurement of which half the molecule is in needed the expenditure of $k_B\,T\,\log 2$ units of work. However, as argued roughly above, this is not so. The question is probed much more carefully and its answer finalised by Charles Bennett and is directly related to the limited ability of matter, or rather “physical systems” to “trap” or “soak up” information. A review of his work can be found in:

Charles Bennett, “The Thermodynamics of Computation: A Review”, Int. J. Theo. Phys., 21, No. 12, 1982.

Here’s how it works. Bennett invented perfectly reversible mechanical gates (“billiard ball computers”) whose state can be polled without the expenditure of energy and then used such mechanical gates to thought-experimentally study the Szilard Engine and to show that Landauer’s Limit arises not from the cost of finding out a system’s state (as Szilard had originally assumed) but from the need to continually “forget” former states of the engine.

We can probe this idea more carefully, as done in Bennett’s paper. As the Daemon converts heat to work, it must record a sequence of bits describing which side of the Daemon’s door (or engine’s piston, for an equivalent discussion of the Szilard engine) molecules were on. For a finite memory machine, one needs eventually to erase the memory so that the machine can keep working. However:

“Information” ultimately is not abstract – it needs to be “written in some kind of ink” you might say – and that ink is the states of physical systems.

The fundamental laws of physics are reversible, so that one can in principle compute any former state of a system from the full knowledge of any future state – nothing gets lost. The system’s state at any former or future time is mapped one-to-one to its state at any other time. So, if the finite state machine’s memory is erased, the information encoded that memory must show up, recorded somehow, as changes in the states of the physical system making up and surrounding the physical memory. So now those physical states behave like a memory: eventually those physical states can encode no more information, and the increased thermodynamic entropy of that physical system must be thrown out of the system, with the work expenditure required by the Second Law, before the Daemon can keep working. The need for this work is begotten of the need to erase information, and is the ultimate justification for Landauer’s principle. Here we also see that one needs to apply the second law to a truly closed cycle in state space. It may on the surface seem that the Szilard engine can go through one of its cycles without any work needed, butwe have not truly brought the the system back to the same state. A bit of information – which side of the door the molecule was on – has been “forgotten” by erasing the Daemon’s memory. More precisely, that bit must have been encoded as a change of state in the physical system’s state somewhere. In many situations, the Daemon might seem to win even over many cycles. But this would be because the subtle change of the overall system’s state is too slight to notice over even many cycles. But, inevitably, the Daemon / Engine’s system’s internal states would at last be so thermalised by this randomisation of its internal states that it could no longer work. The effect would be very slow and subtle. A side note here is that Chemists sometimes talk about the Gibb’s (or other e.g. Helmoholtz) free energy as the enthalpy less the work needed to expel the excess entropy the reaction products have relative to the reactants. Although quite different in context, it is the ultimately the same kind of idea here. If the reaction products have lower entropy than the reactants, i.e. their internal states can encode fewer bits, then the excess information stored in the states of the reactants has to wind up encoded somewhere outside the reaction products if the mapping between the total system state at any two times is to stay one-to-one.

Interestingly, it is the fact that the laws of physics are reversible that really gives this argument its legs. From this fact, i.e. the one-to-one mapping between total system states at all time, we know that erasing of memory must lead to thermalisation / changes in system state elsewhere in the Daemon / Engine system to keep this mapping one-to-one; so that the World can remember how it got from former to present states.

Classical thermodynamics itself relates information soaking capacity of materials to heat capacities. The experimentally measured entropy (or, often, that inferred from macroscopic measurements) $S_{exp}$ of a system is the Boltzmann entropy, and this can be shown to overbound the actual Shannon information needed to specify the classical state of the system. This latter information is related to the Gibbs entropy, as discussed in:

E. T. Jaynes, “Gibbs vs Boltzmann Entropy”, Am. J. Phys. 33, number 5, pp391-398 May 1958;

Gibbs and Boltzmann entropies are equal when the state constituents are statistically uncorrelated. So the Boltzmann entropy $S_{exp}$ can be thought of as an information capacity; to map it from a quantity in joules per kelvin to bits we compute $\frac{S_{exp}}{k_B\,T\,\log 2}$. The addition of everyday quantities of heat to a system corresponds to very big information additions: the addition of one kilojoule of heat to a system at $300K$ (roomish temperature) corresponds to “complexifying” its state by $3.5\times10^{23}$ bits, roughly equal to our estimate of the Earth’s computer systems’ total storage capacities in 2013. Another example: the entropy of formation of a mole of water (18 grams) at atmospheric pressure is $70\mathrm{J\,K^{-1}}$, or $7.4\times10^{24}$ bits. The classical microstate of all the molecules in this water would need up to seventy times our estimate of the Earth’s computer systems’ total storage capacities in 2013 to specifiy completely.

In the field of Black Hole Thermodynamics there is the Berkenstein Bound (see the Wikipedia page with this name), which is the maximum amount information, measured as Shannon entropy, that can be encoded in a region of space with radius $R$ containing mass-energy $E$. The bound is:

$$I\leq \frac{2\,\pi\,R\,E}{\hbar\,c\,\log 2}$$

where $I$ is the number of bits contained in quantum states of that region of space. This bound was derived by doing a thought experiment wherein Berkenstein imagined lowering objects into black holes (see this Physics Stack Exchange question) and then deduced the above bound by assuming that the second law of thermodynamics holds. It works out to about $10^{42}$ bits to specify the full quantum state of an average sized human brain. This is to be compared to estimates of the Earth’s total computer storage capacity, which is variously reckonned to be of the order of $10^{23}$ bits (see the Wikipedia “Zettabyte” page, for instance) as of the time of writing of this post (2013).