The Heisenberg Picture of Quantum Mechanics from the Classical Hamiltonian Formalism

I gave this answer on Physics Stack Exchange to the following question:

Let’s have formula of full time-derivative of physical value in Poisson’s formalism:
$$\tag{1}\frac{df}{dt} = -\left\{H, f\right\}_{P. br.} + \frac{\partial f}{\partial t}$$
where $\left\{A, B\right\}_{P. br.}$ is Poisson’s bracket, and $H$ the Hamiltonian.
By using quantum mechanics the equation can be rewritten in form
$$\tag{2}\hat {\frac{d f}{d t}} = \hat {\dot {f}} = \frac{i}{\hbar }[H, f] + \hat {\frac{\partial f}{\partial t}}$$
It is a bit like Heisenberg picture, but in $(2)$ in the left side there is operator of full time-derivative of function (also, in the right side – operator of partial time-derivative). In Heisenberg picture in the left side is full time-derivative of operator, which represents function.

So my question is: can I get Heisenberg formalism from $(1)$?

and the answer, slightly edited, is as follows:

As with most if not all (ultimately) things in physics, one does not derive $(2)$ from $(1)$, one guesses $(2)$ from $(1)$ and then confirms the soundness of the guess through experiment. I’m being a bit flippant with the word “guess” as a slight satirisation of ourselves as physicists when we (myself definitely included) lose site of the fact that we are not coming up with mathematical proofs. Lest I should sound a bit condescending – you can take the following as a criticism and correction that I have sternly delivered many times to myself when I have come up with similar questions as well as this same one.

So let us rephrase our question as “how does (1) motivate (2)”? (1) describes the time evolution of the value of some smooth function $f = f(\psi)$ on a manifold $M$ we call “configuraton space” when a system’s state $\psi \in M$ evolves along a path in $M$, which path is defined by the classical Hamilton’s equations of motion. Everything is deterministic, the co-ordinates represented by generalized positions $\mathbf{p}$ and momentums $\mathbf{q}$ are in principle definable to infinite precision and one can meaningfully talk about system “positions” and “momentums” all at once to any precision. We can think of a general smooth function $f(\psi)$ as an infinite precision measurement on the system when the latter’s state is $\psi$.

Along comes Heisenberg and says that measurements are the only things that are real and that infinite precision specification of a state in configuration space as done in classical Hamiltonian and Lagrangian mechanics is meaningless, at least at quantum levels, because precise measurement of some of the co-ordinates in state space means that other ones can only be measured roughly – i.e. the uncertainty principle is born.

This new thinking doesn’t just make the configuration space concept awkward, it altogether destroys the concept of configuration space, so we have something genuinely new. If we are to salvage our classical ideas, then this can only be done through guess, analogy, inspiration, not derivations. The only derivations we do are the other way around: once we have guessed the new theory, we must show that we can derive the old one as an approximation holding when the experimental conditions are those that validated the old theory.

So how do we find “guesses” and “inspirations”? The answer was different for just about every practitioner of early quantum mechanics, just as you will likely get many answers to your question – all right and working well as inspiration for the particular person putting them forward. So here’s how I like to think of things.

After Heisenberg, we are left with some state $\psi$ disembodied from any configuration space whatsoever, and this state defines the values various observables, which are operators modeling measurements, together with special recipes for interpreting what measurements these operators yield when state $\psi$ prevails. Since we haven’t got a configuration space, we really haven’t got much to work with, so we just say $\psi$ belongs to some Hilbert space and the observables are the operators on this space see my description here. I, unlike Heisenberg, also like to begin with the general Schrödinger equation, since that is really assuming not much more than linearity together with the assumption that the system’s description does not vary in time: this yields

$$i\,\hbar\,\mathrm{d}_t \psi = \hat{H}\,\psi$$

(see my explanation of this here). At this point $\hat{H}$ is some observable that defines the system’s innards in some way. Thereafter we can convert from the Schrödinger to Heisenberg picture (also as in the same answer I just cited), wherein the state no longer evolves but instead the observables do:

$$\mathrm{d}_t \hat{M} = -\frac{i}{\hbar} [\hat{H}, \hat{M}] + \partial_t \hat{M}$$

and at this point we (or rather, Dirac in 1926) notice the likeness between the observable’s evolution and that of the classical measurement evolving in configuration space following the your equation (1) involving the Poisson bracket. It is this likeness or analogy that motivates people to guess that quantum theories can be guessed from classical ones by replacing Poisson with Lie brackets. There are also other mathematical analogies that some people (including me) find comforting and motivating: for example, Poisson brackets can define Derivations (in the algebraic, not logical, sense of the word) on the linear space of smooth functions on the configuration space manifold $M$, just as Lie brackets defines derivations on the tangent bundle of some general smooth manifold. Ultimately, the proof, though is in the success of any theory in foretelling experimental outcomes.

Now it is worth mentioning than, after the fact, researchers (notably Weyl, Wigner Groenewold and Moyal) did find a way to reformulate quantum mechanics so that both your equations (1) and (2) can be shown to belong to a more general whole. The classical phase space can be “regenerated” in the quantum phase space, where now the information in the state $\psi$ in Hilbert space is replaced by a completely logically equivalent quasi probability distribution which is a distribution function over the phase space co-ordinates $p_j$ and $q_j$: see the Wikipedia page on the Quantum Phase Space Formulation: this probability distribution is real valued but can be negative in small regions of the phase space. However, such regions are small in the sense of “small enough to represent simultaneous precision in $p$ and $q$ that would violate the uncertainty principle”. Over regions allowed by the uncertainty principle, the integrated distribution is positive, so that it becomes a “classical” probability distribution if you imagine “coarse gaining” the quantum phase space so that it is approximated by discrete points, each taking the place of a volume in phase space corresponding to a simultaneous precision in $x$ and $p$ that is allowed by the uncertainty principle. Observables now become, through the Wigner-Weyl transform, operators on the phase space – generalized marginal probability integrals – and operator composition is replaced by the star product. The Lie bracket of operators is replaced by the Moyal bracket. We now get to a generalized version of your equation (1):

$$\mathrm{d}_t \hat{M} = -\{\{\hat{M}, \hat{H}\}\} + \partial_t \hat{M}\quad(3)$$

where the Moyal Bracket $\{\{\;,\;\}\}$ is your classical Poisson bracket plus higher order terms of order $\hbar^2$ So we come the full circle. The Moyal brackett is a generalized derivation (again in the algebraic sense) that includes a continuous deformation between the classical Poisson and quantum mechanical Lie brackets, parameterized by a real parameter $\hbar$ (with the Poisson bracket resulting as $\hbar\rightarrow 0$). This is extremely elegant and compelling, but it is important to heed that all this was done *after the fact* in the years just before 1949. This kind of generalization is the kind of “derivation” of classical physics as a limit of the more general quantum physics that I spoke of at the beginning of my answer.

Lastly, another path between the classical and quantum ideas can be seen in the time evolution of the hypersurfaces of constant action in classical configuration space, which can be shown to be described by the Eikonal equation (a reformulation of the Hamilton-Jacobi equation), which, in this context, is a nonlinear Schrödinger equation: see the Wikipedia page on the Hamilton-Jacobi equation. Now, thinking of these hypersurfaces of constant action, the Eikonal equation is an approximation of the Helmholtz equation that holds when the scalar field varies slowly in space compared with the wavelength. So, our motivation goes, maybe the Hamilton-Jacobi equation is simply the approximation that holds when the rate of change of the action $S$ is small compared with some quantum of action $\hbar$ per unit “distance” in configuration space. So we “reverse engineer” this reformulation of the Hamilton-Jacobi equation to get the actual Schrödinger equation using the relationship between the Eikonal and Helmholtz equations to help us. Once we have the Schrödinger equation, we can get your equation (2).

Be sure also to explore a history of the early ideas of replacement of Poisson by Lie brackets in the paper