From Newton–Euler to Hamilton–Jacobi

Let’s say that we want to reformulate “Newtonian” particle mechanics so that it looks analogous to wave motion, in order to make our knowledge of one math subject applicable to another. A light ray, minding its own business, propagates in the direction perpendicular to its wavefronts. So, the particle momentum should be perpendicular to lines of something, meaning that it should be the gradient of something:
$$
\vec{p} = \vec{\nabla} S \, .
$$
We don’t know what $S$ is, except that it’s a field whose value presumably depends upon position and time, and it ought to satisfy some equation. Can we find an equation for it?

Well, from mechanics we know that the time derivative of the momentum is the force. And in the absence of friction, we can write the force as minus the gradient of the potential energy, which feels relevant here:
$$
\frac{d\vec{p}}{dt} = -\vec{\nabla} V \, .
$$
Combining these equations, we get
$$
\frac{d}{dt} \vec{\nabla} S = -\vec{\nabla} V \, .
$$
But wait! A particle riding along in a field sees two contributions to the field value changing. If the particle is sitting still, then the field value at its current spot will change if the field itself changes. A moving particle can also just move over to a spot where the field value is different, even if the field is constant over time. We learn how to cover this in hydrodynamics, by splitting the derivative into two terms, one with a velocity dependence:
$$
\frac{d}{dt} \vec{\nabla} S
= \frac{\partial}{\partial t} (\vec{\nabla} S)
+ (\vec{v}\cdot\vec{\nabla}) \vec{\nabla} S \, .
$$
And this is what equals minus the gradient of the potential.

(Where does that dot product $\vec{v}\cdot\vec{\nabla}$ come from? Well, imagine we’re in one dimension, moving through a field $f$. In a tiny interval of time $dt$, we move a distance $v\,dt$, and over this distance, the field value changes by an amount $(v\,dt)(\partial f/\partial x)$. Extending this to three dimensions gives the expression above.)

Now, we can write the $\vec{v}$ here as the momentum divided by the mass $m$, and we have declared that the momentum is the gradient of our mystery field. Therefore,
$$
\frac{\partial}{\partial t}(\vec{\nabla} S)
+ \frac{1}{m} (\vec{\nabla}S \cdot \vec{\nabla})
\vec{\nabla} S
= -\vec{\nabla} V \, .
$$
Let’s exchange the order of differentiation on that first term:
$$
\vec{\nabla} \left(\frac{\partial}{\partial t} S\right)
+ \frac{1}{m} (\vec{\nabla}S \cdot \vec{\nabla})
\vec{\nabla} S
= -\vec{\nabla} V \, .
$$
It looks almost like each term can be the gradient of something. So, we stare at the one that isn’t for a while, until we remember from calculus that
$$
\frac{d}{dx} \left[\left(\frac{df}{dx}\right)^2\right]
= 2 \frac{df}{dx} \frac{d^2 f}{dx^2} \, .
$$
By the same logic, we find that our equation for $S$ is what we’d get by taking the gradient of both sides of
$$
\frac{\partial S}{\partial t}
+ \frac{1}{2m} (\vec{\nabla}S)^2 = -V \, .
$$
Or, isolating the time derivative and flipping the signs to give it a more conventional presentation,
$$
-\frac{\partial S}{\partial t}
= \frac{1}{2m}(\vec{\nabla} S)^2 + V \, .
$$

The Hamilton–Jacobi equation just generalizes this! We can write it as
$$
-\frac{\partial S}{\partial t} = H(\vec{x}, \vec{\nabla} S, t) \, .
$$
Another common notation is to write $q$ for position instead of $\vec{x}$, suppressing the little arrow and just remembering that $q$ can be a whole list of coordinates (for multiple particles, potentially). Then
$$
-\frac{\partial S}{\partial t}
= H\left(q, \frac{\partial S}{\partial q}, t \right) \, ,
$$
with
$$
p = \frac{\partial S}{\partial q} \, .
$$
This discussion was inspired by a 1965 paper of Nathan Rosen. We have actually followed Rosen’s logic in reverse, starting with the familiar instead of justifying the unfamiliar by transforming it until it looks recognizable. A more conventional treatment can be found in José and Saletan’s Classical Dynamics: A Contemporary Approach (2012), section 6.1.