+- MKLab (https://mklab.gr)
+-- Forum: [INDEX] (https://mklab.gr/forumdisplay.php?fid=1)
+--- Forum: SCIENTIFIC COMPUTING (https://mklab.gr/forumdisplay.php?fid=4)
+---- Forum: COMPUTATIONAL MATHS (https://mklab.gr/forumdisplay.php?fid=68)
+----- Forum: ARTICLES (https://mklab.gr/forumdisplay.php?fid=89)
+----- Thread: Introduction to Automatic Differentiation (/showthread.php?tid=2320)
Introduction to Automatic Differentiation - mklabgr - 10-01-2026
Automatic differentiation (AD) is presented as a powerful technique for computing derivatives of functions implemented as computer programs. Unlike numerical differentiation, which approximates derivatives using finite differences and can suffer from truncation and round-off errors, and symbolic differentiation, which manipulates mathematical expressions and can lead to enormous intermediate expressions, AD systematically applies the chain rule while executing the program. It can compute derivatives to approximately machine precision with only a small constant-factor increase in arithmetic work. This makes it useful in optimization, sensitivity analysis, physical modeling, probabilistic inference, and machine learning.
The article first explains forward-mode automatic differentiation using dual numbers of the form $x+\epsilon x'$, where $\epsilon^2=0$. The ordinary value and its derivative are propagated simultaneously through every operation in a program. For example, multiplication automatically produces $(x+\epsilon x')(y+\epsilon y')=xy+\epsilon(xy'+x'y)$, reproducing the product rule. Because the chain rule is automatically respected, the coefficient of $\epsilon$ at the end is the desired derivative or directional derivative. In terms of linear algebra, forward mode efficiently computes products of the Jacobian with vectors, $J_xf\cdot v$, without explicitly constructing the potentially enormous Jacobian matrix.
The complementary technique, reverse-mode automatic differentiation, is particularly important when a function has many inputs but relatively few outputs—for example, when computing the gradient of a scalar objective function. Instead of propagating derivative information forward, reverse mode first records the intermediate computations in a structure commonly called a tape and then propagates gradient information backward. This makes it highly effective for computing gradients, but it introduces additional memory and implementation complexity because intermediate values must generally be stored. The author also discusses important difficulties such as perturbation confusion when differentiations are nested, the treatment of higher-order functions and data structures, and the fact that AD differentiates the actual computational procedure rather than necessarily the ideal mathematical function being approximated.
Finally, Radul examines the implementation and software-engineering challenges of AD. Simply using dual numbers can introduce substantial runtime overhead through memory allocation and operator dispatch, while reverse mode requires careful management of stored computation. The article also warns about cases where differentiating an approximation can produce a poorer derivative than directly implementing a known approximation to the derivative.
Different AD systems—including operator overloading, source-to-source transformation, and programming languages with AD built into their design—make different trade-offs concerning performance, nesting, higher-order functions, custom derivatives, and abstraction. Overall, the article presents AD not merely as a faster way to calculate derivatives, but as a meeting point between calculus, numerical computation, and programming-language design. [/url]
[url=https://alexey.radul.name/ideas/2013/introduction-to-automatic-differentiation/]ARTICLE