Derivation Of The Effect Of Atmospheric Refraction On The Observed Elevation Angles Of Peaks

Atmospheric refraction slightly increases the observed elevation angle of a peak relative to the observer. See The Effect Of Atmospheric Refraction On The Observed Elevation Angles Of Peaks for an introduction to the effect. This page gives the derivation of the formula given on that page.

If you are not familiar with similar derivations, please stop here and just accept the formula at face value. While I normally have no problem answering questions about anything I've posted, there are simply too many things to explain here if you are not familiar with the methods used. If you want to learn how to do such calculations yourself, you probably need to get a B.A. degree in physics or math. (Let me know if there are other routes toward understanding this calculation. In fact, I'm not even sure if everyone with a B.A. in physics or math can follow this calculation, or whether it takes graduate work to do it.) I'll be happy to answer questions from those who understand basically what I'm doing below, and just need some help with the details.

The formula is derived by using the principle that light travels on the path that takes the least time to traverse. I vary a parameter that describes the path to calculate the path of least travel time.

Assume the observer and observed peak are at the same elevation Z = Zo. Let x be the horizontal coordinate, with x = 0 midway between the two. Assume that the distance between the two is 2*d. (At the end I'll redefine the parameter d to be the distance between them, but it is convenient for now in writing the formulae to do it this way.)

Symmetry implies that there can be no term proportional to x, and hence the smallest excursion from a straight line must be a quadratic. Hence assume that light take a quadratic path between the two:

(1) Z = h * [ 1 - (x/d)^2 ] + Zo

The equations are numbered for easy reference. The element of path along this line is:

(2) ds^2 = dx^2 + dZ^2 = dx^2 * [ 1 + (dZ/dx)^2 ]

From differentiating equation (1):

(3) (dZ/dx) = - 2 * h * x / d^2


(4) ds^2 = dx^2 * ( 1 + 4 * h^2 * x^2 / d^4 )

The speed of light is c / n, where c is the speed of light in a vacuum and n is the index of refraction. The formula for the index of refraction makes the following look very messy, but it is very simple in principle.

The optical index of refraction as a function of elevation and temperature is:

(5) n = 1 + { 2.9e-4 * exp(-Z/10 km) } / { 1 + (2.9 / 760) * [To - alpha * (Z - Zo) ] }

from Allen, C.W. Astrophysical Quantities. "The factor 2.9 * T makes an approximate allowance for the change of water-vapour content with temperature". Also, I have assumed that the temperature follows a linear lapse rate with altitude.

For small changes in Z:

(6) n ~ 1 + a + c * (Z-Zo)


(7) a = 2.9e-4 * exp(-Zo/10 km) / (1 + 2.9 * To / 760)

(8) b = 2.9 * alpha / {760 * (1 + 2.9 * To / 760) }

(9) c = a * (b - 1/10 km)

The time it takes light to traverse this path is:

(10) integral over path of (ds / v ),

where v is the speed of light at a given point on the path. Since v = c/n, equation (10) becomes:

(11) integral over path of (n * ds / c)

(12) = (1/c) * integral over path of { 1 + a + c * (Z-Zo) } * dx * (1 + 2 * h^2 * x^2 / d^2) },

where I have made the approximation that the sqrt(1 + x) = 1 + x/2 for x small.

It is straightforward to do that integral, obtaining:

(13) (1/c) * { (1+a)*2d + (4/3)*c*h*d + 4*h^2*(1+a)/(3d) }

To make the time a minimum, take the derivative of the time and set it equal to zero, obtaining:

(14) h = -c * d^2 / [ 2 * (1+a) ]

The angular refraction is

(15) dZ/dx (evaluated at x=d) = - 2 * h / d.

Substituting for h from (14) gives:

(16) theta = c * d / (1+a) .

Now multiplying and dividing by 2 to express this formula in terms of the distance (2d) between the two points gives:

(17) theta = c * (2d) / [2 * (1+a) ]

Now redefining d as the distance between the two points [letting (2d) be replaced by (d), with the new meaning for d] gives:

(18) theta = c * d / [2 * (1+a) ]

All the calculations have been in the metric system so far, so to give theta in radians when d is expressed in miles, the formula becomes:

(19) theta = [ 1.6 * c / { 2*(1+a) } ] * d,

which is the formula given in The Effect Of Atmospheric Refraction On The Observed Elevation Angles Of Peaks.

Note that this calculation assumes quite a bit. The real atmosphere can vary markedly horizontally, can have temperature inversions, can change its humidity, and have additional components like dust that change the index of refraction. The observer and observed peak are not always at the same elevation assumed in the derivation of this formula. Hence there are no guarantees that this formula will even in most cases give accurate results.

However, this formula certainly gives a rough expected value for the atmospheric refraction, and hence I have used the nominal value corresponding to an altitude of 6000' with a temperature of 70° F. and a normal temperature gradient of 6.5° C./km. Using this nominal value eliminates any bias in the calculated angles on average.

Go to:

Copyright © 1998-2000 by Tom Chester.
Permission is freely granted to reproduce any or all of this page as long as credit is given to me at this source:
Comments and feedback: Tom Chester
Last update: 11 April 1999, with a typo in equation (11) corrected on 1 August 1999 thanks to Dale Hinds, and a typo in equation (7) corrected 24 November 2000 thanks to Rich Caruana.