The amplitude of the sampled signal is a continuous value. Digital systems represent values using a finite number of bits. Hence the amplitude has to be approximated and expressed with these finite number of bits. This processes is called quantization. The quantized values are integer multiple of a quantity called the quantization step. An example of two bit (or equivalently four level) quantization is shown in Fig. 8.1. For the quantizer , where is the maximum voltage (peak-to-peak) that can be expressed within an error of .
Quantization distorts the sampled signal affecting both the amplitude and spectrum of the signal. This is evident from Fig. 8.1 for the case of a two bit four level quantized sine wave. The amplitude distortion can be expressed in terms of an error function , which is also called the quantization noise. Here is the output of the quantizer. The variance of quantization noise under certain restricted conditions (such as uniform quantization) is . The spectrum of quantization noise extends beyond the bandwidth of (see Fig. 8.2). Sampling at the Nyquist rate () therefore aliases the power of the quantization noise outside back into the spectral band of . For radio astronomy signals, the spectral density of the quantization noise within can be considered uniform and is (assuming uniform quantization). Reduction in quantization noise is hence possible by oversampling (i.e. ) since it reduces the aliased power. For example, the signal to noise ratio of a digital measurement of the correlation function of s(t) (see Section 8.5) using a Nyquist sampling and a two bit four level quantizer is 88% of the signal to noise ratio obtained by doing analog correlation for Nyquist sampling and 94% if one were to sample at twice the Nyquist rate.
The largest value that can be expressed by a quantizer is determined by the number of bits () used for quantization. This value is for binary representation. The finite number of bits puts an upper bound on the amplitude of input voltage that can be expressed within an error . Signals with amplitude above the maximum value will be `clipped', thus producing further distortion. This distortion is minimum if the probability of amplitude of the signal exceeding and is less than . For a signal with a gaussian amplitude distribution this means that , being the standard deviation of .