Categories of Image Operations

All image operations or functions may be separated into four broad categories: point, neighborhood, morphological, frequency domain, and miscellaneous operations.

Point Operations

Point operations are defined as functions that are performed on each pixel of an image, independent of all other pixels in that image. Operations can be unary, meaning that a single image is modified, or they can be binary, which means that two images are combined in some manner. In certain situations, a point operation may be tertiary, implying that three images are used. Point operations are a very powerful class of image functions that can be cascaded one upon another to create interesting and useful results.

Numeric Transformation Functions (May Use Look-Up Tables)

The most powerful unary point operations are numeric transformations. These operations are used in many situations and for many different purposes. These are also  known by the mechanism that is sometimes used to implement them, the look-up table, or LUT. These operations may be described by the following equation:

T_x,y = function (P_x,y)

or

T_x,y = lut (P_x,y)

where P_x,y is the input pixel value, function is the numeric transform to be applied to each pixel. These functions are sometimes implemented with a look-up table in which case the second form of the equation above applies. T_x,y is the transformed pixel value. The following figure illustrates how this works. For every pixel in an image, P_x,y, the value, i, is used as an index into the LUT. The value, k, in the table at that location becomes the new, or transformed, pixel value. The length of the look-up table is equal to the number of possible pixel values. For 8-bit images, LUTs are 256 entries long, for 12-bit images they are 4096 entries long, etc.

Performing the LUT operation is trivial. The challenge involved in numeric transformations is not in applying the look-up table, but in generating the functional data placed in the LUT. This data is also known as a transfer function and can be written as a mathematical expression. For example, a look-up table that inverts an image to obtain a negative image can be described by the function:

f(x) = max - x

where max is the maximum value that a pixel can have (255 for an 8-bit image) and f(x) is the transformed value.

Simple subroutines can be implemented to perform such transformations.  Many such image operations can be described in mathematical terms and can be implemented as a look-up table. A few additional examples are listed below:

f(x) = x/2             (decrease intensity by half)

f(x) = 0   for x < Tx, (threshold an image)
     = 255 for x > Tx  (where Tx is the threshold value)

f(x) = X^gamma          (gamma modification)

It is sometimes easier to visualize these functions as graphs rather than as equations. Image application programs often provide graphical user interfaces with appropriate controls that allow users to modify the transfer function interactively. The following figures illustrate several typical functions, along with the results of applying each function to an image. The horizontal axis represents each value that an input pixel can have, and the vertical axis shows the output pixel value.

The first example is known as the unity function because the output image is identical to the input image; it is linear with a slope of 1. The second example inverts an image. These transfer functions are unique in that they do not cause any loss of information in the image, which is not necessarily true of other transfer functions.

The third example will increase the contrast of an image. Low intensity values in the input image, say 0 to 31, are all mapped to 0. Similarly, high intensity values, perhaps 224 to 255, all become 255. The remaining input values, 32 to 223, are linearly distributed over the output range of 0 to 255. This line has a slope greater than 1. The transfer function can be written in mathematical form as:

f(x) = 0                          for x < 32

     = (x-32)*(256/(224-32))      for 32 < x < 223

     = 255                        for x > 223

The dark areas of the input image become darker, while the bright areas become brighter. The fourth example has the opposite effect, decreasing image contrast. Its transfer function, which has a slope less than 1, is as follows:

f(x) = x*(max - min)/256) + min

The variables min and max represent the desired minimum and maximum output values. At first glance, it appears that increasing and decreasing contrast are complementary functions, so a user could, for example, increase contrast, then decrease it, and expect the resultant image to be the same as the original one. Unfortunately, this is not true. Numeric transformations such as these, in fact nearly every image processing technique, cause information to be lost. In increasing the contrast, all of the input values between 0 and 31 become 0 in the output image. Attempting to decrease the contrast of that image will not restore the lost data.

The next example illustrates a threshold operation, which means that below some input value, all output pixels are set to the minimum (black), and above some value they all become the maximum value (white). This also illustrates that transfer functions need not be linear, but may take on any shape or curve. The next example, referred to as increasing gamma, is similar to contrast enhancement. Pixel values of 0 and 255 in the input image map to the same output values, but the darker portions have their contrast lowered and the brighter portions have contrast increased. The image, overall, is darkened. Gamma modification is a useful tool when compensating for display monitors characteristics.

Another important use of transfer functions or look-up table operations is to perform color correction and color modification. Just as a photographic processing laboratory can modify a particular color within a picture by altering the exposures of different primary colors, an image manipulation program can control the color balance of a digital picture. The following example shows how this is done. Since a color image is composed of three primary colors (red, green, and blue), each of which can be thought of as a gray scale image, the data of each color plane can be operated on independently.

Two examples of color modification are shown above. The first shows how to make an image less red, while the second will make an image more yellow. In addition to color intensity modification, the same principles are used to control features of brightness and darkness by modifying all three color planes in unison and, as was shown earlier, contrast is controlled by changing the slope of the transfer function. As with other digital image operations, this simple process can greatly alter the look of an image.

Look-up table based transfer functions are frequently implemented in hardware. It should be remembered that such transfer functions affect the entire image. They operate on all portions of an image in the same way. As described earlier, application of a transfer function is not usually reversible.

Arithmetic Operations

Since digital images are composed of numeric values, arithmetic operations can be performed on them. These are classified as binary operations because two images are used to create a resultant image. This can be expressed in the following equation:

C_x,y = A_x,y <operation> B_x,y

where C is the resultant output image, A and B are the input images, and <operation> is the arithmetic operation performed such as addition or subtraction. The values of each pixel in the images, locations of which are denoted by _x,y, are independent of all other pixels in the image.

There are many applications for these functions. For example, consider an image with many superfluous bright and dark pixels, referred to as noise, that detract from the fine detail of the scene. If several, essentially identical, images of the same scene are captured, the noise, which is random, will appear in different locations and with different intensities in each image. This fact can be used to eliminate the noise by adding the images together then dividing by the number of images:

R_x,y = (A1_x,y + A2_x,y + ... + An_x,y)/n

where Al, A2, etc., are the single input images, n is the number of input images, and R is the resultant image. This process is called image averaging. High quality results can be achieved by averaging three or four images, but even two will remove most of the noise.

Two other operators, not strictly arithmetic but very useful, are the minimum and maximum functions. The minimum operator compares the pixel values of two input images and assigns the lesser of the two to the output image. Similarly, the maximum operator assigns the greater of the two values to the output image.

Remember that the pixel values are usually integer numbers within a limited range. For 8-bit images, this range is 0 to 255. If a pixel of one input image has a value of 150 and the other input image is has a value of 130 and if the values are added together, the resultant value of 280 is outside the range of an 8-bit integer. If this operation is allowed, how can the resulting number be represented. The overflow condition may be handled by wrapping the number and maintaining only the least significant bits. In this example, the resultant pixel would be assigned a value of 24 (280 less 256) and a very dark pixel would appear where a bright one was expected.  This is not the result we want.  Similarly, we need to be conscious of possible underflow conditions when subtracting two images.

The results usually need to be clipped to the minimum or maximum allowable pixel values. In the above example, the resultant pixel value of 280 would be clipped to 255. Similar numeric considerations exist for multiplication and division.

Logical Operations

Another set of binary point operations are those involving the logic functions: AND, OR, NAND, NOR, XOR, and XNOR. The equation to implement them is the same as that for the arithmetic operators:

C_x,y = A_x,y <operator> B_x,y

The logical operations are performed at the bit level between the pixel values of the input images in order to generate the resultant image. The following table provides the truth tables for each operator.

Image Compositing (alpha filtering)

It might be necessary to combine, or composite, portions of two images to create the desired resulting image In this situation a third image, called a matte is used.  The matte image contains the alpha or blending value to be used at each pixel location.  The compositing operation is also known as alpha filtering.

C_x,y = (A_x,y * M_x,y) + (B_x,y * (max - M_x,y))

Every pixel in the foreground image, A, is multiplied by its corresponding value in the matte image, M. The same pixel in the background image, B, is multiplied by the inverse of M. The results of these two multiplications are then added together to form the composite image. Where the matte is full intensity (255 for an 8-bit image), all of the foreground image pixel value is used and none of the background is seen. Similarly wherever the matte is black, no foreground image is seen.

The image becomes interesting at the transition between the white and black areas of the matte. If the matte image is not a threshold, i.e., if it does not transition directly from black to white but has shades of gray alone the matte boundaries, the resulting composite image will have a mixture, or merging, of the foreground and background images at those pixels. This mixing is important because it makes the foreground appear naturally integrated with the background.

The actual processing involved in creating a composite image is relatively simple. The difficulty arises in generating the matte. An important technique for merging a foreground and background image is called blue screen compositing. The method for accomplishing this is to capture the foreground subject in front of a brightly illuminated blue screen. Then a matte is generated, based on the blue areas of the foreground image, that allows the subject to be composited over the desired background.

This is the process used during a television news broadcast that permits a meteorologist to walk in front of maps and satellite photographs with apparent ease. In reality, the person is standing in front of a blue (or sometimes a green) screen, while the person's image is composited with the desired background. In television, this technique is called chroma keying.