Convolution is everywhere around us. For example, when we try to take an image through a camera and the image gets blurred because of shaking of hands, the blur introduced can be explained as a special filter/kernel (blur in this case) is convoluted over the actual image and gave the ultimately blurring effect. The convolution of a temporal or spatial signal with another signal produces a modified version of the initial signal. The modified signal may have better feature representation than the original signal suitable for a specific task. For example, by convolving a grayscale image as a 2D signal with an Edge-Deception filter (namely Sobel filter), the edges in the original image can be highlighted. Edges in an image can correspond to object boundaries, changes in illumination, changes in material property, discontinuities in depth, and so on, which may be useful for several applications.

Step-by-Step Execution

Any digital image is a 2D discrete signal. The convolution of an \( N \times M \) 2D image \( x(n_1, n_2 ) \) with a 2D image-processing filter \( h(n_1 ,n_2 ) \) is given by \begin{equation} \label{eq:convolution} y(n_1, n_2 ) = \sum_{k_2=0}^{N-1} \sum_{k_1=0}^{M-1} x(k_1, k_2 ) \, h(n_1-k_1, n_2-k_2 ) \\ \text{where} \; \; 0 \leq n_1 \leq N-1 \, \text{,} \; 0 \leq n_2 \leq M-1 \end{equation}

The image-processing filters work on a grayscale image’s (2D) signal to produce another image (2D signal). In cases of multi-channel images, one must process each image channel as a 2D signal or convert the image into a grayscale image.

To keep the length of the output image the same as that of the input image, image padding is used, i.e., the original image has been zero padded. As per the process of convolution, the filter is flipped and slid over various areas of the original image, and the convolution sum is computed at each coordinate point. This is mainly because filter is of smaller size. The figure below shows step-by-step execution of convoluting a filter over an image.

Now as the basics of convolution is understood, we now give the example of some common filters.

Mean Filter

The Mean filter or Average filter is a low-pass filter that computes the local average of the pixel intensities at any specific point. One such typical kernel is \( \left[ \begin{smallmatrix} 1/9 & 1/9 & 1/9 \\ 1/9 & 1/9 & 1/9 \\ 1/9 & 1/9 & 1/9 \end{smallmatrix} \right] \)

In this implementation, we are using python, OpenCV and Scipy to perform basic operations on the image, such as reading the image, converting the image from RGB format to grayscale format, and finally the convolution operations. Both Scipy and OpenCV are an open source packages that has a rich set of methodologies for image processing. Readers are encouraged to explore more on Scipy and OpenCV here and here respectively. The example below show the use of this filter using Scipy functions.

    import cv2
    import matplotlib.pyplot as plt
    import numpy as np
    from scipy.signal import convolve2d
    img = cv2.imread('lena.jpg')
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    plt.imshow(gray,cmap='gray')
    mean = 0
    var = 100
    sigma = var**0.5
    row,col = 512,512 # This is the dimension of the input image
    gauss = np.random.normal(mean,sigma,(row,col))
    gauss = gauss.reshape(row,col)
    gray_noisy = gray + gauss
    plt.imshow(gray_noisy,cmap='gray')
    ## Mean filter
    Hm = np.array([[1,1,1],[1,1,1],[1,1,1]])/float(9)
    Gm = convolve2d(gray_noisy,Hm,mode='same')
    plt.imshow(Gm,cmap='gray')
                                 

Notice how the image gets blur after applying the mean filter.

Median filter

Median filter are used to remove salt and pepper noise. This type of noise appears as black and white pixels in the images and is generally caused by sudden disturbances while capturing the images. Median filter remove such noise as it replaces each pixel in a neighborhood with the median pixel intensity in that neighborhood based on the filter size. The example below show the use of this filter using Scipy functions.

    # First we artificially create some salt and pepper noise
    ## The idea is to randomly replace original pixels with back and white pixels- I.e.,
    ## If the random value is zero we will replace the image pixel with a low value of 0 that corresponds to a black pixel
    ## If the value is 20 we will replace the image pixel with a high value of 255 that corresponds to a white pixel
    np.random.seed(0)
    gray_sp = gray*1
    sp_indices = np.random.randint(0,21,[row,col])
    for i in range(row):
        for j in range(col):
            if sp_indices[i,j] == 0:
                gray_sp[i,j] = 0
            if sp_indices[i,j] == 20:
                gray_sp[i,j] = 255
    plt.imshow(gray_sp,cmap='gray')

    ## Now we want to remove the salt and pepper noise through a Median filter.
    ## Using the opencv Median filter for the same
    gray_sp_removed = cv2.medianBlur(gray_sp,3)
    plt.imshow(gray_sp_removed,cmap='gray')
                                 

Notice how the image gets corrected after applying the median filter.

Image Sharpening using Gaussian Filter

The Gaussian filter is a modified version of the Mean filter where the weights of the impulse function are distributed normally around the origin hence, the intensity falls in a Gaussian fashion away from the origin. Gaussian filters help to reduce noise by suppressing the high-frequency components which comes at the cost of final image being blurred, called Gaussian blur.

For sharpening an image, the original image is first convolved with the Gaussian filter to produce an image that has Gaussian blur. This image is then subtracted from the original image to get the high-frequency component of the image. A small portion of the high-frequency image is added to the original image to improve the sharpness of the image.

    Hg = np.zeros((20,20))
    for i in range(20):
        for j in range(20):
            Hg[i,j] = np.exp(-((i-10)**2 + (j-10)**2)/10)

    plt.imshow(Hg,cmap='gray')
    gray_blur = convolve2d(gray,Hg,mode='same')
    gray_high = gray - gray_blur ### note that the order is important, i.e., this operation is not associative
    plt.imshow(gray_high, cmap='gray')

    gray_enhanced = gray + 0.025*gray_high
    plt.imshow(gray_enhanced,cmap='gray')
                                 

Sobel Filters for Edge-Detection

Edges are important features for an image and help detect objects and local changes within an image. Edges are generally present on the boundary between two regions in an image and are often the first step in retrieving information from images. The Horizontal Sobel filter \( H_x = \left[ \begin{smallmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{smallmatrix} \right] \) detects edges in the horizontal direction, whereas the Vertical Sobel filter \( H_y = \left[ \begin{smallmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{smallmatrix} \right] \) detects edges in the vertical direction. Both are high-pass filters since they attenuate the low frequencies from the signals and capture only the high-frequency components within the image. The example below shows the logic required to convolve the image with the Sobel filters.

    Hx = np.array([[ 1,0, -1],[2,0,-2],[1,0,-1]],dtype=np.float32)
    Gx = convolve2d(gray,Hx,mode='same')
    plt.imshow(Gx,cmap='gray')

    Hy = np.array([[ -1,-2, -1],[0,0,0],[1,2,1]],dtype=np.float32)
    Gy = convolve2d(gray,Hy,mode='same')
    plt.imshow(Gy,cmap='gray')

    G = (Gx*Gx + Gy*Gy)**0.5
    plt.imshow(G,cmap='gray')
                                 

Code on Git

The above code can be found on Github.

Source

This tutorial is influenced by Chapter 3, "Convolutional Neural Networks", from the book "Pro Deep Learning with TensorFlow", by Santanu Pattanayak