Machine Vision Handbook

Machine Vision is related to, but distinct from Computer Vision, Image Processing, Artificial Intelligence & Pattern Recognition. The subject is concerned with the engineering of integrated mechanical-optical-electronic-software systems for examining natural objects and materials, human artifacts and manufacturing processes, in order to detect defects and improve quality, operating efficiency and the safety of both products and processes. It is also used to control machines used in manufacturing. Machine Vision necessarily involves the harmonious integration of mechanical handling, lighting, optics, video cameras, image sensors (visible, UV, IR and X-ray sensor arrays, as well as laser scanners), digital, analogue and video electronics, signal processing, image processing, computer systems architecture, software, industrial engineering, human-computer interfacing, control systems, manufacturing, existing work practices and quality assurance methods. Machine Vision is not a scientific endeavour; it is a branch of Systems Engineering. Hence, consideration of application requirements pays a key role in the design of practical vision systems. The basic philosophy of Machine Vision is set out in this chapter and the structure of this book is outlined. This is a pragmatic, empirical, experimental subject and is not unduly burdened by the search for mathematical rigour, or theoretical purity. There is one simple design maxim: if it works, use it! This is justified here by consideration of lessons learned from four practical application studies. Later in the book these same ideas will emerge many times over.

This chapter is concerned with the design of artificial vision systems that attempt to emulate one particular human activity: inspecting and manipulating highly variable natural objects and human artifacts. During the last quarter of the twentieth century and the first decade of the twenty-first century, Machine Vision, evolved from an exotic technology, created by academics, into one that is of considerable practical and commercial value, and which now provides assistance over a wide area of manufacturing industry. Hitherto, Machine Vision has been applied extensively in industry to tasks such as inspecting close tolerance engineering artefacts, during or shortly, after, manufacture. However, Machine Vision technology can also be applied to those areas of manufacturing where natural materials and other highly variable objects are processed. Our discussion in these pages is aimed at those areas of manufacturing where wide tolerances are encountered, as well as activities, such as agriculture, horticulture, fishing, mining, etc., where similar conditions apply. However, we should not lose sight of the fact that certain so-called high-precision industries are plagued by significant degrees of uncertainty. For example, the electronics industry is renowned for working with high-precision components (integrated circuits, printed circuit boards, etc.). However, it is also concerned with highly variable entities such as solder joints, flexible cables and components with flying-lead connectors (resistors, capacitors, diodes, coils, etc.). This chapter is relevant to applications such as these, as it is to examining fruit, vegetables, animals, fish, etc.

Since this is a book about artificial vision, it may seem strange that we include an essay on vision in the natural world. The reason is simple: we need to convince the reader that there is no single model for vision that is ideal for all circumstances. Our own visual system, sophisticated though it is, would not suit a housefly, eagle, owl, chameleon, or even a horse. Each of these animals has managed to thrive within a specific type of environment, for far longer than human beings have walked on two legs. Even though it is very limited in its visual capability compared to human beings, a frog has a vision system that has served it well for many millions of years. In many instances, an industrial Machine Vision system is like a frog: both can do certain well-defined tasks, without attempting to achieve the subtlety of discrimination and the high level of intelligent scene analysis that a human being can accomplish. In nature, there are four scenes that nearly every organism has to recognise: food, threat, mate, and shelter. This is not so very different from an inspection system that is merely intended to distinguish between ‘good’ and ‘faulty’ products. Animals do, however, have to cope with highly variable environments and cannot change the viewing conditions as a vision engineer can do to make his/her creation cheaper, faster, or more reliable. So, our task is usually simpler than designing a machine as complex as a frog. We must never make the mistake of thinking that only we see the world as it really is. We are only sensitive to a narrow range of optical wavelengths. We cannot sense polarisation, whereas bees and appropriately designed machines can. We cannot sense very low light levels that some animals have to use. Why should we want to replicate a system like ours that is confused by numerous visual illusions? The natural world has produced many different models for vision. For example, there are over 40 different types of eye. The compound eye of the insects and our own eye are obviously very different. Even the eye of a relatively close relation, the cat, is significantly different from our own, having a vertical pupil in bright light. Some animal eyes move the lens to focus on objects that are moving along the line of sight, rather than alter the shape of the lens, as our own eyes do. There is nothing particularly special about our colour vision either. Mantis shrimps, for example, have 12 different colour pigments in their eyes. Spiders are not content with just two eyes as we are; they have eight! This variety does more than present us with a collection of interesting curiosities; it provides inspiration! Vision engineers can benefit from learning about non-human vision systems. Perhaps the most important point of all is that vision engineers should have the confidence to be different; we should not be constrained always to try and emulate human vision. Nature isn’t! Although it is immensely powerful, human vision is sometimes ineffective, and in other cases grossly over-sophisticated, for the humble tasks required of industrial vision systems. It is always vain to do with more what can be done with less. So, if frog vision works, use it!Vision is pervasive in the animal kingdom and is a sensory capability that is variously relied upon to find reproductive mates, suitable food, and/or shelter while avoiding predators or other dangers. The receptor organs (e.g., eyes) and associated visual system not only provide the basis for visual perception of the surrounding environment but also drive and guide the visuomotor responses (e.g., locomotion, eye movements, reaching to grasp, etc.) in reaction to changes or objects of interest in the world. Although animal vision has intrinsic limits that vary from species to species as well as different properties (e.g., colour vision) that may arise for specific purposes in the course of evolution, its main characteristic is that it represents a generalised adaptation that can deal with a diverse range of problems and challenges. In contrast, machine vision is narrowly designed for a particular purpose or function and its related systems are normally highly constrained. This chapter offers a broad summary of human and animal vision, with an emphasis on vertebrates, paying particular attention not only to its basic design properties and limitations but also the sheer diversity of biological solutions to the problem of perceiving the world and responding appropriately. The hope is that such an examination of vision from a variety of perspectives will inform and inspire the future design and engineering of machine vision.

Telecentric lenses provide constant perspective and large magnification depth of field. For this reason, they are often well suited for gauging and a variety of other demanding Machine Vision applications. This is especially true when viewing objects that have variable depth and/or lateral position. Lighting issues relating to the use of telecentric optics for viewing and lighting are discussed, together with their ability to provide sensitive detection of defects in clear or flat and shiny materials. Telecentric lenses can view objects, with constant perspective, anywhere within a cylindrical volume. On the other hand, hypercentric lenses can view objects within a conical region, whose apex is in front of the lens. (A fish-eye lens views objects within a conical region, whose apex is behind the lens.) As a result, hypercentric lenses are able to view the top and sides of an a cylinder simultaneously.By adding a spacer between certain types of hypercentric lens and the camera, it is possible to view inside a bore, even when the viewing aperture, at the camera end of the bore, is very small. Both telecentric and hypercentric lenses are only able to work effectively if the front refracting element is larger than the object being viewed. Manufacturing large conventional (i.e. curved-surface) lenses is very expensive. Fresnel lenses provide a convenient alternative. There are three common types of optical component made by forming numerous very small grooves and/or mounds on a plane surface. Fresnel lenses employ very fine prismatic surface features on a thin plane plate. Micro-lens arrays consist of numerous small lenses (lenslets) placed close together on a plane substrate. In both of these cases, the optical formulae explained in Chap. 5 are applicable, because the surface features, while small, are still much larger than the wavelength of light. On the other hand, Diffraction (or Binary) optical components have such small surface features that they cause interference due to diffraction; refraction usually plays no significant part in the operation of these devices. Some hybrid refractive-diffractive lenses are made but these are uncommon.

Our everyday experience does not alert us to the need to choose lamps for Machine Vision carefully. The human eye-brain complex is able to compensate for slow variations in both the colour and intensity of ambient lighting, without our ever becoming aware that any change has taken place. As a result, people often remark with surprise how bright the room suddenly appears when a light bulb has been changed. Unlike people, machines are very sensitive to variations in both colour and brightness. Incandescent filament bulbs and fluorescent tubes are used in the home and office, while mercury and sodium discharge lamps are commonly employed in street lighting and in factories and warehouses. For many years, advertising has relied heavily on “neon” displays. In addition, there is a new variety of lamps available, including a wide range of LED-based lighting units, cold cathode tubes, compact fluorescent bulbs and plasma panels. Arc lamps have been used in specialised scientific and industrial applications. For Machine Vision, the following properties of lamps are important:3D radiation patternAbility to control the lamp brightnessBrightness, including long-term variations due to lamp ageingDriver (AC /DC, frequency, closed-loop control)Efficiency, measured by the ratio of the optical emission power to the electrical power consumedElectrical noise generatedHeat producedInfra-red content of the beamLife-time, measured simply by how long before the lamp becomes so dim as to be useless, or stops emitting altogetherLong-term degradation of emitted powerPower-line rippleShapeShort-tem stability of emitted powerSpectrum (This requires a much more detailed analysis than simply stating the “colour”.)Ultra-violet contentThe features of the major types of lamp are discussed. LEDs are almost ideal for many industrial applications and in recent years, there have been major advances in the design of LED arrays for Machine Vision. It is possible to build arrays in a variety of shapes. An array may contain LEDs of many different and precisely known colours. LEDs are efficient and stable over a long period. The lighting pattern can be adjusted dynamically, under software control.

The image acquisition sub-system consists of one, or more, light sources, lenses and possibly mirrors, filters, prisms, polarisers, etc. It also contains one, or more, opto-electronic transducers (i.e., cameras and/or scanners). Simply knowing what “building bricks” are available is not enough; we must be able to construct an integrated system. Object presentation and transport are important and require that we address mechanical handling issues: grasping the object, as well as moving it before, during and after image capture. The type of viewing background is important, as are numerous seemingly mundane tasks, such as eliminating ambient light, building vibration-proof optical mountings and keeping optical components clean, dry and cool. In most applications, the placement of the camera relative to the object under inspection and the lights is critical; the image quality derived from the same object may vary from very poor to excellent, depending on the lighting-viewing conditions. This chapter introduces the catalogue of Lighting Viewing Methods (LVMs) in Chap. 40. The latter presents qualitative descriptions of a variety of “front end” configurations for a vision system. There can be no rigid formula-based approach to designing a lighting-viewing system. This does not, in any way, diminish the need for sound mathematical analysis, where it can be applied appropriately (e.g., lens design). There can be no equation to tell us when it is most appropriate to use, say, diffuse, or dark-field illumination, monochromatic or polarised light, or a combination of these. The system designer must use his/her judgment and experience. Most importantly, he/she must look at the widget and its surroundings! Two diagrammatic representations of the spatial relationship between the lights and camera are presented here. These are useful for recording and summarising the results of experimentation, which the author asserts is the only practical approach to designing the image acquisition sub-system.

This chapter gives an overview of why scanning systems are used rather than more straightforward imaging systems. Different classes of scanner and optical systems are described along with the tradeoffs in performance that have to be made when balancing different performance requirements. Detailed performance of a particular parabolic mirror, large area scanner is presented.

Automated X-ray Inspection (AXI) is useful for two types of application: detecting foreign bodies, particularly in food, and examining the internal detail of safety-critical products. At first sight, AXI merely seems to be a straightforward extension to Machine Vision but there are several complications. There is only a limited scope for adjusting the spectrum of an x-ray beam. It is not easy to build x-ray mirrors and lenses, so we cannot effectively focus, or collimate, an x-ray beam. Since an x-ray ray tube generates a conical beam, the images it forms are complicated. The diverging beam introduces parallax errors. Furthermore, objects near the x-ray source appear to be bigger than those further away. Features along the beam are superimposed, creating a jumble of detail that is difficult to analyse. AXI has its own set of special features that distinguishes it from other areas of Machine Vision. One particular point emphasises this. X-rays present a clear danger to health, so there is a need for good radiation screening and a variety of other safety measures. Mathematical and software modelling of AXI systems can often help in devising effective image analysis algorithms. These are simple and effective when inspecting close-tolerance accurately aligned artifacts. In this case, a point-by-point comparison of the test image and a reference image may be effective. As the image variability increases, more complex inspection algorithms may be required, often involving the use of sophisticated AI methods. The chapter concludes with a discussion of twin-beam x-ray systems that can improve the detection of defects and foreign bodies.

Illumination-invariant image processing is an extension of the classical technique of homomorphic filtering using a logarithmic point transformation. In this chapter, traditional approaches to illumination-invariant processing are briefly reviewed and then extended using newer image processing techniques. Relevant hardware considerations are also discussed including the number of bits per pixel required for digitization, minimizing the dynamic range of the data for image processing, and camera requirements. Three applications using illumination-invariant processing techniques are also provided.

A discussion of image processing must begin by considering how images are represented. The most commonly used format is that of a 2-dimensional array for monochrome images, while colour images are usually represented by three such arrays. The array format is generated by most cameras. While other representations present advantages in certain, rather restricted situations, they must usually be derived from an image array. Grey-scale image processing functions can be arranged into several major categories. The simplest, called monadic operators, process one image at a time, by performing a fixed intensity mapping from each of its pixels to the pixel at the corresponding position in the output image. Dyadic functions operate on two images; the intensities of pixels at corresponding addresses are combined to calculate a value for the pixel at the same address in the output image. Local operators process a single image. In order to calculate the intensity at a point (i,j) in the output image, several pixels surrounding point (i,j) in the input image are combined. For example, the average, maximum, or minimum intensity within the 3∗3 pixel neighbourhood surrounding point (i,j) might be computed. Such a process is repeated for all (i,j). Numerous filtering functions can be invented by extending the concept of “neighbourhood”. The function that is applied to the pixel values within a given neighbourhood can be linear or non-linear. The latter gives rise to a useful family, given the name morphology. Another group is based on the application of a monadic function that is derived in some way from the intensity histogram, or other statistical features, of the input image. Another, varied family is based on Fourier, and other integral transforms of the 2-dimensional intensity function. Image warping and geometric transforms form another family of grey-scale image processing function. Some grey-scale operators have obvious counterparts for binary images. However, the latter also require functions such as object counting, feature detection, measurement of position, orientation, size and shape. Binary image processing operators do not fall neatly into well-defined categories but can often be described functionally by drawing on everyday concepts such as that of skeleton, rubber band, compass, ruler and our intuitive understanding of shape fitting. The processing of colour images is not covered here as it is discussed in detail elsewhere in this book (Chap. 16). For the moment, let it suffice to say that grey-scale operators can be applied to the individual RGB, HSI or CMYK components of a colour image.The purpose of this chapter is to outline some of the basic techniques used in the development of industrial machine vision systems. These are discussed in sufficient detail to understand the key ideas outlined elsewhere in this book. In the following discussion we will frequently indicate the equivalent QT operators for the vision techniques described. (QT is an interactive image processing system and is described in Chap. 21. Chap. 41 describes the function of each QT operator.) QT commands appear in square brackets. In certain cases, sequences of QT commands are needed to perform an operation and these are similarly listed.

Colour is an important feature of many industrial artefacts. Moreover, a very large proportion of packaging materials are coloured. The inspection of coloured objects is therefore of considerable importance. In addition, natural products and raw materials are often inspected, graded or sorted on the basis of their colours. As so often happens in Machine Vision, the prime requirement is for verification rather than unassisted recognition. Sometimes the range of allowed colours can be specified quite precisely. On other occasions, it has to be learned from carefully selected samples. Our scope for flexibility is limited by the need to use commercially available image sensors. These almost invariably provide RGB output signals. We shall therefore concentrate primarily on pixel-by-pixel recognition of colours based on RGB data. Several techniques exist, including “brute force” methods that rely on standard Pattern Recognition algorithms, while others are based more firmly on Colour Science. Multi-spectral data allows a more refined approach to colour recognition than is possible with RGB. It requires more sophisticated image acquisition methods but can use well-established Pattern Recognition algorithms. Our goal is to provide image segmentation techniques that can be used to identify regions according to “standard” colours (“red,” “yellow,” “green,” etc.) and colours that are specific to one particular situation (i.e., product or factory). A colour recognition program generates one or more binary images that can then be measured using the techniques described in Chaps. 14 and 18–20. In this sense, colour recognition serves simply as a means of segmenting an image.

When a vision procedure has been devised, perhaps using an interactive image processing system, it must be engineered before it is ready for use in a factory floor system. A prototype procedure found in this way has probably not been optimised for speed, compatability with other parts of the system, or ease of implementation. It may not even be the most effective procedure of the same general type. For example, it might employ one image filter, such as an edge detector, whereas another would perform just as well and be much easier to implement. This chapter explores a range of grey-scale and binary image processing functions, demonstrating that many algorithms present parallel options. It is well to investigate every stage of an image processing procedure to see if it can be replaced by a more convenient, or more efficient, alternative. Each part of an image processing procedure must be compatible with all of the others. Sometimes, the way an image is represented can make a large difference in the ease of implementation, or speed of execution. On some occasions, a mathematically rigourous procedure (algorithm) can be replaced by a much more convenient “rule of thumb” (heuristic), without serious loss of performance. If used sensibly, heuristics may make a system work in a very much more cost-effective way than would otherwise be the case. Heuristics can sometimes replace algorithms, with little or no loss of performance. In some instances, heuristics can greatly improve the speed compared to an algorithm, so that slightly reduced accuracy is acceptable. It is important to bear in mind what the real objective of a vision system is. Suppose we wish to build a machine to count large numbers of objects like dice. Should we count white rectangles, representing the top surfaces of the dominoes, or black spots and then divide the result by three? On a sufficiently large sample, these two procedures produce similar results. Thinking in terms of statistical performance is an important part of designing an effective vision system. Heuristics can often be made to cope safely with situations that have never been seen before. A “catch all” rule, such as “If all else fails, switch off the power,” can guarantee the safe operation of a complex plant, even if it occasionally reduces production efficiency unnecessarily. Emotional objections to procedures that are know to fail occasionally can sometimes be very strong but it is well to consider the frequency of such an event. However costly a mistaken decision is, its overall significance may be small if it occurs very infrequently. Heuristics can often be biased deliberately to improve safety, or to improve productivity. One particularly useful heuristic rule for an automated inspection system is to refer doubtful cases to a human being for examination and/or rework. There is another very useful heuristic that we should always apply: try to improve the quality of the “input” images by adjusting the lighting-viewing conditions!

This chapter considers how best to locate 2D object shapes in digital images. Simple paradigms such as boundary tracking followed by analysis of centroidal shape profiles lack robustness and are incapable of coping with occlusion, breakage or gross shape distortions, whereas the Hough transform readily overcomes these problems as an integral property of its voting schema. This is an important advantage, especially in practical situations such as those where lighting is non-ideal. In addition, the Hough transform is excellent at locating straight lines, circles, ellipses and other well defined shapes: it can even locate arbitrary shapes that are defined by so-called R-tables. The method also has the advantage of being able to make sense of patterns of point features, and is far less computation intensive than the maximal clique graph matching method often used for this purpose. Being a form of spatial matched filter, the Hough transform has high signal-to-noise sensitivity, and for simple shapes is highly efficient, but when applied to arbitrary shapes of variable orientation, size and individual variability, measures have to be taken to limit its computation. Overall, the Hough transform provides a very useful, robust search technique for locating known types of object in digital images.

Morphology applies certain simple rules and shapes, such as squares, circles, diamonds, cubes and spheres, to process images. The objective is usually to identify features of interest in images as a prelude to performing high-level inspection, or machining, functions. Navigating around complex objects is facilitated using ‘anchor point’ features detected by morphology. The chapter begins by defining four basic morphology operations for binary images: erosion, dilation, opening, and closing. The application of these operations to analysing images from real-world vision tasks is discussed, as are techniques for fast implementation. Further operations akin to morphology are also described, including skeletonisation, hit-and-miss transform, and binary template matching. The extension of binary morphology into grey-scale morphology is also described and illustrated.

In a series of papers starting in 1994, Waltz described a powerful and unconventional method of implementing many important neighbourhood image processing operations. These techniques, taken together, have been given the acronym SKIPSM (Separated Kernel Image Processing using finite-State Machines). Note that “Separated” here does NOT imply that the operations must be linearly separable. The resulting implementations are almost always faster, and sometimes much faster, than the same operations implemented in conventional ways on the same computational platforms. This chapter presents the basic concepts of image processing using FSMs (finite-state machines), lists many specific image-processing operations which have been implemented with FSMs, gives examples of several typical operations (along with the C-code used to implement them), outlines a completely routine procedure for generating and optimizing SKIPSM C-code for a large class of image operations and neighbourhood sizes, and concludes with some execution-speed comparisons which support the claims of fast execution.

QT is the latest in a series of interactive image processing tool-kits for prototype development of Machine Vision systems. Experience gained over more than 35 years has shown that a command-line system with short mnemonic function names is an effective tool for developing vision algorithms. Parameter lists are also required for some functions. When the user types a command, such as neg (negate), thr(89,127) (threshold) or lpf (low-pass filter, blur), the operation is performed immediately and the result is displayed, together with the image before processing. When running on a modern computer, typical response times for processing an image of good resolution are rarely more that one second. Measurements are obtained in an obvious way. For example, to count the number of distinct objects (blobs) in a binary image, the user types x = cbl, while the command y = avg computes the average intensity in a grey-scale image. Binary, grey-scale and colour images are all processed by QT, which is built on top of MATLAB™. The advantage of QT over MATLAB lies in its greater ease of use. The command names are short but most importantly, the user does not have to keep track of source and destination images. QT always processes the image displayed on the left of the computer screen. It then replaces this with the result of the operation selected by the user and displays the input image for the last processing step on the right. In addition, to image processing commands, QT provides facilities for a range of other operations, including image capture from local or remote cameras, controlling external devices, speech synthesis, on-line documentation, consulting the Catalogue of Lighting-Viewing methods (Chap. 40). Any MATLAB function can be invoked from QT and programming new functions is straightforward. Many of the illustrations in the book were prepared using QT and several chapters exploit its ability to express image processing algorithms succinctly. Since it is a prototyping tool, QT is not fast enough for any but the least demanding factory applications. However, it is a powerful exploratory tool in the hands of an experienced user. On numerous occasions, QT or one of its predecessors, has been used to discover/invent algorithms very quickly in response to new visual inspection and control applications. QT has been interfaced to Prolog to produce an even more powerful language (PQT) that facilitates intelligent reasoning about images (Chap. 23).

This chapter will detail a free image analysis and software development environment for machine vision engineers. The environment provides high-level access to a wide range of image manipulation, processing and analysis algorithms (over 300 to date) through a well-defined and easy to use graphical interface. Users can extend the core library using the developer’s interface, a plug-in, which features automatic source code generation, compilation with full error feedback, and dynamic algorithm updates. The chapter will also discuss key features associated with the environment and outline the advantages in adopting such a system for machine vision application development. (http://www.cipa.dcu.ie/)

Some potential applications of Machine Vision, particularly those involving highly variable objects (Chap. 2), require greater reasoning power than standard image processing languages can provide. This chapter is based on the premise that an intelligent vision system can be made, by the simple expedient of embedding image manipulation and measurement routines, such as those provided by QT (Chap. 21), within the Artificial Intelligence language Prolog. The great strength of Prolog is its ability to perform symbolic reasoning. It is a declarative language. That is, it operates by searching automatically for a solution to a problem, given a description of (part of) the world. The user specifies the desired characteristics that a solution must have. Prolog's search engine then hunts for the specific conditions that must exist to obtain a solution. If the user chooses, or programs Prolog appropriately, it will search for multiple solutions. Unlike a conventional imperative language, Prolog does not have any instructions. Instead, it accepts, a series of statements that are then tested individually for their veracity. It does not possess any of the usual flow-control tools, such as IF ... THEN, FOR, WHILE, CASE, etc. Instead, it relies on back-tracking and recursion. The former allows temporary assumptions that have been made while searching for a solution to be revised before continuing the search. A software interface interconnecting QT with a modern implementation of Prolog (SWI-Prolog) was designed. The result, called PQT, allows image processing operations to be invoked procedurally within a Prolog program, or executed interactively, as in QT. Image measurement values can be used to define temporary values for (i.e. instantiate) Prolog variables. Using PQT, it is possible to define general spatial relationships (left, above, concentric, parallel, etc.), geometric features (T, X, U, V), shape characteristics (round, curved, elongated), colour and texture (yellow, smooth, spotty). These would be much more difficult to define and use in a standard programming language. PQT can also be used effectively to control external peripheral devices but not in real time. PQT inherits Prolog's limited ability to analyse and extract meaning from natural language sentences, such as spoken commands. While PQT was designed to perform vision-based tasks, such as inspecting food products, it is also capable of the high-level reasoning needed to incorporate them into a well-balanced meal.

Human beings are hard-wired to discover and categorize patterns as a way of understanding the world. Humans can easily read poorly handwritten text, parse and understand human speech, recognize faces in a crowd, classify people as old or young, etc. However, it is difficult for a machine to solve these kinds of perceptual problems. In this chapter, we explore some of the concepts and techniques of machine-based pattern recognition, including statistical-based approaches and neural network-based approaches. We illustrate the concepts with case studies taken from image and video processing.

Field programmable gate arrays (FPGAs) offer a convenient and flexible platform on which real-time machine vision systems may be implemented. This chapter discusses how FPGA based design differs from conventional software design for implementing image processing algorithms. Simply porting from software to hardware can give disappointing results. The process of transforming an algorithm for mapping it onto an FPGA is described in some detail, illustrating how a range of low-level image processing operations may be efficiently implemented. Systems integration issues such as interfacing, debugging and tuning are also discussed.

A vision system for gauging two types of automotive parts is described in this article. One of the part types is a power steering connector, in which the depth and width of the groove, and the distance between the start of the groove and the end of the power steering line are gauged. For the second type of part, a crimped connector attached to a brake hose, measurements of interest are the two diameters of the crimp and the bell length, where the hose is inserted into the connector. A standard video camera is used to acquire the image of a back-illuminated part which is digitized and captured with a frame grabber. The basic hardware to accomplish the gauging tasks consists of a standard video camera, light source, frame grabber and industrial personal computer. In order to minimise hardware costs, a standard 50 mm C-mount camera lens and extension tube was used with the video camera. Consideration had been made to use more expensive telecentric optics so that parts placement would not cause a change in magnification with a resulting loss of accuracy. With the 50 mm lens, however, magnification effects were lessened due to the greater standoff distance between camera and part. Combined with a modern personal computer, rapid image acquisition and analysis provides 100% inspection at full production rates. Since the gauging rate exceeds the production rate by a significant factor, a single computer and frame grabber with camera multiplexer can process data in real time from up to four measurement stations simultaneously.

During the manufacture of audio equipment, there is a need to use a robot to handle small cylindrical coils with flying leads (i.e. attached at only one end). A minimum of five cameras is needed: one overhead and four others, viewing the sides of the coil. High-contrast images can be obtained using back-lighting. Simple thresholding can then produce “clean” binary images. From the plan view of the coil, it is possible to locate the wire ends in the horizontal, [X,Y], plane.The reasoning needed to choose the best side view is well suited to PQT (Chap. 23) but QT (Chap. 21) would find this more difficult. From the side view, it is possible to derive one horizontal (X or Y) coordinate and the vertical coordinate (Z). Since each side view uses a different coordinate system, it is necessary to translate the side-view results into a “universal” coordinate system. Before this can be done, the system must be calibrated. (Appendix E) By combining overhead and side views, it is possible to locate the ends of both wires in 3-dimensional [X,Y,Z], space. Locating another point close to the end of a wire allows its orientation to be estimated. This is repeated for the plan and side views, giving two angular measurements, [θ,φ]. Five parameters [X,Y,Z,θ,φ] are needed by a robot to grasp the wire. (Again, a change of coordinate system is needed.) The ambiguity caused by the fact that there are two loose wires can be resolved easily by reasoning in PQT. The task of resolving real or apparent entanglement of the wires is discussed. This requires the interpretation of pairs of images and is, once again, suitable for Prolog-style reasoning in PQT. In this application, there is a high degree of uncertainty due to the flexible nature of the wires. While the image processing is trivial, the use of Artificial Intelligence techniques is essential. The matrix calculations needed to perform the coordinate-axis transformations are well suited to MATLAB and hence QT and PQT.

A common machine vision task is to verify that a product has been properly fabricated or assembled. In the following chapter, a vision system is described for confirming that a type of gear has been machined properly. The main problem in this application is the relatively large depth of the gear which results in a more complex image than the desired silhouette of the part. The resulting image shows a portion of the inner wall due to the varying magnification because some points on this wall are closer to the lens than others. Employing telecentric optics, however, greatly simplifies the problem since only light rays parallel to the optical axis create the image so that a good silhouette of the part is obtained. Gear teeth can then be isolated using mathematical morphology techniques to verify that the part has been properly broached.

The Crosshead Inspection System (CIS) utilizes machine vision technology for on-line inspection of a crosshead which is a diesel engine component. The system includes three functional modules:1.Part handling subsystem presents parts for inspection and accepts or rejects them based on signals from the image analysis software.2.Image acquisition hardware consisting of optics, light sources and two video cameras to collect images of inspected parts.3.Image analysis software to analyze images and send pass/fail decision signals to the handling subsystem.The CIS acquires and inspects two images of each part. The upper camera generates an image of the part’s top surface, while the lower camera generates an image of the so-called “pockets” on the part bottom. Both images are acquired when a part-in-place signal is received from the handling system. The surface inspection camera and light source are positioned at opposed low angles relative to the surface. Irregularities manifest themselves as dark regions on the surface. These regions are detected, measured and compared to user specifications. The pocket inspection function detects the presence of tumbler stones in which the contrast of these stones is enhanced with circularly polarized lighting and imaging. The graphical user interface of the CIS provides easy setup and debugging of the image processing algorithms. A database module stores and archives part inspection statistics for later evaluation. The inspection rate is sixty parts per minute.

The major aim of this chapter is to detail the technology associated with a novel industrial inspection system that is able to robustly identify the visual defects present on the surface of painted slates. The development of a real-time automated slate inspection system proved to be a challenging task since the surface of the slate is painted with glossy dark colours, the slate is characterised by depth profile non-uniformities and it is transported at the inspection line via high-speed conveyors. In order to implement an industrial compliant system, in our design we had to devise a large number of novel solutions including the development of a fully customised illumination set-up and the development of flexible image-processing procedures that can accommodate the large spectrum of visual defects that can be present on the slate surface and the vibrations generated by the slate transport system. The developed machine vision system has been subjected to a thorough robustness evaluation and the reported experimental results indicate that the proposed solution can be used to replace the manual procedure that is currently employed to grade the painted slates in manufacturing environments.

A system for the inspection of stemware for geometrical defects and glass imperfections is described in this paper in which cameras at three stations are used for enhanced defect detection. Internal glass quality, contaminants and optical nonuniformity are evaluated at the first station. The second station inspects the glass rim and stemware shape and detects internal defects such as cracks. The third station inspects the stemware base for geometrical and internal defects. Glass irregularities are optically enhanced using a striped illumination pattern and detected using morphological processing. Geometric inspection is enhanced through the use of converging illumination at the second station, while the third station utilizes telecentric optics. Progressive scan cameras and frame grabbers capable of simultaneous image capture are used at each station. The system software comprises six modules consisting of the system manager, I/O manager, inspection module for each station, and stemware sorting and logging module. Each module runs as a separate process. Applications communicate with each other through TCP/IP sockets, and can be run in one or more computers. Two personal computers are used for the system described here.

Analysis of shattered automotive glass for determining particle size, shape and count is legally required to demonstrate that safety criteria have been met. Manual methods are labour intensive, subjective and prone to errors. This paper presents an approach for automating this analysis. Problems associated with missing boundary segments can cause serious errors unless corrective measures are applied. Techniques for efficient extraction of particle boundaries and restoration of missing segments are presented.

The stability of glass substrates is an important concern for the flat panel display industry. High-resolution displays have very tight geometrical requirements and alignment of the various display components is critical if good performance is to be obtained. Prior to development of manufacturing processes for these displays, it was necessary to determine how glass substrates change during the various processing steps. This chapter describes a system to measure electrode patterns before and after critical processing steps for color plasma panels. The electrode patterns, which are made of thin-film gold, form a series of parallel electrodes. In order to measure electrode locations, a vision system consisting of an X-Y stage, a video camera, a frame grabber, and personal computer was used. Images captured with this setup were processed to minimize the effects of noise and improve accuracy. A gray-scale interpolation technique in which the centroids of the electrodes are calculated was used to enhance measurement resolution.

This paper discusses simple but effective one-dimensional morphological techniques for the identification of primary and secondary peak locations associated with light patterns reflected from glass surfaces. A common optical technique for measuring glass thickness and related properties is to observe light reflected from the front and rear surfaces of the glass. Two reflections can be observed when an appropriate structured light source is used to illuminate the glass: a bright primary reflection from the front surface will be observed, along with a much fainter secondary reflection from the back. The secondary reflection is difficult to detect reliably, due to the large difference in intensity compared to the first reflection, the presence of noise, and possibly overlap between them. The methods described in the paper have been implemented successfully for two separate applications, using standard matrix and linear cameras. In both cases, the signal is pre-processed using one-dimensional morphological and linear methods, to normalize the background and remove noise. Further morphological operations are performed, to identify the peaks associated with the primary and secondary reflections.

An important aspect of machine vision is the inspection of products during manufacture, in order to achieve control of quality. This is as necessary for food products as it is for metal or plastic parts, though there is a difference in that food products are highly variable: e.g., even two fairly ideal apples will have marked shape, colour and other differences, and this will have to be taken into account when devising inspection algorithms. Another important aspect of food inspection is the speed with which product location has to be carried out, as the process of location involves unconstrained search: contrariwise, product scrutiny often involves examining a relatively small area of an image, and can often be carried out much more quickly. To understand the process of algorithm design, this chapter takes a number of case studies relating to the location of objects in food and cereal images. These range from finding Jaffacakes, cream biscuits and cereal grains to identifying insects, rodent droppings, moulds and foreign seeds amongst the cereal grains. In the first three cases the objects had to be scrutinised for defects, whereas in the other cases the cereal grains had to be scrutinised for contaminants.

The development of a machine vision system for automated high-speed produce grading is described. Image processing techniques were used to obtain an estimate of the volume of each item, which was then related to the weight through a closed-loop calibration. Accurate weight estimation led to more accurate and better control over the spread of package weights. This reduced the average package weight by approximately 20%, with a standard deviation of 2.5 g for a nominal 100 g package. Improved processing efficiencies doubled the throughput and significantly increased the profitability of the packinghouse.

A large clothing retail store wishes to quantify the creasing properties of the fabrics used in shirts, blouses, dresses, etc. Hitherto, creasing has been judged subjectively and inaccurately, by comparing a sample with carefully designed models, photographs, etc. There is a need for a more objective measurement than this provides and a laboratory-based vision system to automate this task has been proposed. A sample of plain unprinted fabric is first creased in a standard way and then stretched gently as it is placed onto a table. Lighting at a low angle is then applied from one side. The image generated in this way has a low contrast but it is sufficient to allow the crease furrows to be seen. Results are presented for several non-linear filters that convert the grey-scale texture image to binary form. Measurements describing the resulting binary texture are defined. A multi-element vector created by combining several such measurements is then presented to a Pattern Recognition system. This is then trained to identify the creasing coefficient defined by the existing human inspector.

This chapter provides qualitative descriptions of a variety of lighting and viewing methods (LVMs). These are presented in alphabetical order, since there is no natural structure for such a list (Table40.1). There are usually five elements within an image acquisition sub-system: light source, illumination optics, image-enhancement optics, main lens and sensor. The positions of these relative to each other and the object being observed must be carefully selected, to obtain a digital image that is easy to process electronically. The physical layout of the optical sub-system and the choice of appropriate components are of equal importance. Chapter 7 describes the light sources available, while optical components are discussed in Chaps. 5 and 6. Scanners and cameras and are described in Chaps. 9 and 10 respectively. Chapter 8 explains some of the concepts necessary to appreciate the methods described here. Of particular note is that of an illumination diagram. This explains, in graphical form, the shape and position and of the light source in relation to the widget and camera. Notice that the diagrams included in this chapter are not drawn to scale; they merely suggest a design for the lighting-viewing sub-system in broad conceptual terms. A multitude of combinations of the methods described below is possible.TitleMethod number3D shape analysis by projecting Moiré patterns1All round view of a cylindrical object2Anamorphic optics3Arbitrary lighting pattern4Automatic gain control and auto-iris lens5Back illumination of shiny objects (array camera)6Back lighting of matt opaque objects (array camera)7Back lighting of matt opaque objects (line-scan camera)8Calculate the field of view – I9Calculate the field of view – II10Calibration: calculating spatial resolution11Circle of light projecting radially outwards12Circular features13Circularly polarising filter suppresses glinting14Coaxial diffuse illumination15Coherent flexible fibre optic bundle16Collapse image into line17Collecting natural light/sun light18Colour filtering19Combined back and front illumination20Combining several views21Conical mirrors view an annulus22Continuously moving web23Crack detection (ferrous and non-ferrous)24Crossed linear polarisers reduce glinting25Dark field illumination (array camera)26Dark field illumination (line-scan Camera)27Detect pimples, pits and changes of refractive index28Detecting particles in a swirling liquid29Diffuse coaxial illumination30Diffuse front illumination31Dual orthogonal-beam X-ray imaging32Dual-wavelength height measurement33Endoscope for inspecting holes34Estimating range during image capture35Examine the inside of a pipe36Fast electronic shutter37Fibre optic image conduit/converter38Fish–eye lens views inside hole39Flexible inspection cell/robot vision cell40Fluorescence and phosphorescence41Focussed annulus of light42Forced thermal emission43Generating light with arbitrary spectrum44Granular material (pellets) in free flight45Grazing illumination46Hand-held sensor for fixed-range close-up view47Hardness testing48Height analysis by projecting patterns49High resolution view of web50Highly variable ambient lighting51Homogenising illumination52Hypercentric (pericentric) lens53Illuminate top surface only54Illuminating a wide moving web55Illumination within a hollow cylinder56Injecting light into glassware57Inspecting a flat moving web using a laser scanner58Inspecting a rotating object59Internal analysis of an aerosol spray or dust cloud60Internal features on transparent objects61Laser bore scanner62Laser pattern projector63Locate object with zero contrast64Long focus optical system using mirrors65Low-definition range information66Magnify one image axis67Mapping by projecting multiple stripes68Mapping range with polychromatic light69Measuring range by projecting patterns70Measuring thickness of a thin film71Measuring thickness of a thin film72Micro-louvres suppress ambient light73Microscopic fourier analysis of transparent film74Motorised filter wheel75Multi-arm image conduit76Multi-camera specular and near specular illumination77Multi-camera structured lighting78Multiple views of a widget79Multi-spectral image sensing80Multi-view stereo imaging81Normal viewing of spherical surface82Objects on a conveyor83Observing bubbles in clear liquid84Omni-directional illumination85Optically tooled bowl feeder86Passive de-speckling of laser87Pipe or hollow cone of light88Polar-coordinate scan of annulus89Projected array of spots90Protecting the camera using a periscope91Protecting the camera with fibre optics92Protecting the camera93Range measurement using talbot fringes94Reading printing in photo-chromic ink95Real-time optical filtering (high-pass)96Ring of light projecting inwards97Rotate image using a dove prism98Silhouette from multiple shadows99Silhouette in restricted space100Size measurement in free fall101Split field viewing102Stabilised lighting103Steerable image sensor104Steering a laser beam in two directions105Stroboscopic illumination106Structured lighting107Switched-colour light source108Telecentric lens109Using mirrors to view front and back110Very large membrane mirror111Vibration analysis: speckle pattern interferometry112View annulus, ignoring central disc113View flat surface without glinting114View orthogonally polarised images115Viewing a bore116Viewing aerosol spray117Viewing inside an irregular transparent object118Viewing small objects (short focus lens attachment)119Viewing small objects/features120Viewing small objects121Viewing stress in a transparent sheet122Viewing through a small aperture123Visual sensor mounted on robot124Visualise changes in refractive index125Visualising heat distribution126Wireless image acquisition127X-ray imaging128Zooming by movement129Table 40.1Alphabetical list of lighting-viewing methodsThroughout this chapter, the word “widget” is used to denote the object that is to be examined.When designing the image acquisition sub-system, a vision engineer should normally follows the following steps:(i)Look at the widget(a)In a darkened room, move a “point” light source round the widget. Rotate the widget as well. Watch the way that its appearance changes(b)Repeat (a) using a broad light source (e.g., stand in front of a window). Ensure that no light falls on the widget from any other direction(ii)Bearing in mind the observations in step (1), consult the LVM Catalogue. Choose one or a few possible lighting-viewing methods(iii)Build a prototype lighting-viewing sub-system based on the idea(s) suggested by the catalogue(iv)Acquire some sample images(v)Process these images using an interactive image processing system, such as QT (Chaps. 21 and 41) or Neat Vision (Chap. 22)(vi)Adjust the lighting-viewing sub-system and repeat steps (4) to (5), until satisfactory results are obtained(vii)Design and build the target system based on the prototypeStep (1) is very simple, yet it is always in danger of being omitted. The phrase that summarises this design process is “Experiment, look and observe”. Over a period of many years, involving numerous applications, the author has found this procedure to be very effective indeed. Never attempt to design an image acquisition system without moving from the desk!.