1 Introduction

Micro-assembly technology is used widely in micro-electro-mechanical systems (MEMS) and micro-opto-electro-mechanical systems (MOEMS) industry. It refers to the process of handling and assembling micro-sized components into micro-devices. These components may have small size of several μm to several mm. To fabricate them, micro-assembly techniques, such as micro-gripper design, control system design, and modeling of adhesive force, are all face challenges[1-5].

Heterogeneous micro-parts produced from various fabrication processes are used for producing complex 3D microstructures in some micro-assembly tasks[6-9]. The micro-assembly workstation which comprises a micromanipulator, high precision positioning stages, a set of visual systems and micro-force sensors is commonly used[10-13]. Micro-assembly has high requirements in accuracy and flexibility.

As the microscope vision has the feature of small view field, it is not easy to obtain micro-parts precise position. Furthermore, visual instability exists in the micro-assembly process, and the micro-positioning accuracy is hampered by the volume of sensors[14]. Traditional movement control strategy based on position is not sufficient for the micro-assembly tasks. In addition, as the frangibility of the microparts, it is easy to be damaged in the assembly process. Position-based method does not take interaction force into account which has a great influence on the measurement at the micro-scale. Proper force control can decrease the high requirement of the position detection accuracy and increase the security in the micro-assembly process.

In the literatures, many studies have been done on using force control for improving the micro-assembly results. Despite the diversity of the approaches, force control is mainly used in two ways. First, the force is introduced to guide the assembly task. Rabenorosoa et al.[15,16] introduced force to guide the micro-part to insert into the rail. The micro-part is gripped by two sensorized fingers. The two sensorized fingers can detect the contact side of the micro-gripper and the rail, and can estimate the contact force between them. The contact force control can correct the trajectory of the micropart when the contact appears. However, the manipulation efficiency is always low. And the complex micro-gripper with force feedback is designed to ensure the security of the micro-parts[17-19]. The manipulation force is controlled in real-time for gripping the micro-parts. Choi et al.[20] designed a variable stiffness joint (VSJ) for a robot manipulator, as well as a control scheme to control the stiffness and the position of the VSJ. However, the design of the gripper is always complex.

To align a small gripper and a special ring object conveniently and safely, without complex gripper design, an effective control strategy based on microscope vision and contact force is proposed in the paper. The microscope vision system is weakly calibrated. The distance of the gripper and ring object is calculated in real-time based on feature extraction from the image captured by microscope vision system. The end-effector of the small gripper is tracked in real-time to keep the small gripper moving in the small field of view. The force of the ring object is detected in real-time. Once the gripper moves into the near region, the force of the ring object detected by the force sensor is introduced to control the small gripper movement. The force of the ring object is in a safe range in the alignment process.

The rest of this paper is organized as follows. In Section 2, the microscope vision system is introduced, the distance calculation method and the feature extraction algorithm are presented. The force sensor and the movement control strategy are described in Section 3. In Section 4, experiments are provided to verify the effectiveness of the control strategy. The paper is concluded in Section 5.

2 Micro-assembly system and distance computation

The task of the vision measurement system is to extract features of the image of the small gripper and the ring object, and to compute the distance between them.

2.1 System configuration

The visual system is shown in Fig. 1. The System consists of a charge-coupled device (CCD) camera, a microscopic lens and a light source. The image captured by the camera is sent to the host computer via local area network (LAN) and processed. The force sensor is connected to the ring object via a connector, and its signals are sent to the computer in real-time. The small gripper is fixed to 3 degree-of-freedoms (DOFs) motor stage. The motor is controlled by the host computer through serial port.

Fig. 1
figure 1

The sketch of visual system

Full size image

The object is a hollow thin cylinder ring with four circular holes spread evenly. Its shape is given in Fig. 2 (a). It is a hollow cylinder with an outer diameter of 5.36 mm, 0.156 mm thickness and 3 mm height. There are four circular holes with diameter of 2.1 mm arranged on the cylinder surface evenly. The small gripper is shown in Fig. 2 (b). The end-effector is part of a cylinder surface, which is used to absorb the ring object via negative pressure.

Fig. 2
figure 2

The sketch of the ring object and the gripper’ end-effector

Full size image

2.2 Distance computation

The position relationship between the small gripper and the ring object in the image is shown in Fig. 3 (a). The pose of the ring object relative to the microscope vision system is adjusted well in advance[21]. The ring object in the image is a thin circle. The end-effector of the small gripper in the image is part of a circle. The position relationship after alignment is shown in Fig.3(b).

Fig.3
figure 3

The position relationship of the small gripper and the ring object

Full size image

Three selected feature points are denoted as O, A and B. The feature line is denoted as L, as shown in Fig. 3 (a). The point O is the center point of the ellipse C 1. The line L is the center line of the small gripper. The point A is the right intersection of the line L and the ellipse C 2. The point B to the point B and Y -axis is vertical to the line L 0, as shown in Fig.3. The image distance of the gripper and the ring object in the Y -axis direction is calculated as

$${D_{yi}} = \frac{{|k{O_x} + b - {O_y}}}{{\sqrt {1 + {k^2}} }}$$
((1))

where k and b are the parameters of the line L, (O x , O y ) is the image coordinate of the point O, D yi is the image distance of the point O to the line L, i.e., the image distance of the small gripper to the ring object in the Y -axis direction.

The image distance of the small gripper and the ring object in the X-axis direction is calculated as

$${D_{yi}} = \sqrt {{{\left( {{A_x} - {B_x}} \right)}^2} + {{\left( {{A_y} - {B_y}} \right)}^2} - D_{yi}^2} $$
((2))

where (A x , A y ) and (B x , B y ) are the image coordinates of points A and B, D xi to the ring object in the X-axis direction.

The image distances D xi and D yi need to be converted to the Cartesian space distances to control the motor movement. The calculation formula is shown in (3).

$$\left\{ {\begin{array}{*{20}{c}} {{D_x} = \alpha {D_{xi}}} \\ {{D_y} = \alpha {D_{yi}}} \end{array}} \right.$$
((3))

where D x is the Cartesian space distance in the X-axis direction, D y is the Cartesian space distance in the Y-axis direction, α is the Cartesian space length of one image pixel, and α is calibrated off-line[21].

2.3 Feature extraction

The background image with the ring object is recorded before the end-effector moves into the field of view (FOV). Then, the images are captured and processed with the end-effector moving into the FOV. The process of the feature extraction and the distance calculation is divided into four steps, as shown in Fig. 4. Firstly, the image area of interest (ROI) is computed, as shown in Fig 5 (a). Secondly, the image ROI R b is calculated by background difference of the current image and the background image. The small gripper upper edge-line L 1 and lower edge line L 2 are extracted from R b . The lines and ellipses are extracted by improved random sample consensus (RANSAC) algorithm[21]. The central line L is determined using the lines L 1 and L 2, as shown in Fig. 5 (b). Thirdly, L 0 is determined from line L and the point O. The point A is computed from the line L and arc edge C 2, the point B is calculated from line L 0 and arc edge C 1. The distances D x and D y are calculated, as shown in Fig.5 (c). Fourth, the area of the end-effector of the small gripper is computed, which is denoted as a rectangle PQMN. The image ROI R b movement control process.

Fig.4
figure 4

The flow chart of feature extraction algorithm

Full size image
Fig.5
figure 5

The sketch of the feature extraction algorithm

Full size image

3 Impedance control system

As error exists in the posture adjustment of the ring object, the ring object is not parallel with the small gripper in the horizontal direction sometimes, as shown in Fig. 6.

Fig.6
figure 6

The sketch of the position relationship of the small gripper and ring object in the horizontal direction

Full size image

In the situation as shown in Fig. 6 (a), the calculated distance can indicate the actual distance between the gripper and the ring object well. In the situation in Fig. 6 (b), the calculated distance cannot correctly reflect the actual distance between them. The gripper will collide to the ring object in the alignment procedure with the guidance of the microscope vision system. It will damage the gripper and the ring object.

To avoid the damage due to collision, the contact force is introduced into the control system as the feedback to control the motor movement. The vision-based position/force hybrid control system is designed by using the impedance control scheme, as shown in Fig. 7.

Fig.7
figure 7

Block diagram of control system

Full size image

In Fig. 7, E x and E y are the expected positions of the small gripper in the X-axis and Y -axis direction. k f is the conversion coefficient from force to position. F x and F y are the forces acting on the ring object. e x and e y are the calculated distance errors, and they are calculated in (4).

$$\left\{ {\begin{array}{*{20}{c}} {{e_x} = {E_x} - {D_x} - {k_f}{F_x}} \\ {{e_y} = {E_y} - {D_y} - {k_f}{F_y}.} \end{array}} \right.$$
((4))

The choice of the coefficient k f is important in the control system. If k f is set to a small value, the small gripper will have a strong stiffness. If k f is set to a big value, it will have an obvious compliance. The calculation of the coefficient k f is given in (5).

$${k_f} = \left\{ {\begin{array}{*{20}{c}} {0,\quad {\text{if}}\,{\text{|}}{E_x} - {D_x}|\, \geqslant {e_t}} \\ {c,\quad {\text{if}}\,{\text{|}}{E_x} - {D_x}|\, < {e_t}} \end{array}} \right.$$
((5))

where e t is a threshold distance that the contact will probably happen. If the distance is larger than the threshold, the contact is considered not happened. c is a constant value calculated using (6).

$$c = \frac{{{e_t}}}{{{F_0}}}$$
((6))

where F 0is the allowable maximum contact force of the small gripper acting on the ring object in the alignment process.

The controller for the gripper’ movement adopts proportional-integral (PI) controller with variable structure, as given in (7). When the relative distance of the gripper and the ring object is larger, the gripper moves a fixed length in a control circle in order to control the small gripper approach to the ring object quickly. When the relative distance is smaller, the gripper is controlled by the PI algorithm to align with high precision.

$$\begin{gathered} \Delta u\left( k \right) = \hfill \\ \left\{ {\begin{array}{*{20}{l}} {{L_{s,}}\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \,\,\,e\left( k \right) > {e_{T1}}} \\ {{K_P}\left[ {e\left( k \right) - e\left( {k - 1} \right)} \right] + {K_I}e\left( k \right),\,\,\,{e_{T2}} \leqslant |e\left( k \right)| \leqslant {e_{T1}}} \\ \begin{gathered} 0,\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad |e\left( k \right)| < {e_{T2}} \hfill \\ - {L_s},\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad e\left( k \right) < - {e_{T1}} \hfill \\ \end{gathered} \end{array}} \right. \hfill \\ \end{gathered} $$
((7))

where k is the control cycle index, ∆ u(k) is the control value before the k-th step movement, L s is the step length in the far distance range, e T 1 is the distance threshold, e T 2 is the allowable given error, e(k) is the calculated distance error before the k-th control cycle, K P is the proportional coefficient, and K I is the integral coefficient.

The movement control in the X-axis and Y-axis are executed simultaneously. Once e x and e y are both in the given range, i.e., |e x | < e T 2 and |e y | < e T 2, the movement control is stopped.

4 Experiments and results

The experimental platform is shown in Fig. 8. The image resolution of the CCD camera (GC2450) was 2448 x 2050 in pixel, its maximum frame rate was 15 frames per second (FPS). The magnification factor of the microscope (Navi-tar) was from 1.05 to 6.75. The full scale of the force sensor (Nano43) was 18 N. The working range of the 3-DOFs stage was 100 mm in each axis, and the maximum movement speed was 400 mm/s.

Fig.8
figure 8

Experiment platform

Full size image

In the experiments, the PI parameters K Px , K Ix , K Py and K Iy were set to 0.5, 0.15, 0.4 and 0.07 according to Ziegler-Nichols method[22]. The expected values E x and Ey were set to 50 μ m and 0μm, respectively. The allowable error range of e x and e y were set to [-10μm, 10μm]. The threshold e T 1was set to 500 μm, and the step length L s was set to 200μm.

As the statistical analysis of the force is obtained through the force sensor, we have found that the noise exist in the force. As the Kalman filter can work very well to filter the noise, the force was filtered by Kalman filter. The force sensor used in the experiments was Nano43, which is well decoupled in X and Y directions. Therefore, the noises in X and Y directions can be separately dealt with. In experiments, the forces from the force sensor in an interval about 14 s were collected at a stationary state. The noise variance was calculated based on the collected data. The noise variance in the X-axis was 9 × 10−4, and in the Y-axis was 5 × 10−4. As the allowable force of the ring object, the contact force F 0 after filtering was set to 1 N. The threshold e t was set to 150 μm. Therefore, the constant c was 150 μm/N. It is calculated by (6). The sample time of the force was 10 ms. The control time was set to 300 ms to ensure that the stepper motor could finish the given motion and the vision measurement could be surely finished.

A series of experiments of the small gripper aligning to the ring object were conducted well with the guidance of the microscope vision and the force information. The control system used was described in Section 3. The images of the alignment process are shown in Fig. 9. Fig. 9 (a) is the image before alignment, Fig.9 (f) is the image after alignment, and Figs. 9 (b)–9 (e) are the images in the alignment process. It can be seen that the end-effector of the gripper was well tracked, and the end-effector was kept in the field of view. The entire image processing time was about 80 ms.

Fig.9
figure 9

The images in the approaching process

Full size image

The introduction of the force is very helpful to avoid collision. A series of experiments were done to verify the effectiveness of the control strategy. Two sets of experimental results were given. Experiment 1 was conducted in the normal situation when the contact did not happen between the small gripper and the ring object. Experiment 2 was conducted in the situation when the contact happened.

4.1 Noncontact approaching

The force in Experiment 1 is shown in Fig 10. Fig 10 (a) is the force in the X-axis direction, and Fig 10 (b) is the force in the Y-axis direction. It can be seen that the force after Kalman filtering varied in a small range. The contact of the small gripper and the ring object did not happen. The distance error in the process is shown in Fig. 11. The distance error in the X-axis was decreased to −6.3μm at last, and in Y-axis was decreased to −4.7μm. They were in the given range. It can be seen that the error decreased to the given range effectively.

Fig.10
figure 10

Force in the experiment 1

Full size image
Fig.11
figure 11

Distance error in the Experiment 1

Full size image

4.2 Contact approaching

The force in the Experiment 2 is shown in Fig. 12. Fig. 12 (a) is the force in the X-axis direction, Fig. 12 (b) is the force in the Y-axis direction. It can be seen that the force after Kalman filtering had an obvious variation at the time of 5.7s. It is the time the small gripper contacted the ring object. Accordingly, the contact happened at the 19th step movement. The movement was not stopped until e x and e y were in the given range. Fig. 13 is the distance error in the alignment process. The error in the Y-axis was decreased to −39.8μm at last, and in the X-axis was decreased to −0.4μm. The maximal contact force in the X-axis was less than 0.1 N, and in the Y-axis was less than 0.4 N. They were in the safe force range. The security of the small gripper and the ring object in the alignment process was ensured.

Fig.11
figure 12

Force in the Experiment 2

Full size image
Fig.13
figure 13

Distance error in the Experiment 2

Full size image

Ten times of experimental results are given in Table 1. It is in the noncontact situation. The final distance error in the X-axis and Y -axis are listed. Both of them were in the given range. Table 2 lists the results of the contact situation. The maximal contact force and the distance errors at last in the alignment process are listed. It can be seen that the max force in every experiment was less than 1 N, and the last errors were out of the given range because of the force. The security in the alignment process was ensured. As seen above, the control system fulfills the system requirement effectively.

Table 1 Experimental results in noncontact situation
Full size table
Table 2 Experimental results in contact situation
Full size table

The experimental results verified the effectiveness of the proposed control system. In the noncontact situation, the controller controls the small gripper aligning to the ring object effectively. In the contact situation, the contact force of the small gripper and the ring object is controlled in a safe range. The security of the small gripper and the ring object in the alignment process are ensured.

5 Conclusions

Under the guidance of microscope vision and force information, postion/force hybrid control system is proposed to control the small gripper aligning to the ring object. The control system mainly has two contributions. First, the microscope vision is weakly calibrated. Only the scale from the image to Cartesian space is calibrated off-line. The relative position of the two objects is calculated in real-time by several image geometry features. Second, to avoid collision of the ring object and the gripper, the force of the ring object is introduced into the control system. The force is converted into the distance error to control the gripper movement. The force of the ring object is in a given safe range in the alignment process. The control system does not have high requirement about the gripper structure for alignment safely. Furthermore, the control strategy is easy to be implemented, and it can be used in the other micro-assembly processes. The effectiveness of the proposed control system was verified in the experiments. In the contact situation, the absolute value of force is less than 1 N. In the noncontact situation, the absolute value of the final distance error is less than 10μm. The entire image processing time is less than 80ms. The gripper is controlled to align to the ring object fast and safely.

In the future work, we will focus on the assembly of the ring object with other micro-parts of small size and complex structure in the three-dimensional Cartesian space. The goal is to assemble a complex micro-object for special purpose. The research of complex geometry structure micro-parts assembly is valuable for the development of the EMES and micro-optical-electro-mechanical systems and micro-electro-mechanical systems.