Evaluation of visual image for remotely controlled ship

The outline of the battery-powered boat “RAICHO-I” which equips a remote-control system is shown in Fig. 3 and Table 1. A propeller of RAICHO-I is rotated by an electric motor with the lithium-ion battery. Less noise and good response compared with an internal combustion engine have been realized.

The outline of the remote-control system of RAICHO-I is shown in Fig. 4. In general, a land operator and a boat are connected by the wireless communication. The boat transmits sensor information such as a camera image and position data. The land operator returns a boat maneuvering command to the boat based on the obtained information. The remote-control system of RAICHO-I forms a Local Area Network (LAN) using long-distance Wi-Fi, connects boat side PC and the land station PC and exchanges data and information with each other. RAICHO-I has two network cameras in the cockpit to reproduce the view of the merchant marines and broadcast live on the land station and satellite compasses for obtaining the boat position information, the heading, and the boat speed.

In this research, the long-distance Wi-Fi is used as the communication method. Wi-Fi is an unlicensed band, can be constructed by anyone and is inexpensive. The system shown in Fig. 4 is used for experiments in Sect. 4 and later. Cameras 1–3 shown here are network cameras, and cameras used for experiments in Sect. 4 and subsequent sections are network cameras. They are used to monitor video data at other locations in real time.

Conditions for image capturing are examined.To examine how much image quality should be obtained, it is necessary to consider the angle of the view of the camera and the resolution of the image. According to the current regulation of Japan, merchant marines are required to have visual acuity in both eyes 6/12 or more vision [6]. Therefore, the first target of obstacle recognition level using a camera is that whether the Landolt C of 6/12 vision can be recognized or not. According to the international standard, the gap length of the Landolt C of 6/6 vision is 1.454 mm [10]. Since the size of Landolt C in other visual acuities are converted by multiplying the coefficient, 6/12 vision is twice as 6/6 vision. This means that the gap length for 6/12 vision is 3.0 mm. It is necessary to have 2 pixels or more in the 3 mm gap to identify 6/12 vision Landolt C. In other words, it is assumed that the width of 1 pixel is 1.5 mm or less.

In this experiment, the Landolt C simple visual acuity test paper (Fig. 5a) [11] is used. Image recordings are made at a distance of 5 m using each five camera types shown in Table 2. We observe focal length, focus type, focal number, resolution and calculated pixel width of them. The angle of view can be derived from the focal length. Calculated pixel width (mm/pixel) is determined from the angle of view and the resolution, assuming that the size of the captured image is the same. After the image capture, we evaluate the captured images to determine if we can recognize 6/12 vision Landolt C of each image taken with each of the five cameras or not.

Only images taken with a 4K video camera could recognize the Landolt C at the required 6/12 vision. Camera images other than 4K video cameras could recognize Landolt C with 6/60, but not 6/12 vision. The images taken with a network and HD video cameras are shown in Fig. 5b, c. The result with the 360° camera is equivalent to the network camera and the DSLR equivalent to the HD video camera, because the width of each single pixel is almost equal.

It is necessary to use a camera with a specification of 0.56 mm/pixel at the maximum if the camera also has the same inspection standard as merchant marines. For the look-out in all directions, it is necessary to use at least six 4K video cameras. In remote-controlled ships, it is necessary to transmit camera images from the ship to the land station where the ship operator is located. At this time, we think that it is difficult to maintain a wireless communication that has a bandwidth that can transmit 4K video. In addition, we need to consider the importance of the real-time performance of the video for look-out for preventing collisions. On the other hand, when using only one camera, it is necessary to prepare a camera that can capture 360° and has a resolution much higher than 4K. It does seem to be such a camera is not realistic. We think that there is a limit to the wide-angle camera that can be used even if it does not necessarily require 0.56 mm/pixel. The images in Fig. 6 were captured by cameras with an angle of view of (a) about 230° and (b) about 80°. Since these images were captured at the same position, the distance to the building in the white frame in Fig. 6 is equal. The images with wide-angle lens are felt farther the distance to the structures compared to narrow-angle lens due to lens curvature. At present, web supposes that the person who remotely controls the ship by watching camera video is a license holder who has practical experience on board. It is considered that the difference in distance sense between the visual observation and the distortion of the videos has an effect on the operator who has been watching visually on board. Although it is possible to correct image distortion due to the lens, we have not considered the correction requirements, because the expected impact to network transmission latency may impair the real-time performance of the system.

To transmit images by the wireless communication, it is necessary to encode and decode images captured by the camera. Encoding includes the space compression that compresses frames one by one, and the time compression that compresses the amount of data by removing only the difference of images between the previous frame and the rest of the frame. The compression method is different depending on the type of codec. Typical codecs include MPEG-2 adopted in Blu-ray, H.264 recommended for YouTube, H.265 with a higher compression rate than the H.264 successor standard and video with lossless compression. In all the experiments in Sect. 4.1 and subsequently, the H.264 is used according to the network camera specification. When the video is decoded, the playback quality may decrease in the case of low bit rate. To investigate the minimum bit rate that is not affected the obstacle recognition, following experiments are carried out. The resolution is set to 1280 × 720. The obstacle recognition is not affected by the boat speed, even if the set value of the frame rate is 10 fps. In order not to increase the necessary communication capacity depending on the size of the frame rate, the frame rate is set to 10 fps. In Sect. 4.2, we describe the method and results when investigating the minimum bit rate value required to recognize a moving object with a fixed camera as in Experiment (1). In Sect. 4.3, we explain the method and results when we actually use remote maneuvering and observe the minimum bit rate value that appears to be satisfactory and sufficient for maneuvering in Experiment (2). In Sect. 4.4, we describe the method and result of measuring the amount of data required to transmit the captured video via wireless communication channel in Experiment (3). Finally, in Sect. 4.5, we summarize our observations results of experiments (1)–(3).