Detection of duplicated data with minimum overhead and secure data transmission for sensor big data

Big data refers to the data sets that are difficult to deal with traditional data processing applications because of its speed, size and variety of data. The big data were generated from activities, sensing devices, mobile devices, Internet, RFID readers etc. One of the key sources of big data is the data from the sensor. The significant amounts of the data from the sensor are either redundant or almost similar. It initiates the requirement of de-duplication of the sensor data. The data from the sensors need to be stored for further process or analysis which requires end-to-end security for the data. A method is proposed in this paper for detecting the similar data with light-weight process using pattern analysis and matching. The distributed encoding process is proposed here for imposing end-to-end security for the generated data with reduced communication overhead. The data received in the processing server are decoded, analyzed and matched with patterns for removing similar and duplicated data. The result shows that the proposed system secures data during transmission with light-weighted processes. The duplicated and similar data are detected efficiently through inline process before the data enter into the storage. Experimental results are given as proof of the above mentioned concept.