Blockchain-based green big data visualization: BGbV

With the growth of technology, IoT generated Big data and its associated analytic have become common in the computing industry. The cloud-based Big data solutions have resulted in a significant increase in energy consumption at host data centers. As most of the time, organization’s small yet distributed storage resources remain underutilized. A considerable amount of energy can be conserved by optimizing the utilization of local storage efficiently.

Nowadays, everything is digitally recorded; IoT systems generate and process massive amounts of data every day. However, to process and analyze these data are not enough. Our brain tends to find patterns more efficiently when the data are visually represented. Data visualization and analytics have an essential role in decision-making for Big data [3]. Data visualization leads to new opportunities by representing the data in innovative and visual means [26]. However, with the huge volume of data generated, interactive, scalable visualization and data processing are considered a constant challenge. The size of this information, while visualization is so excessive that an ordinary processing technique is insufficient. Construction of each perception may requires in one hand traversing the whole data or on other reviewing a small subset of the entire data [11]. For this purpose, expensive servers are used. Therefore resources and their efficient utilization is a major concern.

Modern IoT systems involve complex interaction among different entities, demanding a secure, efficient, and distributed infrastructure. Therefore, distributed data visualization is considered an excellent solution to achieve perception and overview of this multi-dimensional massive data [19]. Big data visualization is different from the conventional visualization of data. The visualization of Big data involves a large variety of data. Consequently, high-level data processing techniques and speed are needed to visualize this vast volume of data [22]. There exist multiple web and user-based techniques for the visualization of Big data. These techniques are useful and provide decentralization but with an increase in cost for buying servers and equipment, which is a significant concern.

Blockchain techniques are attractive and have gained researchers’ interest due to their decentralized architecture, immutability, and anonymous record-keeping without centralized authority [6]. Blockchains initially gathered interest to support cryptocurrency. However, due to its decentralized nature, blockchain has emerged as a distributive computing technology that provides a secure environment to support various interactions [21].

Blockchain is considered a highly immutable, decentralized, and modular platform to transfer assets during the last few years. It is considered a generalized approach for implementing decentralized computing applications. Blockchain technology is a decentralized and unchangeable database that is highly transparent and secure. It uses a peer-to-peer network with no central authority. It contains transaction records, and these records are shared among all participating nodes. These properties of blockchain make it a suitable candidate for decentralized Big data visualization with advantage of achieve cost-effectiveness, high security, and readily available results.

We have proposed a green solution BGbV as a support to existing distributed visualization systems. The BGbV framework is based on the blockchain platform, i.e., “Hyperledger Sawtooth” and is targeting to achieve the following features:For our research, a big temporal data set for the crime rate has been taken as a case study to perform the proposed research. Data are gathered from decentralized nodes and visualization is created according to user’s requirements. We have proposed a solution BGbV that will benefit in many ways, including less cost of buying resources for decentralized computing and lower energy consumption. It will allow users/people and organizations to gain secure and readily available data visualization that is decentralized and provides productive outputs for analysis.

In the rest of the paper, “Literature survey” presents the literature review. “BGbV: blockchain-based green big data visualization” section presents the BGbV paradigm. “Analysis of proposed system” discusses features and analysis of the BGbV framework. “Implementation and testing results” presents the implementation and testing details of our solution. We conclude this paper in “Conclusions and future work” and suggest future work.

Many researchers have addressed Big data and its issues [24, 31]. Exploring Big data offers numerous appealing features, but specialists and experts are also confronted with many difficulties while investigating such mines of data [23]. Many data sets are too large and complicated to manage on available memory units and are distributed across the clusters of computers [17]. Due to the growing nature of Big data, it is hard to avoid its challenges. The most immediate issues that need to be addressed are storage issues, management issues, and processing issues [16]. It is challenging to handle data connectivity issues, storage limitations, and data processing capabilities in real-time Big data [23]. The exponential growth in structured and unstructured data has compelled the need for an efficient and reliable storage approach. Therefore, the reliability of devices matters a lot concerning storage approaches chosen for handling Big data. While processing, backup, and archiving Big data, there could be many challenges like storage medium, data replication, and duplication [1]. Big data applications use high-speed servers and equipment, which results in increased cost.

A distributed paradigm is considered a suitable replacement for costly supercomputers. Distributed approaches are decentralized and aim to disrupt the existing central and conventional ways to deal with huge volumes of data. It also ensures to handle and deal with new expanding client’s needs and application demands. Data storage, data access, data transfer, and data visualization activities use distributed computing with low-cost machines to make Big data analysis and processing possible within a reasonable cost and time [20]. In distributed computing systems, the focus is on data representation. It attempts to addresses challenges of processing, interaction, and representation of data in proficient manners.

Instead of textual/numerical analysis data, visualizations are producing better perceptions and understanding of data. However, power and speed limitations are there when visualizing Big data. This leads to the scalability issue [9]. Current processing technologies and systems cannot satisfy the needs of big growing data, processing of data, and visualization. The increase in speed and storage capacity is much lesser than the amount of Big data. Speed and complex processors are needed [24]. Data visualization is a significant way to deal with Big data. It helps to get an overall perspective of Big data and find information esteems. It also helps to find the hidden patterns in data. The challenge is to manage the parallelism that blends between distributed and shared techniques with such massive data. For the issue mentioned above, the requirements of machines are a fundamental concern. Many researchers have shown that large-scale distributed data visualization is an input\(\setminus \)output-bound problem. When interactivity is required, security and data access become significant problems, mainly when the data are distributed over wide-area networks. Many user interfaces and web-based data visualization techniques are available, which provide efficient decentralization with an additional cost for buying servers and equipment. This is the challenge for these techniques. Tables 1 and 2 provide Big data visualization techniques. Table 2 also provides websites to get these tools for the visualization of data. These tools offer visualizations of data based on its features. One of the key challenges in modern distributed data visualization today is how to provide users with an interactive experience. Techniques presented in Tables 1 and 2 are being used in many distributed applications to create and achieve the intended visualization of data [25]. These distributed data visualization frameworks are presented to attain interactive results and to address scalability. Distribution in these techniques is achieved with the help of substantial remote servers. Similarly, high-performance computing capability is needed to get interactive results [14].

Blockchain is emerging as new distributed technology. The blockchain concept is based on distributed ledger maintained by multiple parties [6]. Using blockchain, we can build decentralized systems [8], which can effectively mitigate problems associated with high access and communication costs. Also, a decentralized blockchain architecture is more resistant to a single point of failure [6]. Experiments show that the scalability issue can be effectively handled in blockchain systems.

Decentralized computing frameworks are aimed in several ways at disrupting the current cloud environment and scalability problem.

For achieving interactive outputs from huge streams and sets of big data, blockchain-based decentralized approaches have received great attention. Traditional distributed peer-to-peer (p2p) networks have inevitable disadvantages, including insecurity and lack of auditing and incentives [12]. Filecoin [7] and BigchainDB [4] are scalable blockchain databases that combine the characteristics of both blockchain and existing distributed databases. In all respects, blockchain is opening doors for solving numerous problems for many applications in a distributed manner [2]. Blockchain-based frameworks offer many benefits for distributed storage with features like availability, no single point of failure, confidentiality, privacy, and integrity [5]. Currently, Filecoin [7], Sia, Swarm, and Storj, etc. are mainstream distributed storage solutions built on blockchain [33].

Sia uses blockchain technology to provide an open market to buy and reserve unused computing space for users. Conditions of storage, such as availability and active duration that are agreed by the participants under file contracts, are encrypted service-level arrangements. These are immediately stored on the Sia blockchain and done [29].

Filecoin is another decentralized blockchain and native cloud service. It produces an incentive-based scheme that provides additional storage. Filecoin uses IPFS, a P2P distributed protocol, where each file is encrypted with a hash key and containing indexing details. It enables large amounts of data to be searched and saved and distributed with high productivity [18].

Another protocol for decentralized storage network is SWARM. It is permission less and communication infrastructure [28]. The principal goal of swarm is to provide the decentralized app’s creators with infrastructure resources. To implement the utilities, SWARM uses smart contract platforms (Ethereum). The strategy for decentralized storage is a peer-to-peer approach.

It is not deniable that blockchain has opened doors for solving many problems for many applications in a distributed manner with provision of various kinds of distributed storage approaches mentioned above [2].

Considering these blockchains based on distributed storage such as Filecoin, Sia, Swarm, etc, the set of transactions are stored within the of blocks of immutable blockchains and generate a type of decentralized database or storage of structured data. However, due to scalability concerns, the size of blocks cannot expand very large, and thus, it can be clearly seen that above-mentioned distributed storage blockchains are not meant to store and handle large amounts of data and will take too long to process and resources to accomplish these executions. These problems lead to increase the overall energy of the network. Due to consumed energies in nodes and long communication distances, the network lifespan, and energy consumption and maximization; it has been a critical issue and it increases the overall energy of the network [27]. Hence, these generic schemes are not much energy efficient also do not address optimizing network and storage cost.

All projects are similar in the way as they are trying to provide decentralized storage, but existing solutions are controlled by centralized third party and are mainly commercial applications. The data stored are available to intermediaries who are providing services. The involvement of intermediaries and third parties leads to an increase in operational and transnational costs and lack of security which makes the system vulnerable.

Hyperledger Sawtooth is also a blockchain framework used in developing and running applications in a distributed manner smoothly [13]. Sawtooth also addresses the challenges of permissioned (private) networks. Sawtooth clusters with different approval can be implemented quickly. No central service may leak transaction trends or other classified details. Sawtooth Blockchain can help to provide efficient data storage and access framework for many private and small organizations and applications [2].

Despite, all good blockchain systems are facing scalability issues. Due to a number of scalability issues and limited on-chain storage capacity, the feasibility of such a blockchain-based data network is restricted [27]. A variety of methods have been suggested to increase scalability while maintaining decentralization and security in this regard. Sharding technique is one of it.

Sharding is one of the most practical approaches for achieving scalability by partitioning network into several shards, to reduce the overhead of scalability [32]. However, the issue is to maintain atomicity during cross-shard operations. Each shard is stored in a separate server instance [30].

Although this spreads the load, but increases the overall cost and increases the energy consumption. Sharding can be a good choice when it comes to public and permission-less blockchains, but for private small-level projects, sharding technique may be costly, complex, and consume more energy. Table 3 presents a comparison between some blockchain-based storage services. Provided in Table 3 is relative estimation four parameters, i.e., closed-source decentralization, exposure of data to third party, network cost and energy consumed, scalability, energy-efficient objectives, and free of cost solution.

Many blockchain-based frameworks offer benefits like availability, no single point of failure, confidentiality, privacy, and integrity [5]. Hyperledger Sawtooth is a blockchain framework used in developing and running applications in a distributed manner smoothly [13]. Blockchain can provide efficient data storage and access framework for many organizations and applications [2]. As the visual representation of Big data communicates ideas more clearly, the need for data visualization is increasing in different domains. However, it is a complex and demanding task to visualize a massive volume of data. Therefore, a distributed and green solution is needed to achieve interactive data visualization with less cost and energy consumption. Using blockchain-based mechanism for distribution and decentralization of big data and then visualize could be a good solution. However, aforementioned blockchain-based techniques also increase the chances of escalate overall network cost with high energy and resources consumption. Also users may somehow be required to be controlled by some third-party service making it centralized somehow.

Therefore, there is need of green solution with minimum energy consumption having low network cost providing optimum storage solution, for small and big levels of projects ensuring security, available and reliable data, and its visualization at same time.

BGbV aims to provide support to distributed data visualization. It ensures security and availability, and, most importantly, provides a framework for better utilization of existing distributed resources. By utilizing small units of storage effectively, it facilitates scalability challenges. We present a green solution that will reduce cost by utilizing small resources that are already available and lessen energy consumption and make it an environmentally friendly solution.

BGbV proposes a distributed architecture of blockchain-based green solution for Big data visualizations. Many organizations have loads of distributed storage resources available in unused disk space of workstations and terminals. Instead of getting new equipment that requires more hardware and power consumption, this solution provides optimal utilization of the existing smaller distributed storage units with the least energy consumption.

In BGbV, we have incorporated the hyperledger sawtooth framework with our front-end application. Hyperledger sawtooth is highly modular framework and isolates the client part of the core system’s application. In our prototype, we have considered small-to-medium-level organization with mid-range resources to store the temporal big data set. According to our scope, the permissioned (private) hyperledger sawtooth blockchain is beneficial as it maintain the privacy and security. We have developed the client part of our application and core system in javascript. Our proposed system presents the design of a blockchain-based data storage and access framework for interactive visualization of big temporal data.

Proposed BGbV is different from existing techniques mentioned earlier, i.e., due to its integration with blockchain technology with optimum resource utilization and least energy consumption making it more green solution. It ensures many significant benefits like effective and optimal usage of distributed storage, temper proof record-keeping mechanism, no single point of failure, and availability.

In BGbV, metadata are stored on the blockchain, whereas the actual files are stored off-chain on multiple locations after distribution. This distribution is according to a pre-defined agreement using a peer-to-peer network. This scheme will provide decentralized storage on an economical and secure basis.

In this section, we present the analysis of the energy and cost saved by BGbV and its adaptation. It is a fact that IT storage infrastructure is under utilized by many organizations. As per a survey conducted by CISCO [10], the utilization of direct access storage on average is 40% and is 60% for the storage access network. Organizations have underutilized distributed storage available in free disk space on several desktops and laptops. Such organizations can improve the utilization of already available resources by employing BGbV for their big data storage solutions. On average, the cloud-based solution for Big data visualization costs more than 2–4 USD per GB per month. Similarly, as Table 3 shows, all decentralized solutions are also commercial solution with associated costs. For most organizations, this cost is besides the cost of maintenance on site underused storage equipment.

In the following analysis, we present the details of energy consumed in storing data using BGbV step as compared to cloud-based storage. it can be seen that, in terms of energy consumption, the utilization of existing direct access storage solutions consumes less energy than cloud-based storage. In BGbV, we propose two blockchains \(\mathbf {BGbV \alpha }\) and \(\mathbf {BGbV \beta }\). As mentioned earlier, blockchain \(\mathbf {BGbV \alpha }\) stores the rating mechanism, and blockchain \(\mathbf {BGbV \beta }\) saves metadata about data fragments of big data stored at multiple nodes. Each block of \(\mathbf {BGbV \alpha }\) has two parts block header and body. Each block’s body contains transaction data comprising approx. 264 bytes. The block header’s information contains block number, version, previous BlockHash, and the time-stamp at which this block is committed. The block’s header is 46 bytes. Thus, the total size of each block is 310 bytes, which approximately equals 0.3 KB. In \(\mathbf {BGbV \beta }\), the body of the block is 134 bytes. By adding the standard header size of 46 bytes, the net transaction size is approximately equal to 0.1 KB. By combining both blockchains of size 0.4 KB to 0.5 KB, the BGbV takes less than 1 KB overhead for storing each fragment.

Figure 4 shows the amount of energy saved using direct access storage devices compared to the cloud-based solution. The energy is calculated byHere, \(E_S\) is net energy saved, \(E_{cl}\) is net energy consumed by cloud, \(E_{BGbV}\) is net energy consumed by BGbV, F sz is the total number of bits in the file, \(\lambda _{cl}\) is the energy consumed to write 1 bit on the cloud, \(\lambda _{HDD}\) is the energy consumed to write one-bit disk, F i are the number of bits in each segment stored, \(N_{nodes}\) is the number of nodes maintaining BGbV blockchain, K is number to fragments, and \(Tr_{BGbV}\) is the number of bits in the transaction. Energy rate \(\lambda _{HDD}\) is kept as 100FJ and \(\lambda _{cl}\) is kept as 1nJ [15].

Figure 4 shows the energy saved against file size ranging from 1 to 60 TB is shown. For BGbV, we have sliced each file into fragments of size 1 GB. Also, we have kept the number of nodes that keep the copy of the blockchain to 100. The red line shows the upward trend in energy saved with respect to file size. Even though the number of fragments and transaction overload is also increased (shown as bars), but net energy saved in mJ improves with larger files. The above calculation is performed to establish the bottom line of the energy saved by considering just storage. Here, any additional costs of bit transfer to the cloud are not considered.

We have calculated only the energy spend in saving original files. It can be seen that backup and replication overhead will show the same trends.

Also, for IOT big data, the greatest obstacle for implementing blockchain in IOT is scalability, as the response to requests expands as the number of computing machine increases. This leads to high energy consumption. Our solution will also help with IOT big data. As mention the proposed system is decentralized and also use free local underutilized storage will be used to save data. The continuously growing IOT data can be efficiently stored and accessed with minimum network energy. BGbV is flexible in terms of source of data. Whether big data are transferred from the cloud or is generated by IOT, the BGbV framework can easily mange it.

This section discusses the implementation and the testing details. This application is developed in HTML 5.0 and Node JS as front-end languages.

In emerging IoT systems, effective resource utilization, security, availability, and throughput of Big data are of main concern. We have proposed BGbV, a blockchain-based solution for effective storage and retrieval of Big data visualization. The integration of blockchain solutions provides cost-effective, secure, and available visualizations of Big data. BGbV presents a mechanism of the utilization of relatively small distributed resources optimally for storing Big data. As our system provides an alternative to the purchase of additional resources, thus reduces energy, costs, and electronic waste. Additional benefits of our scheme are the high availability and better extensibility.

In future work, the crowd-sourcing of storage nodes will be explored by adding an effective incentive mechanism. Nodes will be awarded incentives depending upon the rating and reviews they get from the users. Developing such an architecture that allows large data visualization with incorporating benefits like high availability, no single point of failure, quality, and interactive outputs will prove beneficial and will noticeably improve the business sector’s performance.