Introduction
Related works
Theoretical basis
Big data definition
Big data management
Big data analysis framework
Framework | Abstraction | Supported languages | Underlying engine | Availability |
---|---|---|---|---|
Hadoop | MapReduce | Java, Python | Hadoop | Open source |
Spark | RDD | Scala, Python, Java, R | Spark, Yarn | Open source |
H2O | Procedural + Library | REST, R, Python | H2O | Open source |
SciDB | Declarative | SQL | SciDB | Open source |
AzureML | Visual User Interface | REST | Microsoft Azure | Microsoft |
R | Procedural | R | R | Open source |
SparkR | Procedural | R | Spark | Open source |
Mahout | Toolkit | Java, Scala | Hadoop, Spark | Open source |
Spark MLib | Library | Scala, Python | Spark | Open source |
Samsara | Declarative | Java, Scala | Spark, Flink, H2O | Open source |
SystemML | Declarative | R, Python | Spark | Open source |
Google ML | Visual User Interface | Python | Google Cloud Dataflow | Google |
Amazon ML | Visual User Interface | N/A | Apache MXNet, TensorFlow, PyTorch | Amazon |
BigML | Visual User Interface | Python | BigML Predict Server | BigML |
Tensorflow | Visual User Interface | Python, Haskell, Java, Go, Julia, R | CUDA, TPU | Open source |
KeystoneML | Procedural + Library | Scala | Spark | Open source |
Message passing interface (MPI)
Memory mapping
Advance Encryption Standard (AES) algorithm
The proposed approach MPIM
First stage: constructing memory map
Second stage: building message passing interface
Third stage: approach validation
Methods and experience
Mathematical model
Implementation
Data gathering
Data processing
No. of processes | Segment size |
---|---|
1 | 0x80000000 |
2 | 0x40000000 |
3 | 0x2aa99999 |
4 | 0x20000000 |
5 | 0x19999999 |
6 | 0x1554cccc |
7 | 0x12480000 |
8 | 0x10000000 |
9 | 0xe380000 |
10 | 0xccccccc |
11 | 0xba19999 |
12 | 0xaa99999 |