Histone-Net: a multi-paradigm computational framework for histone occupancy and modification prediction

Cells are fundamental building blocks of living organisms. Cells constitute tissues, tissues form organs and combination of organs give birth to organ systems [1, 2]. The way different living organisms grow, survive, develop and reproduce is regulated by an instruction manual called Deoxyribonucleic Acid (DNA) or Genetic code [1, 2]. The genetic code is organised into chromatin in a series of nucleosomes, where in each nucleosome, DNA is wrapped around histone octamers which are made up of four pairs of histone proteins (H2A, H2B, H3 and H4). A graphical representation of nucleosome construction with Histone Octamer and DNA binding is illustrated in Supplementary Figure 1. In the process of gene regulation, nucleosomes play an important role as gene transcription is blocked in regions, where DNA is tightly packed by nucleosomes. Nucleosome occupancy affects epigenetic silencing [3], cell replication [4], differentiation [5], and re-programming [5]. Determining whether DNA around histone octamer is tightly wrapped or loosely wrapped, a genetic task known as histone occupancy determination has profound importance in genetic research [6, 7]. Accurate determination of histone occupancy can facilitate deeper understanding of DNA accessibility to proteins [8, 9], chromatin functions [10, 11], and occupancy correlation with promoter strength [12].

Similar to histone occupancy, histone modifications (acetylation, methylation, phosphorylation, sumoylation, and ubiquitylation) are responsible to regulate multifarious biological processes including chromosome wrapping [13, 14] transcriptional activation and de-activation [15‐17], damaging and repairing of DNA [18, 19]. For instance, histone amino (N)-terminal tails modifications influence internucleosomal exchanges and are capable to modify chromatin structures which ultimately affect gene expression [20] and give birth to many complex diseases, such as Cancer [13].

To acquire a deeper comprehension of epigenetic regulation at cellular level and to pave way for the development of drugs specifically targeting cancer treatment, and histone altering enzymes [21], histone modification detection is essentially required [22]. Histone modifications [23] largely affect the availability of DNA to different transcription factors and ribonucleic acid polymerases. Histone octamers repeat themselves across all nucleosomes in histone sequences, hence properties of nucleosomes primarily rely on incorporated area of histone sequences with specific acetylation and methylation sites level [24, 25]. In addition, considering methylation of histone proteins H3 and H4 mainly regulate the core activity of DNA replication [26], and acetylation of different histone proteins impact chromatin structure as well as gene transcription [27, 28]. A thorough analysis of histone acetylation and methylation areas in histone sequences can decipher the association of histone modification with metabolism which mediates diverse epigenetic abnormalities in multifarious pathological conditions [29].

Developing a robust computational approach for accurate histone occupancy and modification prediction has been an active area of research, since the public availability of ten benchmark data sets developed by [30]. From ten benchmark histone marker data sets, two belong to histone occupancy, three are related to histone acetylation, and five are related to histone methylation. Across 10 different benchmark data sets, histone sequences having occupancy, acetylation, or methylation level greater than 1.2 belong to positive class and lower than 0.8 belong to negative class. To perform binary classification across all ten benchmark histone markers data sets, [30] proposed the very first computational approach for histone occupancy and modification prediction for yeast genome. Their proposed approach utilized occurrence of higher order residues to generate statistical representation of histone sequences and Support Vector Machine classifier.

Using 10 different benchmark data sets facilitated by [30] related to histone occupancy, acetylation, and methylation, to date, a number of computational methodologies have been developed [31‐36]. Prime focus of existing computational approaches [30, 34‐36] has been to generate a rich statistical representation of histone sequences. In this regard, few researchers have utilized bag of words-based approaches [30, 34], whereas others have utilized one hot encoding scheme to generate statistical representation of histone sequences [31‐33, 35]. While, bag of words-based statistical representation only manages to capture residue frequency and neglects rich semantic information. One-hot encoding lacks to capture comprehensive contextual information and correlations of residues. Furthermore, bag of words and one-hot encoding schemes face the curse of dimensionality issue with the induction of higher order sequence residues.

Recently, [36] proposed a deep learning approach for histone occupancy and modification prediction. For each sequence, they transformed one-hot encoded vector of higher order residues into image-like tensor through the assignment of each higher order residue to a pixel in an image by making use of Hilbert curves. Image-based representation of histone sequences was passed to a CNN model for the extraction of important residue correlations and dependencies. Although image-based representation manages to find discriminative sequence residues, however, fails to handle transnational invariance of residues mainly due to the supreme attention towards local residue context. Despite the fact that histone sequences are primarily comprised of four basic residues [adenine (A), cytosine (C), guanine (G), and thymine (T)], treating them as a simple string of repetitive letters neglects their biologically relevant and inherent spatial configuration as well as interaction between sequence residues. Complex molecular spatial composition of histone sequences indicates the relevance of a rich statistical representation which can effectively capture long-range dependencies of residues. However, due to the lack of comprehensive understanding of sequence residue patterns, a rich statistical representation scheme for histone sequences related to histone occupancy, methylation, and acetylation does not exist.

Building on these deficiencies, for the establishment of an improved and more robust histone occupancy and modification landscape, the paper in hand develops a lightweight computational multi-paradigm framework, namely, “Histone-Net”. Considering the efficacy of neural language modelling in diverse Natural Language Processing (NLP) [37] and Bioinformatics tasks [38] for capturing long-range dependencies and relatedness of sequence residues as well as improving the generalizability of predictive pipeline. Histone-Net makes use of neural language modelling to generate a rich distributed representation of histone sequences. Inspiring from the extensive usage of FastText model to generate word or higher order residue embeddings in an un-supervised manner for diverse NLP (e.g., text classification) [37, 39] and Bioinformatics tasks (e.g., protein family classification [40], enhancer prediction [41], n6-methyladenine sites prediction [42]). Histone-Net generates un-supervised higher order residue embeddings (DNA2Vec) of histone sequences. Furthermore, Histone-Net presents a different application of FastText model, where it incorporates histone occupancy and modification information while learning higher order residue embeddings (SuperDNA2Vec) of histone sequences. To investigate which distributed representation leaning scheme better captures coarse-grained and fine-grained relations of higher order residues, a rigorous intrinsic evaluation of both kinds of embeddings is performed by mapping high-dimensional feature space into low-dimensional feature space using Principal Component Analysis (PCA) and t-distributed Stochastic Neighbor Embedding (t-SNE) schemes. Extrinsic evaluation of both types of embeddings is also performed using three different machine learning classifiers (Random Forest, AdaBoost, Support Vector Machine).

Existing approaches [31‐36] have not been evaluated in cross-domain binary classification paradigm, where for each histone sequence analysis task, model is trained on one type of histone marker and tested on another type of histone marker. We explore the performance potential of proposed Histone-Net approach in cross-domain binary classification paradigm. Furthermore, a critical analysis of existing computational approaches reveals that in all existing approaches, 10 different checkpoints are obtained by rigorously training the single model separately over 10 benchmark genomic data sets to predict histone occupancy, methylation, and acetylation areas in histone sequences. In this strategy, one needs to know the target histone marker beforehand to select appropriate checkpoint amongst all model checkpoints while making prediction over unseen histone sequences. More recently, [31] developed a deep learning approach “DeepHistone” to simultaneously predict different histone markers associated with particular sequence. However, DeepHistone is only capable to detect the type of histone marker modification and unable to predict histone occupancy and modification levels. Inspiring from the work of Yin et al. [31] who treated the identification of histone markers as multi-label classification problem, we develop a multi-label classification paradigm to deal with the expensive overhead of generating separate model checkpoints for ten benchmark data sets belonging to three histone sequence analysis tasks. More specifically, in multi-label classification paradigm, performance of Histone-Net is evaluated in terms of its ability to simultaneously predict histone marker type, its occupancy, acetylation, and methylation levels. A comprehensive evaluation of proposed approach under the hood of intra-domain and cross-domain binary classification as well as multi-label classification paradigm proves the dominance of proposed approach over state-of-the-art predictor, its generalization potential across multiple histone markers as well as power to simultaneously predict histone type, its occupancy, acetylation, and methylation areas using a single deep learning model.

This section illustrates different modules of computational framework Histone-Net, benchmark binary classification data sets, the process used to develop a multi-label classification data set, and evaluation metrics used to evaluate the integrity of Histone-Net in binary and multi-label classification paradigm.

Proposed computational framework Histone-Net is implemented using Scikit-Learn [55] and Pytorch [56]. To perform a fair performance comparison of Histone-Net predictive methodologies with state-of-the-art histone occupancy and modification predictor [36], following Yin et al. [36], in both adapted DeepHistone [31] and proposed Histone-Net approach, randomly chosen 90% sequences are used for training and 10% sequences are used for testing. From 90% training sequences, 10% sequences are used as a validation set. We perform a large scale experimentation to develop an optimal model for histone occupancy and modification prediction. We assess the performance of DNA2Vec and SuperDNA2Vec sequence embeddings of 8 different dimensions (25, 32, 50, 64, 75, 100, 128, 150) using three different machine learning classifiers (RFC, AdaBoost, SVM). These sequence embeddings are prepared by averaging the statistical vectors of higher order residues present in them. We find that DNA2Vec and SuperDNA2Vec 100-dimensional sequence vectors mark best performance for intra-domain and cross-domain binary classification paradigms, whereas 64-dimensional sequence vectors perform better for multi-label classification paradigm. In all settings, embedding generation model is trained for 10 epochs, where we tweak the dropout from 0.1 to 0.5 only during SuperDNA2Vec embedding generation. From different batch sizes (32, 64, 128, 256), learning rates (0.001-to-0.008), and decay rates (0.91-to-0.99), proposed deep learning approach performs better when it is trained with a batch size of 64, Adam [57, 58] optimizer decay rate of 0.95, and learning rate of 0.008.

To find optimal hyperparameter values for machine learning classifier, we tweak quality of split, number of estimators, kernel type, degree, gamma, and penalty parameter using GridSearch [59]. We find that tree-based machine learning classifiers perform better with gini criteria using 50 number of estimators, discriminative classifier SVM performs better with radial basis kernel, degree of 2, penalty parameter (C) of \(2^{-5}\) and gamma value of 0.001. After finding optimal DNA2Vec and SuperDNA2Vec sequence vectors as well as hyperparameter values, we perform experimentation with 11 different higher order residues ranging from 2-to-12 to determine which higher order residue-based sequence embeddings comprehensively help the classifier to make accurate predictions. For 10 benchmark data sets of 3 different histone sequence analysis tasks, we generate 99 (tasks/unique data set groups * k-mers * machine learning classifiers = 3 * 11 *3) predictive pipelines for DNA2Vec sequence embeddings and 330 (data sets * k-mers * machine learning classifiers = 10 * 11 *3) predictive pipelines for SuperDNA2Vec embeddings. Proposed Histone-Net approach generates 110 (data sets * k-mers * deep learning classifiers = 10 * 11 *1) predictive checkpoints. From different higher order residues, we find that 7-mers to 11-mers sequence embeddings mark best performance. More specifically, 7-mers DNA2Vec and 11-mers SuperDNA2Vec sequence embeddings mark best performance across all classifiers. To evaluate adapted DeepHistone approach [31] across 10 benchmark histone markers data sets, we utilize the source code and parameters provided by Yin et al. [31].

This section performs comprehensive extrinsic and intrinsic evaluation of DNA2Vec and SuperDNA2Vec sequence embeddings. It compares the performance of proposed approach with machine learning classifiers, adapted convolutional neural network-based approach DeepHistone [31], and state-of-the-art image representation-based predictor HCNN [36].

Histone-Net web server makes the lives of genomics researchers and practitioners easier by facilitating an interactive and user-friendly web interface capable to perform robust histone sequence analysis. Unlike other web server developed for biomedical sequence analysis which only supports inference on new sequences and even that for one particular task. Histone-Net web server can be used to perform and visualize a multi-dimensional exploratory analysis of histone sequences. In addition, it can be used to train diverse predictive pipelines from scratch, tweak most crucial hyper-parameters, inference on new histone sequences for a variety of histone sequence analysis tasks including histone occupancy, acetylation, or methylation level prediction under binary and multi-label classification paradigm. Different modules of Histone-Net web server provide interactive session artifacts which can be downloaded and used for various purposes.

Researchers have experimented with a variety of statistical representation learning approaches and strategies (from distributed representation to attention mechanism) to capture relatedness of residues, their diver interactions, and distribution among different classes. This paper develops unsupervised higher order residues embeddings of histone sequences using FastText model and explores a different application of FastText model to develop SuperDNA2Vec which encapsulates histone occupancy and modification information while learning higher order residues embeddings in a supervised manner. It presents a computational multi-paradigm framework Histone-Net to perform a comprehensive intrinsic and extrinsic evaluation of 2 differently learned embeddings using 3 machine learning classifiers. In addition, it develops a precisely deep neural network Histone-Net for robust histone occupancy, acetylation, and methylation prediction. A comprehensive empirical evaluation of Histone-Net in intra-domain and cross-domain settings under the hood of binary and multi-label classification paradigms proves its effectiveness over state-of-the-art, generalization potential across multiple histone markers, and aptitude to simultaneously predict histone type, its occupancy, acetylation, and methylation levels.