From Regex to Transformers: A Hybrid Framework for Cyber Threat Indicator Extraction from Unstructured Text

Automated extraction of Indicators of Compromise (IOCs) from unstructured cybersecurity threat intelligence reports remains a critical challenge due to the volume, variety, and complexity of modern threat data. This study introduces a hybrid and transformer-based IOC extraction framework that combines pattern-based regular expressions, spaCy Named Entity Recognition, and a fine-tuned DistilBERT token classification model, improving coverage and accuracy over traditional manual or rule-based methods. The approach is implemented as an end-to-end, fully automated pipeline integrating data preprocessing, token-level annotation, model training, inference, and an interactive deployment interface via Streamlit and a RESTful API, enabling real-time extraction and structured output for practical cybersecurity workflows. A comprehensive evaluation and visualisation framework, including token-level BIO-tagging, per-type and overall NER metrics, and interactive colour-coded entity highlighting with sortable tables, facilitates both quantitative assessment and intuitive interpretation of extracted IOCs. Experimental results demonstrate that the transformer-based model achieves perfect precision, recall, and F1-score, significantly surpassing baseline regex+NER methods. This framework provides a scalable, accurate, and practical solution for enhancing threat intelligence analysis and accelerating incident response.