The influence of data and harmful nodes is mitigated by the service development between fog networks and IoT/user devices, relying on trust management. Therefore, establishing a trustworthy environment for fog layers and IoT is essential. The generated data or tasks are not always guaranteed to be standard and might contain unusual statistics that raise questions about their validity. Similar to other distributed networks, it faces numerous security challenges, especially insider threats. Traditional security schemes differ in their approach; one balances energy consumption, security, and transmission performance to fulfill information convergence and control requirements, while the other creates a multi-dimensional selective transmission scheme to achieve true reality. This study introduces a novel routing algorithm based on a mixed fitness function that is energy-efficient and relies on two-way trust. The algorithm's primary components involve selecting safe nodes based on a tolerance constant and choosing the best nodes for routing from among these safe nodes. The proposed algorithm uses the Open Development Model Algorithm (ODMA) for clustering and integrates a multi-path routing strategy with a multi-hop communication mechanism both within and outside the cluster. Additionally, a combined fitness function considering factors such as energy, trust, quality of service, connection, distance, hop-count, and network traffic is used to determine the optimal and safest path. Various evaluation criteria are applied in simulations when a denial of service attack occurs. Test results show that the proposed approach outperforms others, achieving a higher confidence level of 0.72. It demonstrated improved detection accuracy of malicious fog access nodes in fog networks by 25%, 18%, and 16% compared to ERTWT, Twl-FTM, and TCO-TM, respectively. Furthermore, the job completion time was reduced to 3.8, 4, and 12 ms, respectively, compared to ERTWT, Twl-FTM, and TCO-TM. Overall, the evaluation criteria of the proposed algorithm have been improved when compared to existing secure routing algorithms.
