In recent years, the need for high security with reliability in the wireless network has tremendously been increased. To provide high security in reliable networks, mobile ad hoc networks (MANETs) play a top role, like open network boundary, distributed network, and fast and quick implementation. By expanding the technology, the MANET faces a number of security challenges due to self-configuration and maintenance capabilities. Besides, traditional security solutions for wired networks are ineffective and inefficient because of the nature of highly dynamic and resource-constrained MANETs. In this paper, the researchers focus on improving reliable data transmission with high security in the MANET using an optimization technique. In the proposed MANET system, the nodes are clustered by utilizing an energy-efficient routing protocol. Then, the modified discrete particle swarm optimization is used to select the optimal cluster head. A secured routing protocol and a signcryption model can be used to improve the transmission security of the reliable MANET. The signcryption algorithm encrypts the digital signature, which can enhance the overall efficiency and confidentiality. The security-based analysis is performed on the basis of packet delivery ratio, energy consumption, network lifetime, and throughput. Finally, the result demonstrates that the MANET with optimization techniques achieves a high transmission rate and improves the reliable data security.