Online social networks (OSNs) are vital to people's daily lives. They offer free services that allow people to connect and interact with family and friends, post comments and images, express views on sports and politics, and influence other users on OSN. The significant risks to OSN security are malicious SMBs, and numerous studies have been done to identify them. This article, a Systematic Literature Review (SLR), aims to determine best practices in SMB recognition. This SLR covers research published between 2008 and 2022. As a result of this study, we classified OSN profiles into real, verified, and fake accounts. The malicious SMBs types are SMBs, spam bots, Sybil and cyborgs, stegobots, political bots, and game bots. We also proposed a classification of SMBs detection techniques, which are ML-based, DL-based, graph-based, anomaly-based, topic-modeling-based, DNA-inspired, genetic-based, and hybrid approaches. Additionally, our study revealed that most public datasets used for SMB detection only include some types of SMB, dependent on Twittersphere, include limited data, and needed to be more extensive, up-to-date and accurate. We studied challenges of SMB detection, like data labeling, features engineering, imbalanced data, etc. Finally, we proposed the opening axis for future works.