Let $G$ be a finite group and let $\rm{Irr}(G)$ be the set of all irreducible complex characters of $G$. Let $\rm{cd}(G)$ be the set of all character degrees of $G$ and denote by $\rho(G)$ the set of primes which divide some character degrees in $\rm{cd}(G)$. The character graph $\Delta(G)$ associated to $G$ is a graph whose vertex set is $\rho(G)$ and there is an edge between two distinct primes $p$ and $q$ if and only if the product $pq$ divides some character degree of $G$. Suppose the character graph $\Delta(G)$ is $K_4$-free with diameter $3$. In this paper, we show that $|\rho(G)|\neq 5$, if and only if $G\cong J_1 \times A$, where $J_1$ is the first Janko's sporadic simple group and $A$ is abelian.