Abstract Many deviations from rational choice imply the neglect of important evidence and suggest the use of simple heuristics. In contrast, other deviations imply sensitivity to irrelevant evidence and suggest the use of overly complex rules. The current analysis takes two steps toward identifying the conditions that trigger these contradictory deviations from efficient reasoning. The first step involves a theoretical analysis. It shows that the contradictory deviations can be captured without assuming the use of rules of different complexity in different settings. Both deviations can be the product of a reliance on small samples of similar past experiences. This reliance on small samples triggers apparent overcomplexity when the optimal rule is simple, but more complex rules yield better outcomes in most cases; the opposite tendency, oversimplification, emerges when the optimal rule is complex, and simple rules yield better outcomes in most cases. The second step involves a preregistered experiment with 325 participants (Mechanical Turk workers). The experiment shows that human decision makers exhibit the pattern predicted by the reliance‐on‐small‐samples assumption. In the experiment, participants chose between the status quo and a risky alternative in a multi‐attribute decision with three binary cues. They used uninformative cues when this strategy was best in most cases yet ignored two informative cues when this strategy was best in most cases. In addition, describing the cues as recommendations given by three experts increased the tendency to follow the modal recommendation (even when reliance on only one of the experts was optimal), but people still behaved as though they relied on a small sample of past experiences.