Sustainability has become a key term for linking environmental, economic and social issues, in both the sciences and politics. Conceptions and frameworks of sustainability have thus arisen to evaluate agricultural systems on their sustainability. Within these conceptions and in political and scientific discourses, what can be understood as the social pillar of sustainability in agriculture varies greatly, especially in regards to the scope and the sustainability standards applied. While rural inhabitants have been subject of various ‘sustainability studies’, the consideration of the social dimension in agriculture is still rather underrepresented. Our conceptual framework can contribute to enhance the understanding of the social dimension of sustainability by utilizing a social science-based approach to comprehend the complexity of social interaction in agriculture: Based on Parsons' system approach, we capture the components of a social system that encompasses agriculture and its embeddedness in society. This includes all major actors, their interactions and institutions. Further, we develop Maslow's hierarchy of needs as well as the rights approach into a sustainability scale. We call the conceptual framework the sustainable agricultural social system. This general framework can later be adapted to local cultural and social settings, serving as a more comprehensive and flexible sustainability framework. •We identify problems in measuring social sustainability in agriculture.•We use the system approach to capture all social actors related to the farm.•We determine the social system's sustainability based on actors' needs and rights.•We merge the agricultural social system with the needs and rights approach to create and provide a conceptual frame for determining sustainable agricultural social systems.