In recent years, the development and application of short questionnaires (“Q-short”) for psychological constructs has been gaining pace. At present, short measures are widely-used for psychological assessment in diverse domains, e.g. personality, social or I/O psychology, psychopathology, social and educational science, and behavioral economics, as well as diverse assessment settings such as research and practice. Their popularity is largely due to the promise of higher efficiency of measurement, lower cost, lower respondent burden, and higher data quality. Besides these obvious advantages, there is also considerable criticism for using short questionnaires of psychological constructs leaving researchers and practitioners in limbo concerning the choice of an appropriate measure for their assessment setting. As the criticism mainly pertains to the methodology of short scale development, the symposium focuses on the construction process. Presenters demonstrate and/or compare construction strategies such as manual and automated approaches (e.g. Ant Colony Optimization) or top-down strategies (starting with a longer version of a scale) and bottom-up strategies (starting with a single item to which further items are gradually added) using empirical as well as simulated data. Aim of the symposium is to make recommendations for the development, validation, and application of short questionnaires in research and applied settings.

The idea of interactionism suggests that human behavior is caused equally by the situation and the personality of the actor. However, personality tests and situation perception are scarce. Here, the B5PS, a test capturing the Big 5 and 42 facets as well as 5 dimensions of situation perception (Situation 5) is used as starting point for the development of a short test. This short test yields scores for the Big 5 and the Situation 5 and was evaluated using a representative sample of 400. During the talk, the construction strategy paying specific attention to the nomological network of the constructs assessed will be explained. Moreover, evidence for the psychometric quality of the short test will be reported and compared with the original test (criterion, convergent, discriminant, and factorial validity as well as construct and test-retest reliability). The mixed-method approach applied here can be generally applied in test construction and can serve as a best practice example.

The present study was aimed at constructing useable, reliable, and valid short scales of two measures assessing proactive personality and supervisor support. For this purpose, we compared Ant Colony Optimization (ACO; Leite et al., 2008) and classical item selection procedures. ACO is algorithm-based, and selects and compares sets of items according to defined criteria. For proactive personality, the two selection procedures (ACO and classical item selection) provided similar results. Both five-item short forms showed a satisfactory reliability and a small, however negligible, loss of criterion validity. For a two-dimensional supervisor support scale, ACO found a reliable and valid short form. Psychometric properties of the short version were in accordance with those of the parent form. A classical short form for supervisor support revealed a rather poor model fit and a serious loss of validity. Benefits and shortcomings of ACO compared to classical item selection procedures and recommendations of ACO application are discussed.

Psychological constructs have attracted increasing attention as valuable predictors of social phenomena. However, most psychological measures include too many items to be practically useful in large-scale research. Because of this, researchers often remove items from these long measures. By doing so, many researchers rely on well-known techniques such as maximizing coefficient alpha. Research has shown, however, that these strategies may result in serious deficiencies. Recently, psychometricians have developed sophisticated methods that are believed to result in sound short scales. From the viewpoint of practitioners, there seems to be little guideline on how to choose and apply these new techniques to optimally shorten a scale.

Against this background, the aim of our talk is three-fold: First, we explain the limitations of old approaches. Subsequently, we will introduce several state-of-the-art procedures. In this context, we will compare “top-down” and “bottom-up” strategies. Top-down approaches refer to item-selection process which start with a longer version of a scale. “Bottom-up” approaches, in contrast, start with a single item. Here, scale components are gradually added. We also distinguish between manual and automated approaches. We use both simulated and empirical data to evaluate these different methods. Finally, we provide recommendations for using appropriate procedures for constructing short measures.