Recent advances in computer-aided text analysis (CATA) have allowed organizational scientists to construct reliable and convenient measures from open texts. As yet, there is a lack of research into using CATA to analyze responses to open survey questions and constructing text-based measures of psychological constructs. In our study, we demonstrated the potential of CATA methods for the construction of text-based job satisfaction measures based on responses to a completely open and semi-open question. To do this, we employed three sentiment analysis techniques: Linguistic Inquiry and Word Count 2015, SentimentR and SentiStrength, and quantified the forms of measurement error they introduced: specific factor error, algorithm error and transient error. We conducted an initial test of the textbased measures' validity, assessing their convergence with closed-question job satisfaction measures. We adopted a time-lagged survey design (Nwave 1 = 996; Nwave 2 = 116) to test our hypotheses. In line with our hypotheses, we found that specific factor error is higher in the open question text-based measure than in the semi-open question text-based measure. As expected, algorithm error was substantial for both the open and semi-open question textbased measures. Transient error in the text-based measures was higher than expected, as it generally exceeded the transient error in the human-coded and the closed job satisfaction question measures. Our initial test of convergent and discriminant validity indicated that the semi-open question text-based measure is especially suitable for measuring job satisfaction. Our article ends with a discussion of limitations and an agenda for future research.

Additional Metadata
Persistent URL dx.doi.org/10.1371/journal.pone.0226408, hdl.handle.net/1765/123401
Journal PLoS ONE
Citation
Wijngaards, I. (Indy), Burger, M.J, & van Exel, N.J.A. (2019). The promise of open survey questions - The validation of text-based job satisfaction measures. PLoS ONE, 14(12). doi:10.1371/journal.pone.0226408