Validating a work group climate assessment tool for improving the performance of public health organizations

Organizational climate and culture

The terms "organizational climate" and "organizational culture" are sometimes treated as different concepts because they arise out of distinct theoretical traditions, but they have many overlapping elements. Stringer explains climate as a subset of organizational culture [2]. Culture applies to the deeply rooted value systems inherent in all organizations and is difficult to change [3]. Organizational strategy, the external environment, organizational arrangements and historical forces all affect the context and milieu within which a work group operates. These "cultural" influences develop outside the work group and are beyond the direct control of the work group manager. Burke describes the relationship between climate and culture in a series of papers that discuss the modification of organizational culture at British Airways [4, 5]. Burke posits that changes to climate are more achievable than changes in culture, because climate is associated with the "transactional level of human behaviour – the everyday interactions and exchanges" [4].

Thus, while every organization has an organizational culture, each work group (or team) within the organization has its particular climate. Tagiuri defines organizational climate as a "quality of the internal environment of an organization that (a) is experienced by its members, (b) influences their behaviour and (c) can be described in terms of the values of a particular set of characteristics (or attributes) of the organization" [6]. Burke [4] stresses that leadership, mission, strategy and organizational culture have an organization-wide focus, whereas climate is experienced and created in the work group or team.

A work group's climate may be similar to or different from the overall organizational climate. High-performing work groups sometimes operate in organizations troubled by declining funding or inadequate leadership at the senior level. The leadership and management practices of a manager can create a positive work climate and strong results within a work group, even if an organization's climate is less than optimal. Regardless of a manager's level, his or her efforts to improve the work group's climate can contribute to strong employee performance and results.

Work group climate and performance

Positive work climate has been identified in a variety of environments as a driver of performance. According to the business literature, there is a positive correlation between climate and performance and also between climate and financial results. "Organizational climate is not the only driver of performance. Economic conditions and competitive dynamics matter enormously. But our analysis suggests that climate accounts for nearly a third of the results" [1].

While many of the factors such as an organization's history, culture, organizational strategies and structures are outside the control of the work group manager, the manager is uniquely placed to influence work climate within the work group. "What the boss of a work group does is the most important determinant of climate. The boss's behavior drives climate, which arouses motivation. And aroused motivation is a major driver of bottom-line performance" [2].

The work group manager, therefore, has a substantial impact on the development of work climate and the productivity of the work group. This relationship was clearly identified by a research project conducted at Harvard Business School in 1968. The project studied the relationship between motivation and organizational climate and reviewed the impact of different leadership styles on three evenly matched teams working on the same production project. The researchers demonstrated that leadership styles affected both the development of work group climate and the productivity of the three teams [7].

The impact of work climate is not restricted to the commercial sector. Research in the health and education fields supports the conclusions from the business literature. For instance, in a study of Canadian staff nurses, Laschinger, Finegan and Shamian describe the relationship between empowerment, job satisfaction, and commitment [8]. A positive work climate creates an environment conducive to the development of trust and empowerment, which in turn leads to high-quality patient care [8].

Positive climate has also been demonstrated to drive success in schools operated by the Department of Education and Employment in the United Kingdom. "Our research demonstrates a significant link between classroom climate and student academic progress. . . to the degree that teachers can develop skills and characteristics that impact climate, so they can hope to more effectively motivate and engage their students" [9].

Over the past decade a number of instruments have been developed to help organizations measure organizational climate, predominantly in US-based organizations in the private sector. However, these tools are often proprietary (and therefore expensive) or complicated and lengthy to administer. Surveys of organizational climate such as the PAHO Organizational Climate Instrument [10], and the Gallup Q12 [11] do not conform to the needs of small work groups. The 80-item PAHO Organizational Climate Instrument is extremely lengthy, and like the Q12, measures individual employees' satisfaction, rather than perception of the overall climate in a work group.

Description of the Work Group Climate Assessment Tool

The Work Group Climate Assessment Tool (WCA) is a self-administered assessment form originally consisting of 14 items: 12 that correspond to three sub-dimensions of climate – clarity, support and challenge – and 2 items that capture perceptions of productivity and quality. These sub-dimensions and the individual items are based on the work of George Litwin and Robert Stringer, who pioneered the study of climate in corporate environments [2, 7].

The WCA is designed to measure climate among intact teams or work groups in the health sector of developing countries. (An intact team is defined as a group of individuals who work together regularly at the same work site, whether in a central or regional office or a health facility.) The WCA is the first assessment tool that has been developed for this purpose. It is intended to measure climate in work groups at any level of an organization. To date, the WCA has been used to measure work group climate before and after the M&L Leading for Performance Improvement Program that was conducted in five sites – Egypt, Mozambique, Brazil, Guinea, and Kenya – and via a virtual distance-learning program for leadership development.

To apply the survey, all members of the work group (both managerial and staff) complete the assessment form. Each team member rates each item. The scores are then tabulated across all respondents, and results for each item and an overall climate score for items 1–12 are calculated for the team as a whole. Results for items 13 and 14 are calculated separately.

At the conclusion of the leadership program, the WCA is applied again among all team members. The post-intervention scores are again calculated for the team as a whole and then compared to the baseline team scores and targets to determine the amount of change produced by the intervention vis-à-vis the anticipated results (climate targets).

Initial validation of the WCA

M&L tested the WCA for face validity throughout 2002–2003 with counterparts in Brazil and Nicaragua as well as with several teams working on the M&L Program in Cambridge. In addition, the WCA was used to collect baseline and follow-up data among participants in the Leading for Performance Improvement Programs in Egypt and Guinea and participants in the Virtual Leadership Development Program in Latin America. The WCA was translated into Portuguese, Spanish and French and pre-tested in the different countries to make sure that it was appropriate for use across cultures. Based on feedback from the field tests, the WCA form and instructions were refined, and ultimately published in 2003 [12]. The expectation was that publishing the tool would allow M&L leadership programs to test and refine the tool in preparation for a full validation study and peer-review.

Based on the work of Litwin and Stringer [7], the original tool incorporated a measure of the importance of each item to the respondent in order to assist teams to prioritize the sub-dimensions that needed more attention. Managers participating in the pre-testing of the instrument, however, tended to rate the importance of all items quite high, and therefore the importance measure was not useful for determining the teams' priorities. Part of the purpose of the current study was to determine if the importance column could be eliminated without compromising the statistical validity of the tool.

Using the data collected during the field tests, M&L examined certain aspects of the tool's validity and reliability. For example, data from Brazil suggest that the tool has discriminant validity. The WCA was applied with three groups of managers in Brazil: one group was in the state of Ceará and had undergone extensive leadership training over a period of five years, while the other two groups were in states that had only begun to participate in leadership training. The spread in mean scores from a high mean in Ceará to much lower means in the other two states suggests the tool can discriminate between high- and low-performing work groups.

In terms of reliability, a Cronbach's alpha of 0.87 was calculated on 122 cases of WCA data collected in Latin America and Egypt. The coefficient alpha suggests that the items in the WCA have a high level of internal consistency. Results of an initial factor analysis conducted on the same data indicate that the assessment items load on a set of three to four factors. However, additional analysis on a larger data set was necessary to determine whether the identified factors relate to the hypothesized sub-dimensions of clarity, challenge and support.

While the WCA was initially validated using Cronbach's alpha reliability coefficient and exploratory factor analysis, a subsequent validation study was necessary to refine these analyses to account for nested data, provide item-level psychometrics and establish construct validity. This paper presents the results of this second validation study. Study participants were a purposive sample of present and past recipients of M&L technical assistance to strengthen management and leadership in the public health sector in developing countries. Participants came from ministries of health in Mozambique and Guinea, the Secretariat of Health for the State of Ceará, Brazil, and Brazilian public health laboratories. The participants represented a wide variety of positions, including central-level ministry staff, district-level managers, hospital administrators, laboratory technicians and clinic personnel.

The participants completed self-administered questionnaires anonymously in a group setting in each participating organization in May 2004. The survey contained two sections: The first consisted of the original 12 climate items from the WCA, the two productivity and quality items, and nine additional items generated to increase the item pool for measurement refinement. The second section consisted of 24 items from one section of the Stringer Organizational Climate Survey [7]. Participants rated each item on a Likert scale, where 1 = not at all, 2 = to a small degree, 3 = to a moderate degree, 4 = to a great degree and 5 = to a very great degree.

The Stringer survey served as the gold standard for this study. This instrument, which has been used repeatedly since 1968, was validated through studies that showed its association with objective measures of organizational climate in corporate settings in the United States [2].

The analysis data set consisted of data from 305 individuals in 42 work groups: Brazil (21 work groups, 182 employees), Mozambique (18 work groups, 97 employees), and Guinea (3 work groups, 26 employees). With 42 work group sites, we had a statistical power of 0.87 to detect validation correlations as low as 0.20 at the standard significance value of 0.05. Values outside the admissible range for a given variable were reassigned as missing. Four cases were omitted from the analysis sample due to miskeyed data.

Analyses included eigenvalue decomposition analysis, confirmatory factor analysis (which included tests for measurement noninvariance by gender, management status and educational level), and construct validity and reliability analyses of the WCA. Once a final set of items had been selected, reliability coefficients for both the work group and individual employee levels of analysis were computed.

Description of participants

Among participants from Brazil and Mozambique, women represented 62% of the respondents, consistent with trends of female participation in the health field. In these two samples, 59 respondents were managers (22%). Individuals had been employed in the health field for up to 40 years (mean [M] = 17.7, standard deviation [SD] = 9.7) and in the rated organization for up to 37 years (M = 11.5, SD = 9.2). Demographic information was not available for the three Guinean work groups.

Item weighting

Respondents to the WCA survey rate each item twice: first according to how the item was currently performed in the work group ("actual performance") and second according to the item's perceived importance to the work group. Models using the "actual performance" ratings were compared to those weighted by "importance" ratings. (A weighted score is the value assigned to the "actual performance" of a given item multiplied by the value of the "importance" assigned to that item.) It was found that in the importance-weighted models the factor structure was stronger, due to improved distributional characteristics of the individual items. As a result, all subsequent models in the study used importance-weighted scores for the WCA items. However, use of the WCA in the field suggests that the weighting of actual scores by importance scores is not easily understood by health managers and is not useful for prioritizing actions to improve climate. To determine if the weighting could be eliminated, we correlated mean individual scores from the "actual performance" ratings and those same scores weighted by the "importance" rating. This correlation was extremely high (R = 0.83, p < 0.01), indicating that the importance column could be eliminated without compromising the validity of the tool.

Eigenvalue decomposition analysis

The variance/covariance matrix of the 21 items from the WCA were submitted to an eigenvalue decomposition analysis, and the resulting eigenvalues were plotted (Fig. 1). (Items 13–14, measuring productivity and quality, were not included in the model and were analysed separately, since they are considered outputs of a well-functioning work group rather than components of climate.) The resulting scree plot clearly shows a unidimensional structure. Scree plots based on eigenvalues obtained from each of the three separate countries (not shown) also indicated a clearly unidimensional construct. Initially, models were conducted by disaggregating the items into the three hypothesized sub-dimensions: clarity, challenge, and support. While the factor loadings were strong for all three sub-dimensions, the factor intercorrelations were very high (0.81 to 0.95), and modification indices suggested that there was considerable cross-loading of items across factors. Because of a lack of factor discrimination and because the scree plot (Fig. 1) shows evidence of a single dimension, all subsequent models contain a single WCA factor.

Confirmatory factor analyses

An initial confirmatory factor model with the full set of 21 importance-weighted items was fitted to adjust for the clustering in the data. In this model and all subsequent models, missing data were handled by means of a maximum likelihood estimation technique, which minimizes bias due to listwise deletion [13]. Factor loadings from this model ranged from 0.37 to 0.68. One item had a factor loading below the standard 0.40 cutoff level [14] so it was omitted from further analysis models.

Noninvariance, or differential item functioning, across various individual characteristics (gender, management status and educational level) was tested by means of a series of multiple group confirmatory factor models in which factor loadings were constrained to be equal across groups. Using a cutoff of 1.00 and above for model-generated modification indices, we identified six items showing evidence of noninvariance across gender, three items showing noninvariance across management status, and three items showing noninvariance across educational levels. (Because there were only 16 respondents with only a primary school educational level, too few to allow estimation, we compared the groups that had finished secondary school [n = 81] and those who had completed university education [n = 160]).

A two-level confirmatory factor model was fitted to 12 importance-weighted WCA items that had at least a 0.40 factor loading in the previous models and that showed no evidence of noninvariance across gender, management status, or educational level. At the work group level of the model, the factor structure of employees' ratings, aggregated to the work group level, was examined for consistency across the 42 work groups. At the individual level of the model, the factor structure of the ratings was examined for consistency across employees within each work group. The factor loadings were constrained to be equal across these two levels of analysis so that a parallel interpretation of work group climate was maintained when considering each level of analysis.

Validation and reliability analyses

Once the factor model had been finalized, correlations between the WCA and a composite of the Stringer items were estimated at both the individual and work group levels. At the work group level, this correlation was extremely high (R = 0.93, p < 0.001), indicating that the eight-item WCA scale captured the same construct as the 24-item Stringer scale in differentiating the work group climate quality across work groups. The correlation at the individual level was moderate (R = 0.48, p < 0.001), indicating that the WCA scale captures a construct that is similar to but different from that captured by the Stringer scale in differentiating among employees' perceptions of climate within the same work group. The ways in which the WCA differs from the Stringer scale in assessing individual perceptions of the same work group climate require further study to be fully understood.

A reliability analysis of the data on individuals was conducted, without adjusting for the clustered design. The internal consistency in the 8 WCA items across individuals was good (α = 0.81). Since this scale is most commonly used to assess the work group itself, the data were restructured and a reliability analysis using the work group as the level of analysis was done. The internal consistency of the eight items across work groups (aggregated across employees within a work group) was also good (α = 0.86). Our final models contained eight items. Considering only the 42 work groups, this number gives us a ratio of more than five observations per factor, satisfying a common rule of thumb in psychometrics. Additionally, since the estimates of these items at the 42 sites draw from data on 305 individuals, these estimates are very stable, reliable and unbiased.