Description of the Work Group Climate Assessment Tool
The Work Group Climate Assessment Tool (WCA) is a self-administered assessment form originally consisting of 14 items: 12 that correspond to three sub-dimensions of climate – clarity, support and challenge – and 2 items that capture perceptions of productivity and quality. These sub-dimensions and the individual items are based on the work of George Litwin and Robert Stringer, who pioneered the study of climate in corporate environments [2, 7].
The WCA is designed to measure climate among intact teams or work groups in the health sector of developing countries. (An intact team is defined as a group of individuals who work together regularly at the same work site, whether in a central or regional office or a health facility.) The WCA is the first assessment tool that has been developed for this purpose. It is intended to measure climate in work groups at any level of an organization. To date, the WCA has been used to measure work group climate before and after the M&L Leading for Performance Improvement Program that was conducted in five sites – Egypt, Mozambique, Brazil, Guinea, and Kenya – and via a virtual distance-learning program for leadership development.
The WCA is divided into two sections. The first section includes 12 items, which were mapped to the three hypothesized sub-dimensions mentioned above of clarity, support and challenge. The items for the original WCA were the following:
-
1.
We are recognized for individual contributions.
-
2.
We have a common purpose.
-
3.
We have the resources we need to do our jobs well.
-
4.
We develop our skills and knowledge.
-
5.
We have a plan which guides our activities.
-
6.
We strive to improve our performance.
-
7.
We understand each other's capabilities.
-
8.
We are clear about what is expected in our work.
-
9.
We seek to understand the needs of our clients.
-
10.
We participate in the decisions of our work group.
-
11.
We take pride in our work.
-
12.
We readily adapt to new circumstances.
The second section, items 13 and 14, relates to perceptions of productivity and quality, which are defined for the respondent on the assessment form:
-
13.
Our work group is known for quality work.
-
14.
Our work group is productive.
To apply the survey, all members of the work group (both managerial and staff) complete the assessment form. Each team member rates each item. The scores are then tabulated across all respondents, and results for each item and an overall climate score for items 1–12 are calculated for the team as a whole. Results for items 13 and 14 are calculated separately.
At the conclusion of the leadership program, the WCA is applied again among all team members. The post-intervention scores are again calculated for the team as a whole and then compared to the baseline team scores and targets to determine the amount of change produced by the intervention vis-à-vis the anticipated results (climate targets).
Initial validation of the WCA
M&L tested the WCA for face validity throughout 2002–2003 with counterparts in Brazil and Nicaragua as well as with several teams working on the M&L Program in Cambridge. In addition, the WCA was used to collect baseline and follow-up data among participants in the Leading for Performance Improvement Programs in Egypt and Guinea and participants in the Virtual Leadership Development Program in Latin America. The WCA was translated into Portuguese, Spanish and French and pre-tested in the different countries to make sure that it was appropriate for use across cultures. Based on feedback from the field tests, the WCA form and instructions were refined, and ultimately published in 2003 [12]. The expectation was that publishing the tool would allow M&L leadership programs to test and refine the tool in preparation for a full validation study and peer-review.
Based on the work of Litwin and Stringer [7], the original tool incorporated a measure of the importance of each item to the respondent in order to assist teams to prioritize the sub-dimensions that needed more attention. Managers participating in the pre-testing of the instrument, however, tended to rate the importance of all items quite high, and therefore the importance measure was not useful for determining the teams' priorities. Part of the purpose of the current study was to determine if the importance column could be eliminated without compromising the statistical validity of the tool.
Using the data collected during the field tests, M&L examined certain aspects of the tool's validity and reliability. For example, data from Brazil suggest that the tool has discriminant validity. The WCA was applied with three groups of managers in Brazil: one group was in the state of Ceará and had undergone extensive leadership training over a period of five years, while the other two groups were in states that had only begun to participate in leadership training. The spread in mean scores from a high mean in Ceará to much lower means in the other two states suggests the tool can discriminate between high- and low-performing work groups.
In terms of reliability, a Cronbach's alpha of 0.87 was calculated on 122 cases of WCA data collected in Latin America and Egypt. The coefficient alpha suggests that the items in the WCA have a high level of internal consistency. Results of an initial factor analysis conducted on the same data indicate that the assessment items load on a set of three to four factors. However, additional analysis on a larger data set was necessary to determine whether the identified factors relate to the hypothesized sub-dimensions of clarity, challenge and support.
While the WCA was initially validated using Cronbach's alpha reliability coefficient and exploratory factor analysis, a subsequent validation study was necessary to refine these analyses to account for nested data, provide item-level psychometrics and establish construct validity. This paper presents the results of this second validation study. Study participants were a purposive sample of present and past recipients of M&L technical assistance to strengthen management and leadership in the public health sector in developing countries. Participants came from ministries of health in Mozambique and Guinea, the Secretariat of Health for the State of Ceará, Brazil, and Brazilian public health laboratories. The participants represented a wide variety of positions, including central-level ministry staff, district-level managers, hospital administrators, laboratory technicians and clinic personnel.
The participants completed self-administered questionnaires anonymously in a group setting in each participating organization in May 2004. The survey contained two sections: The first consisted of the original 12 climate items from the WCA, the two productivity and quality items, and nine additional items generated to increase the item pool for measurement refinement. The second section consisted of 24 items from one section of the Stringer Organizational Climate Survey [7]. Participants rated each item on a Likert scale, where 1 = not at all, 2 = to a small degree, 3 = to a moderate degree, 4 = to a great degree and 5 = to a very great degree.
The Stringer survey served as the gold standard for this study. This instrument, which has been used repeatedly since 1968, was validated through studies that showed its association with objective measures of organizational climate in corporate settings in the United States [2].
The analysis data set consisted of data from 305 individuals in 42 work groups: Brazil (21 work groups, 182 employees), Mozambique (18 work groups, 97 employees), and Guinea (3 work groups, 26 employees). With 42 work group sites, we had a statistical power of 0.87 to detect validation correlations as low as 0.20 at the standard significance value of 0.05. Values outside the admissible range for a given variable were reassigned as missing. Four cases were omitted from the analysis sample due to miskeyed data.
Analyses included eigenvalue decomposition analysis, confirmatory factor analysis (which included tests for measurement noninvariance by gender, management status and educational level), and construct validity and reliability analyses of the WCA. Once a final set of items had been selected, reliability coefficients for both the work group and individual employee levels of analysis were computed.