Skip to main content
Download PDF

Monitoring and evaluation of human resources for health: an international perspective

Human Resources for Health volume 1, Article number: 3 (2003) Cite this article

Abstract

Background

Despite the undoubted importance of human resources to the functions of health systems, there is little consistency between countries in how human resource strategies are monitored and evaluated. This paper presents an integrated approach for developing an evidence base on human resources for health (HRH) to support decision-making, drawing on a framework for health systems performance assessment.

Methods

Conceptual and methodological issues for selecting indicators for HRH monitoring and evaluation are discussed, and a range of primary and secondary data sources that might be used to generate indicators are reviewed. Descriptive analyses are conducted drawing primarily on one type of source, namely routinely reported data on the numbers of health personnel and medical schools as covered by national reporting systems and compiled by the World Health Organization. Regression techniques are used to triangulate a given HRH indicator calculated from different data sources across multiple countries.

Results

Major variations in the supply of health personnel and training opportunities are found to occur by region. However, certain discrepancies are also observed in measuring the same indicator from different sources, possibly related to the occupational classification or to the sources' representation.

Conclusion

Evidence-based information is needed to better understand trends in HRH. Although a range of sources exist that can potentially be used for HRH assessment, the information that can be derived from many of these individual sources precludes refined analysis. A variety of data sources and analytical approaches, each with its own strengths and limitations, is required to reflect the complexity of HRH issues. In order to enhance cross-national comparability, data collection efforts should be processed through the use of internationally standardized classifications (in particular, for occupation, industry and education) at the greatest level of detail possible.

Background

The importance of sound empirical evidence for informed policy decision-making and the monitoring of progress towards achieving health system goals and human resources for health (HRH) development in particular is widely recognized. Human resources for health are central to managing and delivering health services, and in most countries account for a high proportion of national budgets assigned to the health sector. Defining precisely HRH can help to identify opportunities and constraints and the potential impact of HRH on population health. HRH analysis can also be used as a tool to sensitize political and social stakeholders to the importance of addressing workforce issues, contributing to the consideration of HRH issues on the public and political agenda [1].

Despite the undoubted importance of human resources to the functions of health systems, there is little consistency between countries in how HRH strategies are monitored and evaluated. In terms of health system inputs, international assessments of health personnel or other non-monetary resources remain less widespread than comparisons of health care expenditures [2]. The global evidence on policy options for health systems tends to be weak. It has been acknowledged that no country has discovered an ideal model, and appropriate policies differ widely across country settings [3].

While each country's situation is unique, a comprehensive framework for examining the contribution of HRH to the achievement of health system goals can be a useful tool towards developing evidence-based policy options. The World Health Organization's analytical framework for health systems performance assessment describes the role of HRH in each of four main functions of health systems [4]. While the approaches used to implement this framework have been the subject of much scientific and political controversy [5–7], a general point of consensus is the critical role of HRH in ensuring that services are delivered effectively.

At the same time, despite the high profile accorded to HRH within both the developed and developing worlds, there has tended to be less discourse on broader health policies and the more specific HRH interventions required to achieve them. Often HRH policy and planning have been simply (and mistakenly) equated with training [8]. However, recruitment and retention of HRH may be affected by a number of different factors; for one, poor working conditions may be contributing to both numerical and distributional imbalances in many countries [9]. Assessing HRH impacts on the delivery of health care services should thus also take into account factors exogenous to the health system, including the heterogeneity of labour markets and the mechanisms for adjustment between supply and demand for health personnel.

This paper presents an integrated approach for developing an evidence base on HRH to support decision-making. Drawing on the WHO framework for health systems performance assessment, we focus on the methods and materials for monitoring HRH inputs. First we examine conceptual and methodological issues for selecting relevant indicators. Then we discuss means for enhancing cross-national comparability. We then critically review the main data sources that might be used to generate indicators. Lastly, we touch on statistical applications and present some empirical findings for illustrative purposes.

Framework and indicators for HRH assessment

The WHO framework on health system performance assessment is based on the concept of health action, and encapsulates any set of activities whose primary intent is to maintain or improve population health, enhance the system's responsiveness to the expectations of the population, and assure fairness of financial contributions to the system. To achieve these main goals, four functions are performed by health systems: financing, stewardship, service provision and resource generation (see Fig. 1).

Figure 1
figure1

Framework for the assessment of health systems performance

Full size image

In particular, health system financing refers to the process by which revenues are collected from primary and secondary sources, accumulated in fund pools and allocated to provider activities. Stewardship involves the aspects of setting, implementing and monitoring the rules for the health system; assuring a level playing field for all actors in the system (particularly purchasers, providers and patients); and defining strategic directions for health systems as a whole. Service provision is the combination of inputs into a production process that takes place in a particular organizational setting and that leads to the delivery of interventions. Resource generation refers to the production of inputs – particularly human resources, knowledge, and physical resources such as facilities, equipment and consumables – for the provision of services. While HRH are directly or indirectly related to each of these four functions and influenced by them, in this paper we will concentrate on assessing HRH inputs as they pertain to the latter two.

Different elements should be borne in mind when selecting and calculating appropriate indicators for HRH monitoring and evaluation. From the perspective of human resources as an input to health action, HRH can be broadly defined as "the stock of all individuals engaged in the promotion, protection or improvement of population health" [4]. This encompasses those working across different domains of health systems: public and private sectors; clinical, research and public health interventions; preventive and curative personal care; etc.

In general terms, a number of criteria have been identified as crucial for selecting indicators, including policy relevance, reliability, validity, simplicity and ability to aggregate/disaggregate information [10, 11]. Indicators on HRH are used for various purposes, such as measures of planning, implementing, monitoring and evaluating policies. To facilitate both the data collection and analysis processes, it is important to focus on a limited and essential number of indicators that are comparable and measurable regularly using standard data sources.

A series of components for selecting HRH indicators for the performance of the services provision and resources generation functions can be found in Fig. 2. The proposed list follows the general framework of health system performance assessment, classified in terms of the level of achievement, distribution (equity) and efficiency (productivity) of HRH [12]. This list is not meant to be exhaustive, nor is it necessarily expected that all these indicators will be used in any given HRH assessment. This will depend on the user's particular circumstances, with the eventual number reduced to an essential minimum in accordance with data availability and quality. This basic basket of indicators is also not intended to be restrictive, and focuses on indicators appropriate for quantitative analyses. Other indicators may be used as appropriate in the context of national and local operations or different types of study designs.

Figure 2
figure2

Selected indicators for HRH assessment

Full size image

Detailed descriptions of the key indicators, including formulas and potential sources for measurement, can be found in the Additional file. Depending on the data sources and classifications used, many of these indicators can be disaggregated by occupation (physicians, nursing and midwifery professionals, etc.), by administrative units (districts, provinces or states, etc.) or other sociodemographic characteristics (age, sex, etc.). At the same time, indicators for some topics highly relevant to HRH analysis must be further investigated and tested. This includes measures of labour relations or dissatisfaction (such as work stoppages or other absences), regulation of health professions (such as rules to ensure standards of training and practice), financing of HRH production (such as fairness of financing for training programmes), risk factors to health workers (such as susceptibility to violence or to contracting infectious diseases including HIV/AIDS) and physical conditions (such as lack of materials or state of the workplace).

Indicators of the health service provision function

Level

A range of indicators can be defined to measure specific aspects of the level of HRH as an input to improving the provision of health services. The starting point of many HRH assessments is the stock of health personnel. Indicators of stock or available supply are usually expressed in terms of densities of HRH with regard to the total population or the population of economically active age. The distinction lies in the capacity of the system versus the allocation of human resources to the system compared to those available. Skill-mix indicators are also commonly used to measure various HRH components in relative terms. These indicators, which can provide a pointer of the priorities and capacities of the system, compare one subgroup to another according to assumed differences in skill levels or skill specializations, such as the ratio of physicians to nurses, or of specialists to generalists.

External migration of health workers, especially highly skilled ones, has long been recognized as a problem for ensuring appropriate coverage of essential services in some countries. The proportions of foreign-born or foreign-trained among the national HRH stock constitute simple but informative indicators on the importance of international immigration within the health sector. On the other hand, they fail to capture the impact of emigration on sending countries or migratory trajectories among workers, which tend to be difficult to measure because of a general lack of reliable and comparable data from source countries.

The participation or not of those with a health-related vocational background in the labour market, and their ensuing participation in the health industry in particular, offer important information for health policy purposes. Indicators on labour force activities capture three main elements: participation (the proportion of individuals with health-related skills currently in the labour force), employment opportunities (the proportion with health-related skills currently employed), and retention (the proportion with health-related skills currently working in a health-related industry). Complementary indicators may include the proportion of health workers engaged on a part-time basis, or the proportion with more than one current job.

Another aspect of labour force activity is the categorization of institutional sector of the work location, usually as public or private sector operations. The public sector includes government-operated facilities and services, while the private sector consists of for-profit or not-for-profit activities implemented by private agents or businesses as well as nongovernmental organizations and religious or other charitable institutions. However, the distinction is not always clear-cut, with some health systems having publicly-owned facilities staffed mainly by privately operating health workers, or large numbers of health professionals working partly in the private sector and partly in the public sector (with this arrangement sometimes being part of their contract with the latter).

Information on income or wages among health workers is of value when discussing countries' health care financing options. In many countries wage costs (salaries, bonuses and other payments) are estimated to represent between 65% and 80% of renewable health system expenditures [13, 14]. A common indicator is the average annual income or occupational earnings. This can be measured in gross or net terms, depending on the nature of the data source. In order to account for differences in working times, an alternative indicator might be constructed as average hourly wages (gross or net). Other complementary indicators for assessing monetary incentives could refer to modes of remuneration (for example, the proportion of health care providers paid by salary, fee-for-service or capitation), or multiplicity of sources of remuneration (such as the proportion of health workers receiving wages from both public and private sector jobs).

Distribution (equity)

A breakdown of HRH by workers' sociodemographic or other characteristics can offer insight into imbalances within the health workforce. Inequitable occupational distributions by geographical location and by gender constitute the main areas of health policy concern for many countries. Geographical imbalances, and especially shortages in rural or poor areas, are reported to have a number of adverse consequences for health systems performance. Other complementary indicators for assessing equity in the health workforce include distributions of vacancy rates, turnover rates and relative wage rates by location (or other criteria).

Occupational clustering by gender is regarded as important not only for assessing equity in human resources opportunities, but also for health services planning. A distinctive feature of HRH in many countries is the high proportion of workers who are women. Studies have shown that increased participation of women in the medical field may be accompanied by differences in working patterns: female physicians are likely to work fewer hours than their male counterparts [15, 16] and to present different styles of care provision that may be reflected in the levels of patient participation [17]. Moreover it has been suggested that certain female-dominated occupations, notably in nursing and midwifery, often are not given a market value commensurate with their skill level, as the work is seen simply as "women's work" [18]. Analysis of gender imbalances in average occupational earnings may reveal the extent to which women and men have equal opportunities in career choice. Indices of inequality might also be used to highlight extremes across income distributions, which may be masked when considering averages alone.

Efficiency (productivity)

Increasing the productivity of health workers has been identified as one of the most cost-effective ways to improve health system capacity and performance [19]. Indicators of productivity include measures such as the ratio of workers' time spent providing health care services as compared to non-care services (meetings, travelling, reporting, etc.), the average number of ambulatory visits per working hour among providers of direct patient care, or the average number of immunizations administered per day by a given provider. Such indicators may reflect in part the intensity of work activities among health care providers, but different values for the indicators may also reflect differences in the underlying reality – for example, linked to the manner in which work is organized or the pattern of ailments. Complementary indicators of productivity may include the average number of hours worked per week among health workers, the ratio of health workers engaged on a part-time basis versus full-time basis, or the ratio of average working time among clinical staff to less-costly support staff.

Indicators of the resource generation function

Level

The proportion of those starting or completing training in a health-related field with respect to the total HRH stock offers insight into the renewal and loss patterns of the health workforce. Monitoring information about entrants/graduates at health training institutions enables anticipation of future HRH supplies in relation to exits due to retirement or external migration. Such indicators may be disaggregated by skill specialization, such as the ratio of entrants/graduates of medical programmes versus nursing programmes. In some cases, where information on numbers of academic entrants/graduates is difficult to obtain, a substitute indicator may be used as the proportion of the total HRH stock in the youngest age groups. This proxy measure can be used to assess the importance of ageing in the health workforce. However, cross-national and cross-occupational comparability may be hindered by differences in the average duration or age of academic training, where the number of trainees who eventually seek employment in the (national) health sector is low, where the number of workers trained abroad is high, or where there is considerable mobility in and out of the workforce over the active age span.

Assessing the education and training levels of the health workforce is a key element for policy-makers. The advance of complex health systems organizations and medical knowledge, as well as the introduction of sophisticated technology, mean that improvements in population health and welfare increasingly depend on the degree of educational attainment and renewal and maintenance of technical capacity among the health workforce. Education-relevant indicators for HRH assessment include the proportion of health workers with a tertiary-level educational attainment and the proportion having undertaken a continuous education course or programme within a specific period. It should be noted that such indicators may fail to capture the notion of quality of education and training and its adequacy regarding the needs of the population and the health care system.

Distribution (equity)

Equity indicators for resource generation show the proportional distribution of new entrants or graduates at health training institutions according to different criteria – for example, the distribution of health trainees according to gender, or their distribution in urban versus rural areas. Monitoring the latter can be especially crucial for assessing policy impacts in terms of imbalances and incentives for staffing in remote geographical locations.

Efficiency

The decision to train more students in the field of health has important financial consequences. The training costs per student might be substantial, especially for physicians and specialists, who must attend medical school for several years to acquire the proper qualifications. Measures of the current average and marginal financial cost of training allow projections of the total costs of training more students, and can be used to compare training costs within and across countries. This can refer to vocational training at tertiary-level educational institutions or through continuous education programmes. A complementary efficiency indicator could be the attrition rate at health training institutions – that is, the ratio of entrants to graduates per academic programme.

Cross-national comparability in HRH monitoring

The range of indicators necessary to describe the profile and monitor the progress of HRH is extensive, integrating many aspects such as occupation, industry, training and others. A number of challenges must be faced in enhancing cross-national comparability. The roles of health care workers vary from country to country, and the professions also have different national histories and cultures. It can be difficult to categorize data when professional boundaries are not well defined.

The use of internationally standardized classifications – such as the International Standard Classification of Occupations (ISCO), the International Standard Industrial Classification of all Economic Activities (ISIC), and the International Standard Classification of Education (ISCED) [20–22] – can provide the backbone for comparing HRH between countries and over time. Classifications provide a framework for the description and comparison of statistics by categorizing items according to shared characteristics. But international classifications are not widely used in health research, and their potential as tools for defining the scope of the health system including human resources and for conducting HRH analyses has generally not been met. Different user areas may have different degrees of interest in the various elements. Studies of health systems and HRH in many countries still apply classifications that are not always comparable with standardized categorizations, although use of or mapping to ISCO and other international standards is becoming more widespread. Even when international classifications are used, they are often not used to full advantage.

The International Labour Office's 1988 revision of the ISCO classification system (ISCO-88) pools occupational titles into a hierarchical four-digit system that can be aggregated to progressively broader groups, representing a value set with respect to the type of work performed or to be performed. Occupations are specified according to the precision needed in major (one-digit level), sub-major (two-digit level), minor (three-digit level) and unit (four-digit level) groups. The basic criteria used to define the system of major, sub-major, minor and unit groups are the "skill level" and "skill specialization" required to carry out the tasks and duties of the occupations [20].

The main occupations of interest with a health care-related specialization fall within two of the ten major groups: group 2 "professionals" (generally well-trained workers in jobs that normally require a university or advanced-level degree for recruitment) and group 3 "technicians and associate professionals" (generally requiring skills at a tertiary non-university educational qualification level). Health practitioners in the professional group include physicians, nursing and midwifery professionals and other health professionals, such as dentists, pharmacists and veterinarians. Among associate professionals are modern health associate professionals (such as medical assistants, dental assistants, physiotherapists, opticians and sanitarians), nursing and midwifery associate professionals and traditional medicine practitioners.

For HRH analysis purposes, it is essential to have jobs classified at least according to the three-digit ISCO-88 level or equivalent. Information at the two-digit level alone does not allow distinction of health occupations from other life sciences occupations. Moreover, different health occupations can be separately identified only at the four-digit level, such as physicians (code 2221) from other health professionals. And certain related occupations aside from medical and nursing practitioners are likewise identifiable only at the four-digit level, in particular medical equipment operators (code 3133), health and safety inspectors (code 3152) and institution-based personal care workers (code 5132).

Even when occupations are classified according to ISCO-88, comparability issues may arise. For example, while physicians are unambiguously identified among the professionals major group (code 2221), the classification of nurses and midwives is less clear, crossing two major groups where they could be recorded: "nursing and midwifery professionals" (code 223) or "nursing and midwifery associate professionals" (code 323). This distinction was designed to reflect differences in tasks and duties that may be a consequence of differences in work organization as well as in education and training. In some countries, the possibility of distinguishing between the two typologies of nurses and midwives remains of limited relevance; inadequacies in the reporting system or incomparability of the education systems and measures of technical capacity may mean that some nursing and midwifery jobs do not fit easily into these two categories.

In addition, health systems employ a large number of workers with non-health occupational backgrounds, such as administrators, accountants, drivers and other support staff. Capturing this wider range of workers entails consideration of occupations across all of the major ISCO-88 groups, so further information on industry may be required. The ISIC classification can form a basis for such analysis. Using ISIC's latest revision requires information compiled at the three-digit level, in order to be able to distinguish the health sector from social work activities [21].

Other relevant classifications include ISCED, a cross-nationally comparable instrument for definitions of levels and fields of education and training [22]; the Central Product Classification, covering all goods and services that are produced or traded, including health services [23]; and the International Classification of Status in Employment, designed to facilitate cross-national comparability in the production and presentation of statistics on jobs by types of economic risk and levels of authority [24]. Using combinations of these classifications when describing HRH it will be possible to identify, for example, employment in non-health activities among those with a health-related education, and employment in health activities with jobs that do not require medical or nursing skills.

In practice, the precision of coding to such standard classification systems will depend largely on the level of specification in the raw data. Accurate occupational, industry and training information is vital in assessing HRH, starting with the most basic of indicators on size and composition of the health workforce. Many countries use national coding systems for occupation and other pertinent areas. However, data on health workforces are more directly comparable once differences in the organization of national health systems and the educational systems for health occupations are attenuated through harmonized classifications. Use among all countries of ISCO-88 to the greatest detail possible, or mapping of national classifications to this standard, would greatly improve cross-national comparability and facilitate use of health workforce information for informed decision-making.

Potential data sources

Effective planning and management of HRH depend vitally on the availability of high quality and timely statistics. Despite a prevailing view that statistics on HRH are scarce, the sources that potentially can produce information relevant to this issue are quite diverse. Routine administrative records, censuses and household and establishment sample surveys are among the sources that allow researchers and decision-makers to calculate different indicators according to their goals and needs. Some of the published empirical studies on HRH have drawn on information in health-related professional registries [25], national accounts gathered by agencies of the United Nations system [26], or records of medical schools and faculties [27]. Other research agendas have focused on using demographic census data [28, 29], labour force survey findings [30], or specialized surveys of health human resources [31].

Two fundamental challenges for data compilation at the international level are to identify appropriate sources and to gain timely access to the data [32]. Frameworks have been formulated that can support efforts to coherently combine statistics from different sources and for different units [33]. However, in many countries, information on the health workforce is fragmented, and the statistics generated by these various sources have received limited public dissemination and generally been underused. Even in countries where updated data are available, it may be difficult to establish the size and composition of the health workforce because the collection and dissemination of representative occupational data using the ISCO-88 standard at the three- or four-digit level (or other comparable national classification) remains less widespread.

Population censuses

Demographic census results can be a valuable source for statistics describing HRH characteristics, in particular if they are available as microdatasets. They can provide accurate information on the total stock and composition of the health workforce as well as differentials by spatial units (administrative districts, states or provinces, regions, etc.), demographic characteristics (age, sex, migration status) and other socioeconomic characteristics (educational attainment, income level, sector of activity). Unlike survey data, censuses do not suffer from problems of sample sizes that are too small to allow estimates with the required precision, so they give researchers greater leverage to disaggregate indicators for various population and administrative subgroups. On the other hand, due to their generally shorter questionnaires, censuses generally contain only limited information on other aspects of labour force activities valuable for in-depth analyses (such as secondary employment or working hours). They are also characterized by longer periodicity (usually only once every ten years), and the results therefore are useful primarily for describing the structure of HRH, in the context of the structure of the population, and related long-term changes.

Many countries have collected information on occupation in their censuses for a long time. However, in many developing countries, especially in Africa, census results remain a greatly underused resource for HRH assessments, and microdata have not been available. Two main problems arise. First, the quality and content of census data can vary widely from country to country, and sometimes from one census to another within the same country, making cross-national and time-trend analyses difficult. For instance, with regard to the last two censuses in Kenya, occupational information was collected in the 1989 round but not in 1999. Second, as already mentioned, where available, it is necessary to have occupational information detailed at least at the three-digit ISCO-88 classification level or equivalent to obtain separate identification of health workers. Many countries disseminate statistics only at the one-digit or two-digit level. Even for developed countries, the classifications used for compiling many indicators relevant to HRH assessment – such as occupation, education and industry – differ from one country to another [29].

For some countries, analyses of census data can be facilitated through collaborative research projects aiming to harmonize microdata variables and structures for public use. Two main census microdata providers can be identified: the Integrated Public Use Microdata Series (IPUMS) and the African Census Analysis Project (ACAP) [34, 35]. Such projects process census microdata series in machine-readable form for multiple countries, and help disseminate the relevant documentation for scholarly research. Information on HRH drawing on occupation data is currently available for four out of the six countries for which microdata are disseminated through the IPUMS-International project (Kenya, Mexico, United States of America and Vietnam) [34]. ACAP has archived census microdata from at least six countries with occupation data detailed minimally at the three-digit ISCO level or equivalent (Gambia, Kenya, Mali, Nigeria, South Africa and Uganda), with plans to eventually facilitate distribution of the data with the aid of tools being developed for guided statistical application [35].

Sample surveys

Representative household or establishment sample surveys, such as labour force or facility surveys, can provide useful information on various aspects of the health workforce. Depending on the sampling frame and implementation, surveys may be representative of the population or health service delivery environment, and national generalization is possible.

Labour force surveys are the key instruments used by national statistical offices to obtain estimates regarding employment characteristics of the population. The target is usually the adult resident population classified as either employed, unemployed or not being in the labour force (i.e. inactive). Surveys collect information via household-based interviews (in person, by mail or by telephone) on a range of labour force and sociodemographic indicators, such as occupation, work status, wages and earnings, industry and educational attainment. Information on occupation among individuals in the labour force is usually processed in terms of the national occupational classification, which is often based on and sometimes mapped to the ISCO-88 standard.

Depending on the sampling and instruments used, labour force surveys may provide statistics on a wider range of topics than censuses, which usually do not enter into depth on labour force issues. They cover a more representative population than national health registries, which often focus only on the public sector. Sample surveys generally require fewer resources than censuses and can be used periodically to monitor the evolution of HRH and their impact on health service provision. Furthermore, detailed information of all aspects of health labour market mobility, job turnover rates, second jobs, etc., can sometimes be obtained only through surveys.

Another source highly relevant to HRH assessment is surveys of health service delivery points. These are generally designed to generate data on the characteristics of health care facilities as well as the composition and spatial distribution of health workers employed in them. Such surveys are usually completed by means of interviews and/or examination of relevant materials and documents in the sampled facilities. In order to allow generalizations, the sampling frame should be designed to ensure representation across various domains: facility type, urban/rural, public/private, etc.

The main limitations of facility surveys are associated with the difficulty in some countries of designing a representative sampling frame – due, for instance, to the absence of an accurate and updated enumeration list of service delivery points. This may be especially true in countries where registration and licensing systems of health care providers are poor, or exclude those outside the public health sector. In addition, the sometimes small sample sizes do not allow for disaggregation of HRH indicators at the subnational level. Confidence intervals and likelihood of statistical errors may also be large for certain indicators when focusing on specific occupations such as physicians or nursing and midwifery personnel, or when completeness of reporting is low.

While the content and design of sample surveys are highly diverse across countries and over time, important efforts have been made to harmonize data for public use in some countries as well as internationally through microdata archives. For example, the Luxembourg Income Study (LIS) undertakes a process to enhance comparability and dissemination of microdata from labour force and income surveys that had already been collected by the central statistical offices across participating countries in the Americas, Asia, Europe and Oceania [36]. Occupational data allowing differentiation of health workers are available for 18 countries through this source [30].

Routine reports

Routine administrative reports include information collected in an ongoing manner by national ministries (such as ministries of health, labour or education) and professional associations (such as regulatory or membership-based bodies for physicians, nurses and other health professionals), as well as other types of continually updated records – for example, registries on entry visas and work permits for international migrants. These sources are commonly used in countries to estimate their HRH stock. Some of these estimates can be obtained through international agencies or institutional federations that compile them from national sources. Routinely reported data provided by Member States form the basis for the WHO database on health personnel [37] and the WHO directory of medical schools [38].

Depending on the characteristics and content of each specific registry, information may be obtainable on HRH in terms of subnational distribution, skills mix, education and training, workforce activities, migration trajectories, nationality, gender or age distribution. Their main strength lies in the continuous nature of the data collection and processing, even if changes to the underlying regulations and administrative procedures may render comparisons difficult over time. Comprehensive data from ongoing national records can be a useful complement to periodic censuses and surveys, but are normally not comparable between countries.

The relevance of such data for reviews of national HRH situations depends on the fraction of the total number of establishments or personnel covered in the country. In many countries there is no regular recording of the numbers and activities of all health personnel, and some emphasize only the public sector or can have variable accuracy for rural areas. The direction and magnitude of biases from registries' data will depend on their specific characteristics. For example, an upward bias will be evident if the system has no mechanisms for the removal of individuals from the records upon retirement, external emigration or death. Another potential issue derives from the tendency for some personnel (notably physicians and, to a growing extent, nurses) to practice health care at more than one location, such as part-time in a public facility and part-time in the private sector. Unless this situation is reflected, information based on providers' registries can be subject to bias due to double counting and/or partial coverage.

Analytical techniques and applications

The empirical approaches that can be used for HRH analysis are numerous, and the selection of the appropriate technique (or techniques) for a given study will depend on many factors, notably the objectives of the study to be conducted, the type of information needed, and the data availability and quality. Such approaches can be broadly categorized as either quantitative or qualitative in nature.

Quantitative applications can range from descriptive techniques, such as univariate and bivariate frequency distributions of HRH by occupation and other background variables, to more advanced statistical methods. Summary descriptives are highly useful for visualizing complex data structures and integrating data items in graphical or tabular form for presentation. Advanced methods can be used for analysing the determinants of a specific HRH matter, involving techniques for testing the effects and interactions of certain variables while controlling for the effects of other variables (covariates). A wide range of models can be applied, depending on the variables and outcomes of interest. This can mean models that yield a single estimated equation with unique correspondence between coefficients and variables, or multiple equations and subsequent coefficients. They may involve time-series analysis or other types of smoothing techniques for projecting data in time/space based on observed trends. Among the advanced estimation methods for HRH analysis found in the published literature are multiple regression (to explain factors affecting the geographical distribution of physicians) [39] and proportional hazard models (to deal with issues of retention of nurses in the health system) [25].

Qualitative methods are techniques for examining non-numeric characteristics of the area of interest. They include the analysis of content from texts, focus group discussions, in-depth interviews and participant observations. Applications to HRH analysis may be especially pertinent for topics such as regulation of health professions, labour relations, and worker satisfaction and motivation (see, for example, [40]).

A strong HRH analysis should ideally be grounded in the basket of indicators previously discussed. For illustrative purposes, Figs. 3 and 4 present examples of how descriptive quantitative techniques can be applied for monitoring HRH within the framework for health system performance assessment. Drawing on one type of source, routinely reported data – as covered by national reporting systems and compiled in the WHO databases on the numbers of health personnel and medical schools – were aggregated by region and then organized in graphical form for presentation. Such data can be used in monitoring the service provision and resource generation functions, respectively.

Figure 3
figure3

Distribution of countries by stock of HRH, according to region, late 1990s

Full size image
Figure 4
figure4

Distribution of countries by density of medical schools, according to region

Full size image

As suggested by these findings, major variations in the supply of health personnel and training opportunities occur by region. For example, some 90% of countries in Africa and South-east Asia are characterized with a density of physicians of less than 50 per 100,000 inhabitants, and about 50% of countries have a similar density of nurses and midwives. In contrast, none of the European countries are marked with a comparably low density. Moreover, some 30% or more of countries in Africa and South-east Asia are ranked in the poorest category in terms of density of medical schools (one per 10 million inhabitants or more), whereas none are found among this group in Europe or the Americas.

Such analyses, while informative on HRH monitoring and evaluation from an international perspective, are subject to certain limitations. The classification of health personnel compiled through routine reports does not follow cross-nationally comparable standards. For one, the data fail to distinguish between professional nurses and midwives and their associate professional counterparts. They also make no obvious distinction between those who are actively working in health care services and those who are not. An important shortcoming of the data on numbers of medical schools is the lack of details on numbers and characteristics of entrants or graduates within these institutions.

No single data source can reflect the growing complexity of HRH issues. Rather, a variety of instruments – each with its own strengths and limitations – can be exploited to produce different types of statistics. The combination of complementary information from various sources can provide useful and rich information on the profile and other characteristics of the health workforce. Moreover, availability of different types of data sources can also serve as a control for the common information they collect, offering means for triangulation [41].

Figure 5 offers an illustration of the uses of triangulation for data quality control. The same indicator, in this case the ratio of physicians to nursing and midwifery personnel (indicator of skills mix), was measured for different countries through two different types of data sources: household-based labour force survey data (as compiled through the LIS project, see [30]) versus the routinely reported data. Simple linear regression was then used to characterize the relationship between the measures obtained from the two sources. Some of the observed discrepancies may be related to the occupational classification or to the sources' representation. The resulting parameters could eventually be used as a basis for adjusting reported numbers to compensate for data discrepancies (although before any adjustment is made, the assumptions underlying the evaluation method must be carefully considered).

Figure 5
figure5

Comparison of the ratio of physicians to nursing and midwifery personnel, according to source of data, 12 countries with developed market and transitional economies

Full size image

Discussion

The purpose of this paper was to improve knowledge about the information on HRH that researchers and institutions collect and why they do it. A number of indicators and data sources were discussed here, focusing on those being used for HRH monitoring and evaluation following the WHO framework on Health Systems Performance Assessment. The indicators were broadly categorized according to the service provision and resource generation functions of health systems. Issues of harmonization of data collection and processing methods for enhancing cross-national and time-trend comparability were addressed. Given that the roles and tasks performed by the same categories of health occupations might vary across countries, the occupational, industrial and educational classifications associated with the definition of HRH should be taken into account when conducting comparative analyses. In turn, the selection of an appropriate analytical approach will depend on the policy questions to be addressed, the resources available and the context of the study. Moreover, regardless of the methodology used, the ability to track time trends will generally be constrained by the periodicity of the data sources available.

Despite the existence of a number of reporting systems (censuses, surveys, routine administrative records, etc.) that can be used to examine HRH issues and their contributory factors, many of these sources remain underused and the information on current efforts for analysis is frequently incomplete. Few available sources have been designed with the sole intention of producing information on HRH. WHO is undertaking work to refine and expand knowledge on HRH issues, including compiling and analysing statistics from existing sources as well as statistics based on specially designed surveys. This strategy includes partnerships with ministries, central statistical agencies and other data providers, research centres, libraries and public health schools as part of a process of capacity building in countries. In particular, four main collaborative projects are currently being implemented for the compilation of an evidence base on HRH from an international perspective.

The World Health Survey (WHS) is being conducted in collaboration with Member States in more than 70 countries from all regions and levels of development, aiming to provide a wide range of quantitative information that can be used to assess population health and health systems performance [42]. A module on health occupations was developed for inclusion in the household-based survey in order to collect comparable baseline information on the current health workforce for primarily monitoring the services provision function. A standard questionnaire was designed to identify all adults in the sampled households who had ever been trained in a health-related field or engaged in the health workforce, with follow-up questions on occupation and training, main work activities, workplace industry and sector, income and methods of remuneration, migration and other topics. The survey instruments were pre-tested in 12 countries in 2002, and fieldwork for the main survey phase will have begun in participating countries by early- to mid-2003.

Another project is the development of an electronic database called the Global Directory of Health Training Institutions, which will provide information on the status and trends of health education worldwide. This is an expansion of the former listing of medical schools [38], building a wider database that also will include training institutions for other health professions such as nursing, dentistry and pharmacy. It will capture selected characteristics of these institutions, notably the numbers and characteristics of entrants and graduates, in order to be able to measure a range of indictors for monitoring the resource generation function.

In-depth assessments of human resources for health are being conducted in 2002–2003 in six developing countries (Chad, Côte d'Ivoire, Jamaica, Mozambique, Sri Lanka and Zimbabwe). The study involves collecting quantitative and qualitative data at the national, institutional and individual levels. Four areas are included in this assessment: regulation of health occupations, health training institutions, health care facilities and health care providers. In order to ensure national representation of facilities and providers, the guidelines for implementation recommend the use of complementary list- and area-sampling frames for coverage of the range of facility types, especially from the private sector, for which available lists are frequently incomplete [43]. The survey questionnaires are similar to those developed for the WHS and Global Directory of Health Training Institutions, but were designed to gather more detailed information on professional licensing and registration, labour conditions, supply and demand for health personnel, and other relevant topics in order to develop short-term and long-term recommendations for HRH policy and planning.

A fourth project is the creation of a meta-database of data sources on HRH. This involves the comprehensive review for all countries of sources of information on human resources in health systems, in order to maximize the stock of methods and materials for international analyses on HRH. The task entails an extensive search process using electronic library sources and World Wide Web-based searches to identify data sources, including general sources such as censuses, labour force surveys and other reporting systems as mentioned above, with special attention to availability of statistics on health occupations and labour market characteristics. In some cases special tabulations on HRH may be directly requested from countries or from other international agencies (such as ILO or EUROSTAT). The development of such a database will allow for distinguishing variables to support consistent cross-country comparisons and within-country in-depth assessments.

As previously mentioned, understanding of the growing complexity of HRH issues requires a variety of methods and materials. Improved coordination of data-gathering activities could act as a catalyst for improving the availability, quality and comparability of data for monitoring and reporting HRH initiatives. HRH assessments should therefore be constructed as a result of a collaborative process between all the health system stakeholders, including health care providers, local and government agencies, nongovernmental organizations and international agencies. It is desirable to further assess to what extent research efforts are concurring, what overlapping may exist, and how research may be improved by filling gaps. Regarding implementation, ways should be found to ensure that the data received and used by international organizations are the most recent and accurate, and are processed through the use of internationally standardized classifications at the greatest level of detail possible.

References

  1. 1.

    Dussault G: Analysis of human resources situation in health: why and how?. Paper prepared for Intercountry consulting on human resource development in the health field, Cairo. 15–18 February 1999

  2. 2.

    Anell A, Willis M: International comparison of health care systems using resource profiles. Bulletin of the World Health Organization. 2000, 78 (6): 770-778.

    CAS  PubMed  PubMed Central  Google Scholar 

  3. 3.

    Feachem R: Health systems: more evidence, more debate. Bulletin of the World Health Organization. 2000, 78 (6): 715-

    CAS  PubMed  PubMed Central  Google Scholar 

  4. 4.

    World Health Organization: World health report 2000: health systems – improving performance. Geneva. 2000, [http://www.who.int/whr2001/2001/archives/2000/en/index.htm]

    Google Scholar 

  5. 5.

    Almeida C, Braveman P, Gold MR, Szwarcwald CL, Ribeiro JM, Miglionico A, Millar JS, Porto S, do Rosario Costa N, Ortun Rubio V, Segall M, Starfield B, Travassos C, Uga A, Valente J, Viacava F: Methodological concerns and recommendations on policy consequences of the World Health Report 2000. Lancet. 2001, 357 (9269): 1692-1697. 10.1016/S0140-6736(00)04825-X.

    CAS  Article  PubMed  Google Scholar 

  6. 6.

    McKee M: Measuring the efficiency of health systems: The World Health Report sets the agenda, but there's still a long way to go. BMJ. 2001, 323 (7308): 295-296. 10.1136/bmj.323.7308.295.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Ugá AD, de Almeida CM, Szwarcwald CL, Travassos C, Viacava F, Ribeiro JM, do Rosario Costa N, Marchiori Buss P, Porto S: Considerations on methodology used in the World Health Organization 2000 Report. Cadernos de Saúde Pública. 2001, 17 (3): 705-712. [http://www.scielosp.org/pdf/csp/v17n3/4653.pdf]

    Article  PubMed  Google Scholar 

  8. 8.

    Martínez J, Martineau T: Rethinking human resources: an agenda for the millennium. Health Policy and Planning. 1998, 13 (4): 345-358. 10.1093/heapol/13.4.345.

    Article  PubMed  Google Scholar 

  9. 9.

    Egger D, Lipson D, Adams O: Achieving the right balance: the role of policy-making processes in managing human resources for health problems. Geneva: World Health Organization. 2000, Issues in Health Services Delivery Discussion Paper, no. 2., [http://www.who.int/health-services-delivery/disc_papers/Right-balance.pdf]

    Google Scholar 

  10. 10.

    Gravitz M: Méthodes des sciences sociales. Paris: Editions Dalloz. 1996

    Google Scholar 

  11. 11.

    Sadana R, Mathers CD, Lopez A, Murray CJL, Iburg K: Comparative analyses of more than 50 household surveys on health status. Geneva: World Health Organization. 2000, Global Program on Evidence for Health Policy Discussion Paper Series, no. 15., [http://www-nt.who.int/whosis/statistics/discussion_papers/pdf/paper15.pdf]

    Google Scholar 

  12. 12.

    Scientific Peer Review Group on Health Systems Performance Assessment: Final Report. Geneva. 2002, [http://www.who.int/health-systems-performance/sprg/report_of_sprg_on_hspa.htm]

    Google Scholar 

  13. 13.

    Kolehamainen-Aiken R-L: Decentralization and human resources: implications and impact. Human Resources for Health Development. 1997, 2 (1): 1-14.

    Google Scholar 

  14. 14.

    Saltman RB, Von Otter C: Implementing planned markets in health care: balancing social and economic responsibility. Buckingham: Open University Press. 1995

    Google Scholar 

  15. 15.

    Hojat M, Gonnella J, Xu G: Gender comparisons of young physicians' perceptions of their medical education, professional life, and practice: a follow-up study of Jefferson Medical College graduates. Academic Medicine. 1995, 70 (4): 305-312.

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Reamy J, Pong R: Physician workforce composition by gender: the implications for managing physician supply. Paper presented at 4th FICOSSER General Conference, Cuernavaca, Mexico. 28–30 July 1998

  17. 17.

    Higginbotham E: Celebrating women in opthtalmology. Archives of Ophtalmology. 1998, 116 (9): 1227-1228.

    CAS  Article  Google Scholar 

  18. 18.

    Salvage J, Heijnen S: Nursing in Europe: a resource for better health. Geneva: World Health Organization Regional Publications. 1997, WHO European Series, no. 74.

    Google Scholar 

  19. 19.

    HRH data: guidelines and data requirements for a human resources for health information system. [http://hrhtoolkit.forumone.com/mstr_hrh_data/mstr_hrh_data.pdf]

  20. 20.

    International Labour Office: International Standard Classification of Occupations: ISCO-88. Geneva. 1990, [http://www.ilo.org/public/english/bureau/stat/class/isco.htm]

    Google Scholar 

  21. 21.

    United Nations Statistics Division: International Standard Industrial Classification of All Economic Activities: ISIC Rev. 3.1. New York. 2002, [http://unstats.un.org/unsd/cr/registry/regcst.asp?Cl=17]

    Google Scholar 

  22. 22.

    United Nations Educational Scientific and Cultural Organization: International Standard Classification of Education: ISCED 1997. Paris. 1997

    Google Scholar 

  23. 23.

    United Nations Statistics Division: Central Product Classification: CPC version 1.1. New York. 2002, [http://unstats.un.org/unsd/statcom/doc02/cpc.pdf]

    Google Scholar 

  24. 24.

    International Labour Office: International Classification of Status in Employment: ICSE-93. Geneva. 1993

    Google Scholar 

  25. 25.

    Holmas H: Keeping nurses at work: a duration analysis. Health Economics. 2002, 11: 493-503. 10.1002/hec.747.

    Article  PubMed  Google Scholar 

  26. 26.

    Wharrad H, Robinson J: The global distribution of physicians and nurses. Journal of Advanced Nursing. 1999, 30 (1): 109-120. 10.1046/j.1365-2648.1999.01056.x.

    CAS  Article  PubMed  Google Scholar 

  27. 27.

    Lambert TW, Goldacre MJ, Parkhouse J: Doctors who qualified in the UK between 1974 and 1993: age, gender, nationality, marital status and family formation. Medical Education. 1998, 32 (5): 533-537. 10.1046/j.1365-2923.1998.00244.x.

    CAS  Article  PubMed  Google Scholar 

  28. 28.

    Johnston G, Wilkinson D: Increasing inequitable distribution of general practitioners in Australia, 1986–96. Australian and New Zealand Journal of Public Health. 2001, 25 (1): 66-70.

    CAS  Article  PubMed  Google Scholar 

  29. 29.

    Organisation for Economic Co-operation and Development: The contribution of census data to improving statistics on employment in health care. Paper presented at OECD Health Data National Correspondents Meeting, Paris. 2002, 25-26.

    Google Scholar 

  30. 30.

    Gupta N, Diallo K, Zurn P, Dal Poz M: Human resources for health: an international comparison of health occupations from labour force survey data. Syracuse, NY: Syracuse University. 2002, Luxembourg Income Study Working Paper Series, no. 331., [http://www.lisproject.org/publications/liswps/331.pdf]

    Google Scholar 

  31. 31.

    Kazanjian A, Pagliccia N: Identifying the population of health managers in one Canadian province: a two-stage approach. Evaluation and Program Planning. 1996, 19 (1): 95-102. 10.1016/0149-7189(95)00042-9.

    Article  Google Scholar 

  32. 32.

    United Nations Population Division: Measuring international migration; many questions, few answers. Paper presented at United Nations Population Division Coordination Meeting on International Migration, New York. 2002, 11-12.

    Google Scholar 

  33. 33.

    Hoffmann E: Developing labour account estimates: issues and approaches. In Household accounting: experiences in concepts and compilation. Vol. 2 Household satellite extensions. New York: United Nations Statistics Division. 2000, 21-40. Studies in Methods, series F, no. 75

    Google Scholar 

  34. 34.

    Integrated Public Use Microdata Series. [http://www.ipums.umn.edu]

  35. 35.

    African Census Analysis Project. [http://www.acap.upenn.edu]

  36. 36.

    Luxembourg Income Study. [http://www.lisproject.org]

  37. 37.

    WHO estimates of health personnel: physicians, nurses, midwives, dentists and pharmacists (around 1998). [http://www3.who.int/whosis/health_personnel/health_personnel.cfm]

  38. 38.

    World directory of medical schools. [http://www.who.int/health-services-delivery/med_schools/index.htm]

  39. 39.

    Wilkinson D, Symon B: Inequitable distribution of general practitioners in Australia: estimating need through the Robin Hood Index. Australia and New Zealand Journal of Public Health. 2000, 24: 71-75.

    CAS  Article  Google Scholar 

  40. 40.

    Newman K, Maylor U: The NHS Plan: nurse satisfaction, commitment and retention strategies. Health Services Management Research. 2002, 15 (2): 93-105. 10.1258/0951484021912860.

    CAS  Article  PubMed  Google Scholar 

  41. 41.

    Galin P: Guía para optimizar la utilización de la información disponible sobre empleo en el sector salud. In Relaciones laborales en el sector salud. Edited by: Dal Poz M, Galin P, Novick M, Varella T. 2000, Quito: Organización Panamericana de la Salud, 19-89.

    Google Scholar 

  42. 42.

    WHS: World Health Survey home page. [http://www3.who.int/whs]

  43. 43.

    World Health Organization: Assessment of human resources for health: survey instruments and guide to administration. Geneva. 2002

    Google Scholar 

Download references

Acknowledgements

An earlier version of this paper was presented at the international workshop on "Human Resources and National Health Systems: Shaping the Agenda for Action", Geneva, 2–4 December 2002. Funding for this research was provided by the World Health Organization. The opinions expressed here are those of the authors, and do not necessarily represent the views of WHO. The authors would like to thank Orvill Adams, Eivind Hoffmann and Suwit Wibulpolprasert for their useful comments.

Author information

Author notes

    Affiliations

    1. Department of Health Service Provision, World Health Organization, Geneva, Switzerland

      Khassoum Diallo, Pascal Zurn, Neeru Gupta & Mario Dal Poz

    Authors
    1. Khassoum Diallo
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Pascal Zurn
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Neeru Gupta
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Mario Dal Poz
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Mario Dal Poz.

    Additional information

    Competing interests

    None to declare.

    Authors' contributions

    All authors contributed to the design and writing of this paper, and read and approved the final manuscript.

    Khassoum Diallo, Pascal Zurn, Neeru Gupta and Mario Dal Poz contributed equally to this work.

    Electronic supplementary material

    Authors’ original submitted files for images

    Rights and permissions

    Reprints and Permissions

    About this article

    Cite this article

    Diallo, K., Zurn, P., Gupta, N. et al. Monitoring and evaluation of human resources for health: an international perspective. Hum Resour Health 1, 3 (2003). https://doi.org/10.1186/1478-4491-1-3

    Download citation

    Keywords

    • Health Workforce
    • Labour Force Survey
    • International Standard Industrial Classification
    • Health System Performance Assessment
    • Census Microdata

    Human Resources for Health

    ISSN: 1478-4491