(use with permission only) return to summary
Draft COST EFFECTIVENSS STUDY REPORT
FOR THE PDA DATA CAPTURE AND TRANSMISSION
September 2004
Chapter One: INTRODUCTION |
1.1 Introduction
The report begins with a brief review of the Health Management Information System (HMIS) of the Ministry of Health (MoH). The HMIS defines the global system within which the project being reviewed was piloted. More so, an understanding of the current system provides the basis for decisions regarding any possible alternative.
The objective of the HMIS in Uganda is to provide an integrated system of relevant and functional information on routine basis. It is designed for use at the health unit, health sub-district, district and national levels for planning, monitoring and evaluation of the health care delivery system. These critically important tasks are necessary in order to continually improve the quality of health care in Uganda. It is the official routine reporting system for the ministry of health. HMIS is implemented countrywide after testing and evaluating it.
1.2 Design of the HMIS System
The HMIS was designed for use at the various levels of the health system for planning, management and evaluating the health care delivery system. The design is aimed at making the system generate accurate, timely and relevant information in order to be able to fulfill the above objective. The system employs a number of tools ranging from the patient/client registers, which are the main primary source of data. The registers are transcribed to the databases, which in turn provide data to compile reports to other levels. Primarily HMIS is meant for use at the points of collection. Input, process, output and outcome indicators are easily generated from the system and the Ministry has made important steps in using the system to monitor the implementation of the Health Sector Strategic Plan (HSSP).
There are 36 different instruments (referred to as Forms, Reports, or Registers) used for recording and transmitting data/information within the HMIS. The Instrument Number and Titles are listed in appendix 1. Reports move from HCs to HSDs, Districts and the Central levels, and each level is required to provide feedback to complete the communication. Reports are shared weekly (notifiable diseases), monthly (OPD and IPD attendances, maternity, ENEPI coverage and FP uptake) quarterly, and annually (mainly on totals and items like buildings which do not continually change much). Thus, some of these instruments, such as registers, are used at the Health Center (HC) level while others (e.g. staff listing) are forwarded to the district level. Data collected at Health Centre II and Health Centre III are sent either to the Health Sub Districts (HSD) or directly to the District office on a weekly and monthly basis. Much of the data sent to the MoH is captured on Forms 105 (Outpatients) and 108 (Inpatients), which are summaries of over 20 Forms and Registers. The forms for collecting the data at the districts and lower level units are provided by the MoH to the.
The technical persons responsible for data collection at the lower levels are the Health Center In-charge and Records Assistants. At the District level, the Director of District Health Services (DDHS) and the HMIS Focal Person and/or Surveillance Focal Persons are responsible for handling and forwarding the data to the Resource Centre on a monthly basis. At the Resource Centre the data management is handled by a team, headed by an Assistant Commissioner, and comprised of Statisticians, Medical Officers and a Systems Administrator.
Although different district have set for themselves deadlines by which to send the data to the Resource Center, the MoH requirement is that this should be done by the 28th day of the following month. Figure 1 below shows the flow of information from the lower to the higher levels. The Feedback to lower levels also follows the same path downwards.
Figure 1: Flow of Data and Information within the HMIS
1.3 About the Uganda Chartered HealthNet
Uganda Chartered HealthNet (UCH) is a local non-government organization registered and operating in Uganda. UCH was started as the Uganda HealthNet Project in 1998 jointly by Makerere University Faculty of Medicine (MUFOM), and Satellife, a United States based not-for profit organization. UCH's mission is to address the information needs of health workers in Uganda through promotion of use of affordable and appropriate Information and Communications Technologies (ICT).
The establishment of UCH is premised on the following notions:
- That good health is the foundation on which healthy communities are built.
- That there is a wealth of health information, which ought to be accessed and utilized by all health workers in Uganda.
- That advances in ICT make it possible for starved and isolated health workers to access and communicate with colleagues elsewhere.
UCH therefore, was started with the following main objectives:
- To promote the use of, and access to health information, using ICTs in all health related services, research and training; through training, consultancy and support.
- To create and maintain a functional network of health information consumers in Uganda.
- To solicit for financial, material, and technical support to facilitate all the above.
UCH is actively involved in partnerships both locally and internationally in lobbying and and advocating for support to develop and implement projects geared towards the achievement of the organizational mission. Collaborating with partners has proven useful in the past and present, which includes, inter alia, Satellife Inc. IDRC, Digital Partners, World Research Institute, Bridges.Org, Ugandan Ministry of Health, and Makerere University Faculty of Medicine.
As a project (Uganda HealthNet Project), participated in a three-country study testing for the value and viability of Hand held computers in Africa. With funding from the Acumen Fund a study was conducted in Ghana, Kenya, and Uganda where Immunization Agents, Medical students and doctors respectively, were equipped with hand held computers for use as tools for data collection as well as digital repositories of health information which could be accessed and utilized anytime anywhere for health care.
HealthNet has also participated in imparting computer literacy to members of staff and students of the Faculty of Medicine, staff at Mulago Teaching and Referral Hospital as well as Mulago School of Nursing and Midwifery. UCH is a pioneer Service Provider of email services in the country through its HealthNet project. Currently, UCH is the implementing partner of the Uganda Health Information Network Project developed jointly with Makerere University, Satellife and funded by IDRC. The UHIN project is about equipping health workers with handheld computers to collect data and access health information over the available Cellular networks.
1.4 Background and Rationale of the Study
The study follows the introduction of hand held computers or Personal Digital Assistants (PDA) in the data entry and transmission processes, henceforth referred to as the PDA system. The PDA system was introduced to work parallel with the existing manual system thereby making it easy to compare costs associated with each system. Note that in this report, the terms manual system has been used to imply the traditional system while the PDA system refers to the two-way information network that used the handheld computers, Jacks, the cellular network and other ancillary equipment.
In light of the vital role played by timely complete and accurate information in planning and executing effective health systems, and the limited resources, this study set out to review the cost effectiveness of two different methods of data collection and transmission in the two sample districts of Mbale and Rakai. The basic rationale for the cost effectiveness study is that resources available are scarce in relation to needs, and therefore it is imperative that they be used in a way so as to yield the largest possible benefits. Thus, cost effectiveness analysis was defined as an evaluation that considers both the costs and consequences of alternatives.
The benefits from the project were to be defined in terms of direct benefits, such as benefits resulting from savings or changes in the composition of inputs (paper costs, printing costs, transport costs, storage costs, staff time, etc), and indirect benefits such as lives saved or increased productivity as a result of getting timely information .
1.5 Objective of the Study
The major objective of the study was to analyze the cost-effectiveness of the two-way electronic information network established by UHIN vis-à-vis the traditional paper or manual approach in the districts of Rakai and Mbale. The study set out to identify and compare the costs of accessing, sharing and communicating information between health care providers, managers and policymakers using this technology versus the existing alternatives for the overall goal of improving the quality of the health of the people of Uganda. By comparing the project’s costs and benefits as well as measuring its health impact the study provides recommendations as to whether the project is viable and the network could be replicated elsewhere in Uganda. Thus, the results of the study are expected to guide policy makers to make informed decisions before recommending full scale roll out to other districts, sectors and countries.
1.5.1 Specific Objectives of the Study
- To develop a clear methodology and guideline that could be used in other settings for carrying out the cost effectiveness study of the two-way information network that uses handheld computers, Jacks, the cellular network and other ancillary equipment
- To assess the economic and health benefits derived from investments in the establishment of the network
- To evaluate the benefits, costs and effectiveness of the network and recommend future directions
1.6 Terms of Reference
The methodology should develop comparative data sets that would help make a detailed analysis of the network in terms of costs, benefits and relative cost effectiveness of the network vis-à-vis the traditional approach. In developing the detailed and comprehensive methodology for the study, the team was to take into account the following:
- Identifying the costs associated with the network and the traditional approach
- Identification of both direct and indirect benefits associated with the introduction of the network
- Identifying data sets required to determine the direct and indirect benefits and designing appropriate data collection mechanism
- Developing a mechanism for determining the monetary value of the direct and indirect benefits (it may not be possible to make valuation of all benefits, but this should be done as it is the core element in the cost effectives analysis)
- Determine the cost effectiveness of the network by making an analysis of the costs and benefits of the network vis-à-vis the traditional approach.
1.7 The Study Team
The following team of professionals both in health economics and practitioners in the health sector was formed for the purpose of undertaking the study.
- Dr. Grace Murindwa (Planning Division, Ministry of Health)/Team Leader.
- Dr. Fred Kakongoro Muhumuza (Research Fellow, Makerere University Economic Policy Research Center) - Secretary.
- Dr. Eddie Mworozi (Consultant/Honorary Lecturer Faculty of Medicine, Makerere University) – Member.
- Dr. Sarah Asimwe (HMIS Section, Ministry of Health) - Member
- Mr. Fred Kakaire (Chief Executive Officer) - Member
- Mr Isaac Shinyenkwa (Makerere University Economic Policy Research Center) – Member.
- Mr. Amos Nzabanita (Bio-statistician, Ministry of Health) – Member.
- Mr David Walter Dongo (UHIN - Project Administrator/Accountant)
1.8 Structure of the Report
The rest of the report is structured as follows. Chapter two presents the methodology and the analytical framework used in the identification and valuation of costs and benefits of the two systems. Chapter three is comprised of the computation of costs and benefits. The study reports ends with findings, conclusions and recommendations.
| Chapter Two : METHODOLOGY AND ANALYTICAL FRAMEWORK |
2.1 Introduction
As per the terms of reference, the methodology should develop comparative data sets that would help make a detailed analysis of the network in terms of costs, benefits and relative cost effectiveness of the network vis-à-vis the traditional approach. The study makes an underlying premise that comparing effectiveness without measures of costs could provide inappropriate information for decisions, just as in the case of comparing costs without measures of effectiveness. To this end, the study sought to establish the nature of costs and relate it to the effects of related activities .
2.2 Methodology
It was preferred to adopt the ingredients method, which relies upon the notion that every intervention uses ingredients that have costs . At the core of the methodology was the need to identify each specific ingredient and its associated cost. Once this was done, it was possible to establish cost effectiveness in terms of additional costs associated with the additional benefits. Thus the initial stage involved a description of the interventions in terms of resources that are required to produce the various outputs. This stage-involved determination of the inputs/ingredients used in the establishment of the intervention.
In implementing the first stage caution was taken to separate the inputs of the PDA system from those of the entire or general HMIS of which the PDA system is a sub-component. Thus interviews were held with responsible persons to derive their time charts in order to establish the proportion of time allocated to the PDA system. A similar approach was adopted for physical resources that are jointly used/shared with other interventions. As noted earlier, the PDA system was just a subsystem of an existing HMIS.
2.3 Sample Selection
The study was conducted in both districts of Mbale and Rakai. In each of the districts the sampled units included 2 HSDs, 8 HC IIIs and 16 HC IIs. In total 52 health units were sampled. Other institutions included the office of the District Director of Health Services (DDHS), members of the Health Management Committees, the office of the Chief Administrative Officer, the Secretary for Health.
2.4 Specification of Ingredients
Dividing ingredients into the following main categories facilitated the identification and specification of ingredients namely: (1) physical structures including equipment and materials; (2) personnel; and (3) other inputs. An additional category of inputs by beneficiaries of the system was ‘loosely’ included in the data collection instrument much as it was not expected to future highly, given the fact that the main beneficiary (Ministry of Health) already had much of the required inputs.
2.2.1 Personnel
Personnel ingredients included all of the human resources required for the alternatives that were to be compared. This category included not only full-time personnel, but also part-time employees, and any consultants or volunteers. All personnel were listed according to their roles, qualifications, and time commitments. Roles referred to responsibilities, such as administration, coordination, training, and so on. Qualifications referred to the nature of training, experience, and specialized skills required for the positions. Time commitments referred to the amount of time that each person devotes to the intervention in terms of percentage of a full-time position. There were certain employees, who allocated only a portion of a full work-time (week, month or year) to the intervention.
2.2.2 Facilities
Facilities referred to the physical space required for the intervention. This category included offices, storage areas, and other building requirements, whether paid for by the project or not. Even donated facilities were identified. All these requirements were listed according to their dimensions and characteristics, along with other information that is important for identifying their value. For example, offices with glass windows have a different value from those with wooden window. Any facilities that are jointly used with other programs were also identified according to the portion of use that is allocated to the intervention. For example, as a standard procedure, the Jack and the PDA storage space were estimated at 0.75 square meters.
2.2.3 Equipment and Materials
These referred to furnishings, instructional equipment, and materials that were being used for the intervention. Specifically, they included office furniture as well as such instructional equipment as computers, scientific apparatus, books and other printed materials, office machines, paper, and various categories of supplies. Both the specific equipment and materials solely allocated to the intervention and those that are shared with other activities were noted.
2.2.4 Other Inputs
This category included all other ingredients, which did not fit readily into the above three categories. For example, it was necessary to include any extra liability or insurance that was required, and the cost of training sessions. Other ingredients included telephone service, electricity, Internet access fees, and so forth.
2.3.5 General Comments on the Process
There were, at least three overriding considerations made in identifying and specifying ingredients.
- The ingredients were specified in sufficient detail that their value could be ascertained at a later stage of the analysis. Thus, it was important that the qualifications of staff, characteristics of physical facilities, types of equipment, and other inputs be specified with enough precision so as to place a reasonably accurate cost value on them.
- The categories into which ingredients were placed had to be consistent. This means that the categories had to be functionally useful in identifying all the ingredients.
- The degree of specificity and accuracy in listing ingredients depended upon their overall contribution to the total cost of the intervention. For example, equipment and personnel costs represented the bulk of the costs. Accordingly, these were given the most attention, on the assumption that categories with a big cost component can distort the results grossly in case an error is made at the identification and cost-establishment stages.
2.3 Identifying and Measuring Benefits and Costs
The process included comprehensive estimates of the projected benefits and costs for all alternatives. Benefits to which a dollar value cannot be assigned (intangible benefits) should be included along with tangible benefits and costs. Intangible benefits should be evaluated and assigned relative numeric values for comparison purposes. For example maximum benefits a value of 1. Evaluating and comparing benefits that have both dollar values and relative numeric values requires extra effort, which allows subjective judgment to be a factor in the analysis.
2.4 Sources of Information
In order to collect the relevant information about the project, it was necessary for the research team to familiarize themselves with the intervention. The familiarization exercise was mainly done in the following three ways: (1) through a review of program documents, (2) through discussions with individuals involved in the intervention, and (3) through direct observation of the interventions during pre-study visits to the pilot districts. During these visits, preliminary interviews were held with key persons involved in the project and the entire health system. Interviews with individuals involved in the intervention, including project designers, administrative staff, hospital personnel of various ranks and support staff.
Apart from generating initial information for the study, the visits also helped to identify the key ingredients and to categorize them as project or non-project-provided. The ingredients had to be identified in sufficient detail in order to be able to attach values on them, but also in order to obtain a clear description of the architectural layout and interface with the whole health system in general and the HMIS in particular. Only after this exercise was it possible for the team to design and complete the research instrument for the detailed data collection exercise.
As already noted, one of the essential starting points was the examination of project documents. These included general descriptions of the project prepared by program staff and outsiders, budgets and expenditure statements, web sites, reports from previous evaluations of the project, and internal communications such as memos and official e-mails.
One of the objectives when conducting interviews was to confirm or contradict the impressions left by documentary evidence. Thus, in reading, interviewing, and observing, the team was searching for agreement and disagreement across sources. Ultimately, the various sources of information aided the process of triangulating upon a reasonable set of cost ingredients.
2.5 Costing of the Ingredients
During this stage cost-values were placed on each ingredient or resource. Given that this was a review of a pilot lasting eight months, emphasis was typically on annual costs rather than costs of a longer time horizon or the entire life of the project. During the costing exercise, the major areas of concern involved imputing value on resources that are not found in standard expenditure or budget documents. These include donated inputs such as volunteers or in-kind contributions that are not found in any official reporting of expenditures or costs. Furthermore, there were investments of a capital nature, which typically last many years but are often paid for and shown as a cost in a single year.
Personnel costs were easily estimated by combining salaries and benefits. These were then scaled down to correspond with the actual time that each individual person spends on project related work.
The cost of facilities could not easily be established with precise accuracy since most of the health entities already had their facilities and it was not obvious what cost-value amounts to the use of any particular portion of the facilities. However, standard techniques for estimating their annualized value, which were adopted, included determination of what it would cost to lease them as well as methods of determining annual costs by estimating replacement value. The annualized value of a facility comprised the cost of depreciation (that is, how much is “used up” in a given year of a facility with a fixed life) and the interest forgone on the undepreciated portion. The same was true of equipment such as furniture and computers or materials such as textbooks that have a usable life of more than one year. Consumable inputs such as energy and telephone costs or supplies were imputed from expenditure records.
2.6 Computing Cost-Effectiveness
The analysis of cost-effectiveness of the two systems in accomplishing the set objectives involved the combination of information on effectiveness and costs of each system. The purpose was to determine, which subsystem provided a given level of benefits at the lowest cost – most cost-effectively. The approach’s key strength was its usefulness in evaluating alternatives that have a limited number of objectives (and measures of effectiveness). We note that in cases where there are multiple measures, cost-effectiveness analysis becomes inappropriate.
2.6.1 Estimating Effectiveness
Estimation of effectiveness was guided by the following questions: What measures of effectiveness should be used, and are they reliable and valid? What evaluation design should be used to gauge the success of an alternative in altering effectiveness (e.g., experimental, quasi-experimental, or non-experimental)? Will the design be successful in establishing a cause-and-effect relationship between the alternative and the measure of effectiveness (that is, will estimates of effectiveness possess internal validity)?
In order to design a measure of effectiveness, it is important to establish means of estimating benefits or utility, which would in turn set the final objective (effect) that one would then consider the cost of attaining in order to deduce the most cost effective alternative of achieving it. Though, there would be nothing wrong with specifying a full range of measures and evaluating alternatives according to their success and costs, it is often problematic to arrive at summative conclusions regarding the overall effectiveness or cost-effectiveness of a particular alternative. It may turn out that one alternative is most effective or cost-effective when assessing one outcome, but not in the case of another.
The alternative was to obtain a single measure that summarizes the overall “utility” or satisfaction that stakeholders obtain from the various project components and, therefore, combine information on several domains of effectiveness. The techniques for doing so are largely structured around the concept of the “multi-attribute utility function”, where the various “attributes” are analogous to measures of effectiveness.
The overall utility was expressed as:
U (x1, x2) = w1U1 (x1) + w2U2 (x2)
Where x1 and x2 capture the various attributes or qualities of data that have been obtained by the different systems, and U1 (x1), for example, captures the change in the quality of information generated from the data resulting from data attribute x1. This is multiplied by w1, which is the utility weight that is attached to this particular attribute. This method is particularly applicable in health related studies (see Saint, Veenstra, & Sullivan, 1999). In general, all the utility weights should sum to 1 such that w1 + w2 = 1.
In particular, the two systems were evaluated with regard to their ability to provide the following attributes of data:
- Timeliness, including getting the data to its destination in the shortest time possible;
- Few or no errors in the data including ability to crosscheck the data sets for any outliers, which may not be of a natural cause;
- Completeness of the data sets both in terms of records and fields; and
- Minimization of data losses.
Obtaining the estimate of overall utility required two key elements in addition to estimates of effectiveness. First, it was necessary to estimate the function that indicates how much utility is obtained from additional units of a given attribute; that is, one has to estimate the functions U1 (x1) and U2 (x2). Second, there was a need for estimating the utility weights, w1 and w2 that indicate the relative importance of each attribute in overall utility. Both tasks were accomplished through questions intended to elicit the preferences of key stakeholders such as members of hospital management committees, Medical Superintendents, Ministry of Health, and District Political leaders .
2.7 Imputing Monetary Values
In some cases, health outcomes can be expressed in monetary terms. There are three general approaches to doing so: (1) standard evaluation designs, including experimental, quasi-experimental, and non-experimental; (2) contingent valuation; and (3) observed behavior . Only the first has been applied with any consistency in cost-benefit analysis in several fields of health and education.
2.7.1 Standard Evaluations
Evaluation designs are overwhelmingly used to evaluate outcomes that are not expressed in pecuniary terms, such as state of health. In economics, there is an extensive non-experimental literature that explores the links between measures of data quality and eventual value of the information. However, in the case of outcomes that are not measured in monetary terms, they were readily converted using information from the responsible officers. For example, in the evaluation of data accuracy and completeness, the number of errors prevented or the reduction of gaps in the data, as noted by the HMIS focal persons at the district headquarters, was used to measure outcomes on these attributes. Ranking this benefit on a bi-polar scale of 1 to 5, in comparison with the old system, made it possible to gauge the magnitude effectiveness of the PDA system.
2.7.2 Contingent Valuation
A second method of valuing benefits is referred to as contingent valuation. It calls upon individuals to honestly assess their maximum willingness to pay for a particular outcome. In the cost-benefit literature, it has found its most frequent use in environmental evaluations (e.g., Cummings, Brookshire, & Schultze, 1986; Mitchell & Carson, 1989). Researchers in contingent valuation have developed a wide variety of methods for eliciting willingness-to-pay estimates from individuals. These are summarized by Boardman et al. (1996) and generally rely upon interview techniques that describe a hypothetical good or service that is subsequently valued by individuals. The methods are subject to some weaknesses as they may yield unrealistic benefits.
However, working closely with the various staff involved, it was possible to impute values to the time-savings that the PDA system has generated in terms of data entry, and the potential savings in terms of walking to present the form as opposed to beeming at the nearest Jack.
2.8 Discounting of Outcomes and Costs
This evaluation was confined to a period of less than one year such that it was not necessary to carryout any discounting of both costs and benefits. However, the expected costs and benefits over the long run have been incorporated in the narrative and interpretation of results in order to guide the rollout process, especially with regard to costs on the MoH and other beneficiaries.
2.9 Assessing the Distribution of Outcomes and Costs
It is always necessary to analyze how the outcomes of projects are distributed across different groups of individuals or institutions. Ultimately, this can affect conclusions about the relative cost-effectiveness of alternatives for different institutions. For example, the Ministry if Health is expected to face high initial costs in terms of capital investments and higher running costs in terms of personnel if the rollout involves substantial levels of new staff. To this end, the study makes suggestions to the most cost effective way of managing the rollout, and recommends avoidance of establishing parallel systems or over involvement of additional staff. The PDA system was found to have high initial costs but very minimal running costs such that attention would largely be placed on the coverage of initial costs.
2.10 Interpretation of Costs and Outcomes
There were two additional steps in a cost analysis. First, costs and outcomes had to be jointly interpreted in order to rank alternatives from most desirable to least desirable. Second, an assessment was done as to whether the conclusions were robust to variations key assumptions of the analysis. This was typically accomplished with a sensitivity analysis. Stability in the costs of the most important parameters, both local and foreign (buildings, personnel, and equipment) made it necessary not to carry out any intensive sensitivity analysis beyond 3 – 5 percent.
2.11 Ranking Alternatives by Cost-Effectiveness
Which alternative provides a given outcome for the lowest cost? This was determined by calculating a ratio of costs to outcomes. In a cost-effectiveness analysis, the cost effectiveness ratio (CER) of each alternative would be obtained by dividing the cost of each alternative (C) by its effectiveness (E), that is:
CER = C/E
This was interpreted as the cost of obtaining an additional unit of effectiveness. The alternative with the smallest ratio is relatively more cost-effective; that is, it provides a given effectiveness at a lower cost than others and is the best candidate for new investments. The interpretation of these ratios is subject to a caveat as comparisons are valid for alternatives that are roughly similar in scale. This was well taken care of given that the analysis considered only small subsystems of the entire HMIS.
In this case, however, the incremental effectiveness approach was preferred given the way in which the benefits or effectiveness was computed. The valuation of benefits was based on using the current manual system as the baseline, such that individuals were expected to infer the additional benefit or cost arising from use of the new system. An aggregation of a bi-polar rating scale produced the respective percentage (positive, zero of negative) depending on whether the PDA system had more benefits, nothing or reduced on the benefits. Thus, the CER was given by the ratio of cost differences expressed as a ratio of the cost of the manual system and the additional benefit (incremental value) offered by the PDA system (positive or negative). The results are presented in the next chapter.
| Chapter Three: COMPUTATION OF COSTS AND BENEFITS |
3.1 Introduction
This chapter contains findings in terms of actual ingredients, interaction processes, costs and effectiveness of the HMIS subsystems that were looked at. It is worth noting that the PDA covered only a subsystem of the entire HMIS such that it was necessary to consider only the areas where comparison would be appropriate. The findings, therefore, mainly consider certain elements of data capture, entry, transmission and usage in the making of decisions. In order to get clearer picture, out of sample projections are made, largely on grounds that the sample data converged fast enough to imply strong similarities across all units within and out of the sample. Otherwise, the sample, in each district included 2 HSDs, 8 HC IIIs, and 16 HC IIs.
3.2 Physical Structures: Equipment, Materials Related Facilities
Physical structures mainly included computer related hardware and software where information was collected on the following aspects: the manufacturer, the make, the model, the year of manufacture, the expected life and the cost. Additional information was sought on the maintenance requirements and possibilities for upgrading, the operating systems supported, network systems and operating characteristics including the screen size, CPU speed, memory sizes and hard drive capacity. For other physical facilities, the following information was desirable: the space available for use in terms of data capture, entry, storage and transmission; capacity and number of the personnel involved and the cost implications.
In order to limit the study to comparable areas, the project start-up costs were omitted from the analysis so as to capture the operational abilities of the PDA system. Such costs, however, would be crucial in the design of the roll out, which is a stage beyond the scope of this study much as this report makes a crucial contribution to the roll out decision process.
The office space considered was that used by the personnel involved in the data handling activities as identified in chapter two. The space could be either for day-to-day office operations or storage of data and related equipment. Computation of the cost of office space was based on an average estimate of a standardized office environment based on survey data about the characteristics of the different offices, the size of the offices, their location (urban or rural outposts), and the prevailing market rates.
Having obtained the cost of office space, the cost was apportioned between information and non-information related activities, on the one hand and between the manual and the PDA systems on the other. The disaggregation was based on information obtained from the responsible health officers at the various levels. An average was computed for each of the personnel categories as some officers, such as the DDHS have less data and information generation activities compared to fulltime Records Assistants and HMIS Focal Persons. The monthly cost was multiplied by the eight months that the project has been in operation to obtain the total cost of office space for each system.
The cots for other ingredients (Xi) were computed in a similar manner using the share of the assets or proportion of lifetime expected to be used by the project in the eight month period (ai) multiplied by the unit cost (Pi) and the number of assets (Ni) to obtain the total cost. As noted earlier, the total cost was apportioned according to the computed weights of usage of the asset by each of the two systems under comparison. Thus, ingredients such as PDA and Jacks had not officially been used by the manual system. The total cost (Cj) per each subsystem (j = 1 & 2) is expressed as a sum over all the physical ingredients (i = 1, 2, 3, … N).
Symbolically, this is expressed as follows:
Where;
Cj = The total cost associated to subsystem j,
ai = Share of asset value used over the period of analysis,
Xi = specific ingredient/asset
Pi = Unit cost of the ingredient/asset
Ni = Number of ingredients.
The results are indicated in table 4.1 below. The results indicate that the PDA system had more use of the physical ingredients mainly because of the electronic gadgets that the manual system did not use at all.
Table 3.1: Computation of Physical Ingredient Costs
Ingredient |
Qty |
Share of Asset used |
Unit Costs $ |
Total Cost ($) |
Man Sys Costs |
PDA Sys Costs |
Office Space |
113 |
0.66 |
22 |
1,641 |
492.2 |
164.1 |
Chairs |
150 |
0.13 |
15 |
293 |
87.8 |
29.3 |
Desks |
150 |
0.08 |
30 |
360 |
108.0 |
36.0 |
File Cabinets/Shelves |
113 |
0.06 |
45 |
305 |
274.6 |
15.3 |
Computers |
20 |
0.16 |
1,200 |
3,840 |
1,536.0 |
3,072.0 |
Printers |
15 |
0.20 |
400 |
1,200 |
360.0 |
480.0 |
UPS |
20 |
0.30 |
200 |
1,200 |
600.0 |
600.0 |
PDA |
200 |
0.16 |
100 |
3,200 |
0.0 |
3,200.0 |
Jacks |
20 |
0.10 |
1,000 |
2,000 |
0.0 |
2,000.0 |
Chargers |
50 |
0.45 |
100 |
2,250 |
0.0 |
2,250.0 |
Computer Accessories |
40 |
1.00 |
100 |
4,000 |
1,200.0 |
1,200.0 |
Paper |
40 |
1.00 |
4 |
160 |
24.0 |
8.0 |
Data Forms |
960 |
1.00 |
0 |
144 |
115.2 |
14.4 |
Total |
|
|
|
|
4,797.8 |
13,069.1 |
Source: Authors’ Computations
3.3 Personnel Costs
There were various categories of personnel who were directly involved in the PDA project either on full time or part time basis. These were located both at the Central level (Medical School) and within the districts. Some of the personnel were entirely paid for by the project while other were only be involved in the project on a partial basis. In order to minimize the computation problems, the various involvements were standardized through assessment of person hours allocated to data handling in general and individual systems in particular. In the case of the district staff, total remuneration included salaries and allowances of the responsible medical personnel and records officers.
The allocation followed information based on time spent on different activities in terms of related meetings, supervision, data capture, data entry, data processing and verification, movement costs other than transport. In addition, the staff provided a description of their involvement in the two systems for purposes or disaggregation of costs between the two systems on the one hand and any other activities on the other. It should be noted that the PDA picked information from the data already entered on the two HMIS forms within the manual system such that the biggest difference was in terms of: the number of duplicates of hard copy forms, travel times for taking data to the HSDs, and data entry by the HMIS focal person. In some cases, data entry times reduced by five hours per week while travel times declined from 8 hours to 30 minutes for some of the respondents.
The bulk of the staff (206 or 85 percent), on the out of sample basis, were lower cadre officers with salaries ranging between 50 and 100 US dollars, while allowances ranged between 30 and 60 US dollars. The inclination within the above salary range was up to 75 percent such that total district related personnel costs amounted to 144,200 US dollars, which was later split up between the various HMIS related activities in the proportion 18.0 percent with 5.0 percent attributed to the PDA system. The splitting was based on time allocations as given by the staff.
In addition, the PDA subsystem had other personnel costs in terms of trainers and technicians. Over the period of eight months, there were six trainers for the first four months and four for the rest of the period. Each trainer was paid US $ 1,250 per month bringing the total cost of trainers to US $ 50,000. Each of the two technicians was paid US $ 833 per month or US $ 13,328 for the entire period.
Thus, over the period of eight months, US $ 70,538 was allocated to the PDA system while 18,746 US dollars were allocated to the manual system.
3.4 Connectivity and Other Inputs
The category of other inputs included costs related to supportive systems such as training of staff, servicing equipment, connectivity. Connectivity costs were equivalent to 9,386.0 US dollars for the reviewed period of eight months . The counterpart cost of connectivity in the manual system was travel costs for data deliveries to the districts and the MoH. The cost of deliveries to the district headquarters from the various HCs was computed from the average distance (5km) and cost of travel per kilometer (US$ 1) that the hospital personnel had to travel to deliver the data, and amounted to US$ 400 per month for 80 units or US $ 3,200 for the entire duration. Secondly, the estimated cost for delivering data from the district to the MoH headquarters was US $ 100 per month per district, giving a total of US$ 1,600.
The equipment proved to be very versatile such that maintenance costs were very minimal. In fact other than the regular training of staff and rectification of errors on the PDAs, the bulk of the servicing costs was related to desktop computers, printers and related systems that largely belonged to the old system but were also available for use by the PDA system. Using market prices, these costs were estimated at 3,571 US dollars, of which 5 percent or US $ 178.6 was allocated to the PDA system, 18 percent or US $ 642.9 was allocated to the manual system. The rest of the costs were attributed to non-HMIS related activities such as day-to-day office management. The allocations are based on time-sharing experiences of the responsible staff.
In general the total cost of connectivity and other inputs related to the PDA system was US $ 9,564.6 while the corresponding amount for the manual system was equivalent to US $ 5,442.9.
3.5 Estimation of Benefits
The PDA system was mainly a subsystem for data entry, transmission and analysis, all of which support the generation of information for management of health systems. In this regard, medical personnel within the health facility or administrators mainly use health data and information, such that they formed part of the sample used in computing benefits. Secondly, focus was put on those who handle the data and generate the required information.
Consideration was given to benefits arising from the two sub-samples in terms of timeliness of the data and information, (reduced time lags), accuracy of the data (reduced errors), and completeness of the data (reduced data gaps). Since the PDA was still not officially and exclusively used by the decision makers at the MoH, the focal persons in this analysis were the HMIS Focal Person at the district, records Officers, Medical Personnel, and members of the Hospital Management Committee.
Inferential questions about costs or benefits arising from the different categories of activities (including data entry, query times, motivation, supervision, and what else one would do with any time savings or pay to realize such time savings. Other miscellaneous benefits include multiple uses offered by the PDA (calculator and reading News papers and access to medical journals on the web) and social prestige. A number of management officers, both from the day-to-day management of the health facilities and from the board (Health Management Committees) have shown great interest in the system and related data issues as shown by the higher frequency of visits to Records Assistants. This was presumed as an indication of improved supervision, which could be equated to the cost of hiring additional staff for such roles. Standard costs, normalized for the actual time that such a person would spend on supervision would be computed to arrive at the actual cost of the service.
National level costs and benefits were not included mainly on two grounds. First, it was necessary to keep the study within comparable subsystem components. There were wide disparities at the national level; for example, reporting was entirely based on the manual system guidelines, which had specified time lags (intended to allow all the data to come in from the various districts). In particular, the data from the manual system, which is used to make reports, is expected to reach the MoH by the third week of the month such that the benefits of PDA data arriving on time were lost out due to the structural design. Second, data cleaning and correction procedures were largely still confined to the district level.
The costing of benefits was based on two approaches depending on the nature of the benefit. All benefits were rated on a bi-polar scale ranging from maximum benefit (+2), no benefit (zero) to maximum negative benefit (-2). Using the baseline of the old manual system as the neutral position, such that in cases where the PDA system added a cost to the staff, a negative entry was recorded, the aggregate summations of the different individual ratings were used to infer the magnitude of deviation of the PDA system from the current manual system.
.
The values for each attribute of data (Av), for each of the various categories of staff (Fk,g) was obtained as a ratio of the actual sum of the ranks given by the staff category and the maximum possible ranking for the given attribute. Symbolically, this is expressed as follows:
Where;
V = any of the different eight attributes (A),
S = total sample size
K = (1, 2, 3, … S) number of staff in each category,
WF = weighted staff ranking,
TF = total possible weighting
The results for the rankings by selected attribute and category of staff are indicated in table 4.2 below.
Table 3.2: Ratios Rankings of Health Information Benefits by Staff Category
|
HMIS Officer |
Records Assistant |
In-Charge or MO |
Administrator |
Management Committee |
Data Accuracy |
0.85 |
0.70 |
0.40 |
0.80 |
0.40 |
Timeliness |
0.95 |
0.65 |
0.25 |
0.65 |
0.20 |
Querying |
0.95 |
0.30 |
0.30 |
0.20 |
0.20 |
Completeness |
0.95 |
0.70 |
0.55 |
0.30 |
0.50 |
Prestige |
0.50 |
0.90 |
0.50 |
0.25 |
0.15 |
Miscellaneous |
0.40 |
0.80 |
0.70 |
0.45 |
0.30 |
Supervision |
0.80 |
0.40 |
0.80 |
0.60 |
0.40 |
Motivation |
0.70 |
0.85 |
0.55 |
0.40 |
0.20 |
Total obtained/ Possible total |
0.76 |
0.66 |
0.51 |
0.46 |
0.29 |
Source: Authors’ Computations
The low ranking by the Administrators and Management Committee members indicates the degree to which such people interacted with the two systems in an independent manner. It did not matter how the information used by such people was generated, given the nature and frequency of their decisions, such that they could not single out the benefits of the PDA system over the traditional one. Thus, certain aspects such as timeliness, accuracy and completeness could not easily be captured from the ranking of such individuals.
The moderate rating by In-charges and Medical Officers (MOs) was largely a result of limited analysis and usage of the data by such staff in their day-to-day activities. Much of the data was collected for purposes of higher-level usage (more as a pre-condition) as indicated by the lack of updated analysis charts posted in most of the health facilities. In some cases, the displayed charts of patient numbers, for example, were for 2002.
Chapter Four: FINDINGS AND CONCLUSIONS |
4.1 Introduction
This section presents a summary of the findings from the computations of costs and benefits in the previous chapter. The results are then used to compute the cost effectiveness index and drawing of conclusions. It is worth noting that the Central level costs, both at the MoH and UHIN have been omitted from the analysis, though they form a major component of the recommendations. UHIN costs grossly affected the analysis as a major overhead project related costs that could not compare easily with the MoH related costs. Thus, the MoH structure dominates the recommendations. The costs are summarized in table 5.1 below.
Table 4.1: Summary of Costs Computations by Each Ingredient
Items Description |
Manual system |
PDA System |
Physical Ingredients |
4,797.8 |
13,069.1 |
Personnel |
18,746 |
70,538 |
Other Ingredients |
5,442.9 |
9,564.6 |
Total Cost |
28,986.7 |
93,171.7 |
