Course Catalog  |  Chapter Directory

NEXT

Module D > Data sources  
 

 

Data Sources 

 

How can you measure that?

Consider a cluster of questions when deciding how to measure an indicator:

• What data sources can give evidence about an indicator?
• What part of the target audience(s) is an indicator applied to?  
• What’s the data interval (that is, timing) required for gathering data? (before and after? Every year of the program?)

 

Data sources

Data sources are tools, documents, and locations for information that can be designed to show what happened in your target audience.

• Anecdotes
• Surveys or feedback forms
• Observation or assessment reports
• Participant projects
• Other organization’s records or test information

Anecdotes:

• Okay for attitudes, although you may just hear from happy people. 
• Worst for skills, as people will want to talk about what they’re good at.
• Good for information on outcomes you hadn’t anticipated.
• Good for quotable quotes.

Surveys or feedback forms:

• Surveys depend on “self-reporting”—fine for confidence in a skill, or interest in a subject—but not objective measures of skill levels or of behaviors.
• Best for attitudes because survey questions can include quantifiable rating scales.
• Weak for behaviors (depending on time between the experience and the survey), and weakest for skills.
• Surveys are good for important outcomes that you were able to anticipate.
• Look for standardized surveys (professionally and expertly designed) that cover the outcomes you’re interested in.  This may save time, but some may need to be adapted.  
  

Behavior observation or assessment:

• With observation, an expert or 3rd party professional organization with expertise watches participant behaviors—mostly during a particular program—and can document objectively whether skills or behaviors have actually changed. Professional assessments include data sources such as teacher checklist of student behavior, portfolio assessment.
• Best for skills and behaviors because they provide objective assessment by third-party observers.
• Bad for attitudes.

Participant projects:

• Best for skills and some behaviors.
• Weakest for attitudes because it is impersonal.
• Example: Tip sheets created by participants in a bird identification workshop show that they’ve internalized knowledge about bird characteristics enough to articulate them clearly.                      

   

  Coach

  Using participant projects, student and participant skills can be assessed through the use of rubrics that evaluate the level of skills.  Six steps to creating a rubric are: 1. Write the student/participant learning outcome; 2. Define the indicators comprising the learning outcome; 3. Identify the levels of quality for each indicator; 4. Assign a point value to each level of quality; 5. Describe the criteria for each level of quality; and 6. Create the rubric table.    

  Examples of rubrics are: Learning Outcome Rubrics for Cultivation Techniques for Crops and Vegetables for (1) Students demonstrating information literacy skills, and (2) Teachers organizing information into usable database/textbook; a Research Process Rubric for Middle School; and an Information Literacy Value Rubric.                                         

   

Status observation or assessment:

• Useful for status or condition. A 3rd party professional organization with expertise can provide observation or assessment results, which accurately reflect the status or condition of interest, e.g. reading scores, grades, graduation rates, attendance, employment.
• Good for knowledge
• Weakest for behavior and attitude
• Many programs start as a way to improve some indicator that already exists but is lagging: reading scores, grades, graduation rates, attendance, employment. Using these scores is inexpensive because they already exist, but both the organization that creates them and the individual participants need to cooperate to make them available to the program. Note: To justify that a change in status or condition is induced by your program is difficult. Your program, if successful, may play only a small part in the change in scores.

 

 

 

Data: What makes sense to use?

Your common sense and knowledge of the situation will suggest what data to use. The kinds of data sources explained give the most common types. Remember that Outcomes Based Planning and Evaluation programs do not require formal research; although some programs may wish to get a consultant involved in evaluation. (References to further information can be found by clicking the Resources tab at the top of the screen.)

No kind of data source is better than another. The data source chosen should depend on what is being evaluated.

 

   

Choosing data sources and the people they’re applied to

If your indicators are specific, usually they will suggest the data source to measure it as well as the people in the target audience it should be applied to.

What people or program data will the indicator be applied to? Consider the following issues.

Examine the visuals below to explore issues related to each program. For background on programs, click Cases.

Some or all of the target audience?
Remember that your target audience is usually stated in general characteristics and you may only involve part of that audience in your program as participants. 

West Dakota Library Rx
All WD residents are part of the target audience, but they cannot all be surveyed. A random sample of residents is assumed to be representatives of the whole group.

Some or all of the participants?

Springfield Library Summer Reading Program 
All the participants are students in an urban library district. Should reading tests be monitored for all participants? At-risk students only? Those who complete a specific number of activities?

People other than the participants?

Springfield Library Summer Reading Program
Are parents likely to be better sources of information about their children (such as number of hours spent reading and attitude toward reading) than the children themselves?

Remember: Collecting data costs time and money. Collect only enough information to figure out if your program is successful, so be specific and concrete. Consider the difference in costs between collecting information about “children” and “children who need after-school tutoring.” Or do you mean “children who participate in at least five tutoring sessions? If you expect 100 children to meet this criterion, should you say instead “a random sample of children who participate in at least five tutoring sessions”? 

 

Library example: Riverton Memoirs program indicators

For the following outcomes and indicators for the Memoirs program, try choosing data sources and how they should be applied. 

Outcome 1: Participants show improvement in their writing.

Indicator : # and % of participants who revise five pieces, commenting on what they tried to improve in each revision. 

Applied To : All workshop participants

Data Source : Writers’ portfolios of participants work

Indicator : # and % of participants whose revised pieces (two before-and-after versions) are judged better than the originals in a blind (no writer or dates given) overall grading by a creative writing specialist 

Applied To : All workshop participants

Data Source : Expert evaluation of participants’ work

Indicator : # and % of participants whose revised pieces are judged better than the originals (for two sets) by a creative writing specialist when judged by the writer’s goals in the revision.

Applied To : All workshop participants

Data Source : Expert evaluation of participants’ work

Outcome 2: Participants demonstrate they feel themselves to be part of a community of writers.

Indicator :  # and % of participants who can name three ways they feel a part of the community of writers.

Applied To : Program participants

Data Source : exit survey

Indicator : # and % of participants who act as part of a community of writers after the program (produce writing, continue library group or join another, attend readings, read memoirs, read regularly about authors’ concerns)

Applied To :Program participants

Data Source : Phone interview with checklist of behaviors

 NEXT

---