Data analysis is a process, not a one-off event. Here are 10 steps to help you to use data effectively in inquiry
1. Focus the data Inquiry. Start with a question: for example, how well are students performing in literacy since the introduction of a particular practice? Or, in what ways does the use of technology in the classroom affect students’ research skills? Questions about differences, gaps, characteristics and qualities, the impact of teaching practices and curriculum and so on provide useful starting points. Find a problem in the data that you care about.
2. Hypothesise. Before looking at the data, generate predictions for what you expect to see in the data, and explore the assumptions that underlie these. The aim is to reflect on personal experience and gain an awareness of the assumptions and expectations you bring to the discussion before looking at data, as well as to create curiosity and anticipation. Predictions and assumptions should be developed concurrently – for example, if you make a prediction (e.g. that most students can answer word problems correctly) then try to identify the assumption behind it (we’ve spent a lot of time practising these). In addition, try to reframe your assumptions (“lower-achieving students are likely to struggle with comprehension”) as predictions (“the bottom third of my class will have scored low on the comprehension section of the test”) that are easily tested. And try to generate many different assumptions and predictions, which is easier when working in a group.
3. Collect. Gather information about a student’s, or groups of students’, learning. Search for data that serves your purpose in investigating student learning, rather than accumulating data for its own sake. Don’t wait for data to come to you, go out there and find it or make it! Use as much relevant data as you possibly can – using just a little bit of data is likely to lead to wrong conclusions. Decision making should not be based in student achievement data alone. Student achievement data describes results, and not the causes. To understand the causes it can be useful to use formative assessment data including students’ class work, your own observations of students, and student ‘voice’ (or views) data.
4. Explore. Seek to look deeply at the data from multiple perspectives to generate thoughtful observations about what the data is telling you. Combine and present information to aid a range of different understandings and to illuminate a number of possible relationships between teaching and learning. This involves differentiating, sorting, comparing and contrasting data. You should be asking “What is causing these results?” before jumping to “What action do we need to take?”
There are four kinds of data which you might compare in order to highlight possible answers:
* demographic data: information about the student, staff and school community population
* student achievement data: standard assessments, norm-referenced tests, classroom assessments and student work
* perceptions data: gathered through questionnaires, surveys and observations
* school processes or programmes data: the programmes students have experienced, classroom practices and assessment strategies.
To understand what data is telling you, for example, about student achievement, you could:
- disaggregate data by demographics (for example, break down by gender or ethnicity) to understand the impact of teaching on different groups of students
- look at data over different years to investigate change
- compare data across categories to explore relationships, for example, comparing perceptions data with student achievement data to see if attitudes influence achievement, or compare school processes data with student achievement data to see if a particular programme led to particular gains for students.
While it is important to sort data by a variety of variables, be wary of making data too complex and overwhelming by differentiating every possible sub-group.
Analyse the data by looking for what seems to ‘pop out’ or what is surprising or unexpected. Identify patterns, categories and trends, but avoid jumping to conclusions or arriving at certainty which can lead to badly-framed problems and premature solutions. Don’t seek to explain or determine problems but continue to mine the data for possibilities. Exploring data takes a long time. In fact, researchers can spend months analysing single data sets.
5. Evaluate. Assess the quality of the data – use the 4 C’s of interpreting data:
o Does the data give the complete picture? Ensure you have a range of sources of data, both formal and informal, and not just numerical data. Be careful not to use data as ‘proof’. It is easy to select data as evidence that confirms existing beliefs about students and teaching rather than using data to uncover and explore issues.
o Is the data consistent with other sources of information, particularly teacher observation data or experience? Is the data compatible with what you know about your students from your ongoing classroom assessments? Are the overall results for your students consistent with your expectations? Draw on your experience and professional judgement – don’t automatically trust the data report. If there are inconsistencies, consider what might be the reasons for them or what might have had an impact on results. But also remember that what you think you might be observing in your classroom may not always be the full story.
o How does the data compare with the standard, your targets or other schools? This is not about making judgements, but instead about seeing how comparison helps you in your search for the meaning of data.
o Might the data be concealing something? Have you considered the full set of results? Have you disaggregated the data in order to view the achievement of particular groups of students? Are there relationships between achievement and absence, for example, or between achievement and attitudes?
6. Organise. Create a data overview, a succinct and well-organised summary of data relevant to your question, based on several data sources, in order to provoke thinking and discussion. Find someone to share this with: what do they notice / infer from the data?
7. Speculate. Select key observations and generate inferences and potential explanations and conclusions for those observations. Avoid jumping to conclusions. Speculate what might be the reasons for the patterns in data that you identify. Remember that the correlation between a set of events or behaviours and particular outcomes for students does not mean there is causality. Create a range of data stories, which help you identify where you might need more information about specific problems or issues.
Explanations for patterns in the data usually fall into one of five categories – stretch your thinking (and the number of potential explanations you develop) by considering each of these areas
For example, “I think we are getting these results beause the current instructional materials do not scaffold the fundamental concepts”.
8. Identify gaps in understanding and collect more data. Most sets of data do not tell the whole story, and you want to be reasonably confident of the cause of a given pattern in the data before determining a course of action to address it. Use of data almost always leads to an understanding that the existing data is not good enough, and new data is needed to deepen understanding.
Determine what other data might help to confirm or verify your explanation or theory. This is called triangulation, and is most powerful when varied and diverse sources of data are used. For example, a qualitative survey of student attitudes will enhance quantitative measures such as those provided by standardised tests. In addition to these, teacher anecdotes or a curriculum analysis can deepen understanding. The idea is to seek multiple and diverse sources of information. In this process your questions of data become more specific. For example, you might decide to examine teaching practices further, through a round of peer observation. Engage another data-gathering cycle in order to test your emerging theories about what is going on for students, and to gain confidence in their validity as explanations.
9. Plan. Decide what solutions you might explore for what you have identified as the potential cause of the data story and determine an action plan. Begin with the end in mind. For example, plan backwards from your intended outcomes, to work out the necessary knowledge, skills and dispositions required in students. Then work out the desired actions of teachers and their necessary knowledge, skills and dispositions, as well as structural requirements, etc.
Keep the focus on outcomes rather than activities or processes. Be wary of deciding too quickly on a potential course of action. Set SMART (specific, measurable, attainable, results-oriented and time-bound) goals for achievement, for example, by being clear on intended outcomes, the necessary steps of action, and the criteria for success. Anticipate barriers to the implementation of your plan. Agree on what data you will use to measure the success of your planned actions (and adapt if necessary).
10. Act and Assess. Share the data and action plans with parents, school boards and colleagues in order to generate partnerships. Respect confidentiality.
Use multiple measures to assess whether student achievement is improving.
Tips for working with data
- Look at data with an inquiry mindset of openness and curiosity. Seek a deep understanding of issues.
- Reserve judgement, tolerate ambiguity and consider a range of perspectives.
- Work in a group. Groups can more thoroughly mine the data for possibilities in order to develop collective meaning and shared understanding.
- Have a critical friend with which to debate potential data stories.
- Recognise and put to use many kinds of data.
- Become knowledgeable about statistical and measurement concepts.
- Allow time – making sense of data is not an overnight process.
- Consider teaching students to examine data about their own learning and use it to set learning goals.