What is data?
Data is the systematic organisation of information in the form of words, numbers and observations, that can be used to aid instructional decision making. For example, as a teacher you might have access to student achievement data from standardised tests such as PATs or eAsTTle, as well as from other tests, assignments and pieces of work you have assessed in class. Data can also include your own observations of students during a lesson, student views and feedback.
Data are relatively easy to collect. The challenge is in their interpretation – looking for clues about effectiveness within the data and investigating the story or explanation underlying these facts and figures. To be data-literate is to be able to collect, analyse, question and make sense of data.
Rather than be a ‘data ostrich’ (Ontario Leadership Strategy, 2013-4), find out below how to learn from and use data for inquiry.
Why collect data?
Data, and tools to gain information from data, can support teachers and schools to make changes to teaching and learning that have a significant impact on student achievement. Data can reveal gaps in student achievement that ultimately lead to recognition of gaps in schools’ and teachers’ instructional practices. Data can help you understand how your school functions, how a group of students learn, and the successes and problems that you are facing. This will help you make decisions on how to prioritise teaching time, where to target additional instruction, and how to adapt the curriculum in line with students’ learning strengths and needs. The use of data should be integral to every planning and decision-making process in schools.
How to use data
Methods for using data are not straightforward or transparent. The challenge is that data alone does not answer questions or provide solutions. You might view data as a tool or lens to look through, that you can use to think about and understand your teaching practice and student learning more fully. Interpretation is paramount.
Data is your starting point in a search for meaning. Data tells you where you are and what is happening. There is a story behind the data, which you need to uncover: the story of why particular situations or sets of outcomes are occurring. Therefore working with data involves identifying patterns and relationships, and constructing a story or theory for what is happening in your students’ learning. This story or theory is then tested by examining other sources of data, or devising ways to collect new evidence.
Ten steps for using data in inquiry
Data analysis is a process, not a one-off event.
1. FOCUS THE DATA INQUIRY.
Start with a question: for example, how well are students performing in literacy since the introduction of a particular practice? Or, in what ways does the use of technology in the classroom affect students’ research skills? Or, how effectively do my Year 6 students demonstrate mathematical reasoning? Questions about differences, gaps, characteristics and qualities, the impact of teaching practices and curriculum and so on provide useful starting points. Find a problem in the data that you care about. Engage your interest, motivation and curiosity in order to develop a positive attitude and mindset for working with the data.
Before looking at the data, generate predictions for what you expect to see in the data, and explore the assumptions that underlie these. The aim is to reflect on personal experience and gain an awareness of the assumptions and expectations you bring to the discussion before looking at data, as well as to create curiosity and anticipation. Predictions and assumptions should be developed concurrently – for example, if you make a prediction (e.g. that most students can answer word problems correctly) then try to identify the assumption behind it (we’ve spent a lot of time practising these). In addition, try to reframe your assumptions (“lower-achieving students are likely to struggle with comprehension”) as predictions (“the bottom third of my class will have scored low on the comprehension section of the test”) that are easily tested. And try to generate many different assumptions and predictions, which is easier when working in a group.
Gather information about a student’s, or groups of students’, learning. Search for data that serves your purpose in investigating student learning, rather than accumulating data for its own sake. Don’t wait for data to come to you, go out there and find it or make it! Use as much relevant data as you possibly can – using just a little bit of data is likely to lead to wrong conclusions. Decision making should not be based in student achievement data alone. Student achievement data describes results, and not the causes. To understand the causes it can be useful to use formative assessment data including students’ class work, your own observations of students, and student ‘voice’ (or views) data.
Seek to look deeply at the data from multiple perspectives to generate thoughtful observations about what the data is telling you. Combine and present information to aid a range of different understandings and to illuminate a number of possible relationships between teaching and learning. This involves differentiating, sorting, comparing and contrasting data. You should be asking “What is causing these results?” before jumping to “What action do we need to take?”
There are four kinds of data which you might compare in order to highlight possible answers:
- demographic data: information about the student, staff and school community population
- student achievement data: standard assessments, norm-referenced tests, classroom assessments and student work
- perceptions data: gathered through questionnaires, surveys and observations
- school processes or programmes data: the programmes students have experienced, classroom practices and assessment strategies.
To understand what data is telling you, for example, about student achievement, you could:
- disaggregate data by demographics (for example, break down by gender or ethnicity) to understand the impact of teaching on different groups of students
- look at data over different years to investigate change
- compare data across categories to explore relationships, for example, comparing perceptions data with student achievement data to see if attitudes influence achievement, or compare school processes data with student achievement data to see if a particular programme led to particular gains for students.
While it is important to sort data by a variety of variables, be wary of making data too complex and overwhelming by differentiating every possible sub-group.
Analyse the data by looking for what seems to ‘pop out’ or what is surprising or unexpected. Identify patterns, categories and trends, but avoid jumping to conclusions or arriving at certainty which can lead to badly-framed problems and premature solutions. Don’t seek to explain or determine problems but continue to mine the data for possibilities. Exploring data takes a long time. In fact, researchers can spend months analysing single data sets.
Assess the quality of the data – use the 4 C’s of interpreting data:
o Do the data give the complete picture? Ensure you have a range of sources of data, both formal and informal, and not just numerical data. Be careful not to use data as ‘proof’. It is easy to select data as evidence that confirms existing beliefs about students and teaching rather than using data to uncover and explore issues.
o Are the data consistent with other sources of information, particularly teacher observation data or experience? Are the data compatible with what you know about your students from your ongoing classroom assessments? Are the overall results for your students consistent with your expectations? Draw on your experience and professional judgement – don’t automatically trust the data report. If there are inconsistencies, consider what might be the reasons for them or what might have had an impact on results. But also remember that what you think you might be observing in your classroom may not always be the full story.
o How do the data compare with the standard, your targets or other schools? This is not about making judgements, but instead about seeing how comparison helps you in your search for the meaning of data.
o Might the data be concealing something? Have you considered the full set of results? Have you disaggregated the data in order to view the achievement of particular groups of students? Are there relationships between achievement and absence, for example, or between achievement and attitudes?
Create a data overview, a succinct and well-organised summary of data relevant to your question, based on several data sources, in order to provoke thinking and discussion. Find someone to share this with: what do they notice / infer from the data?
Select key observations and generate inferences and potential explanations and conclusions for those observations. Avoid jumping to conclusions. Speculate what might be the reasons for the patterns in data that you identify. Remember that the correlation between a set of events or behaviours and particular outcomes for students does not mean there is causality. Create a range of data stories, which help you identify where you might need more information about specific problems or issues.
Explanations for patterns in the data usually fall into one of five categories – stretch your thinking (and the number of potential explanations you develop) by considering each of these areas
For example, “I think we are getting these results because the current instructional materials do not scaffold the fundamental concepts”.
8. IDENTIFY GAPS IN UNDERSTANDING AND COLLECT MORE DATA.
Most sets of data do not tell the whole story, and you want to be reasonably confident of the cause of a given pattern in the data before determining a course of action to address it. Use of data almost always leads to an understanding that the existing data is not good enough, and new data is needed to deepen understanding.
Determine what other data might help to confirm or verify your explanation or theory. This is called triangulation, and is most powerful when varied and diverse sources of data are used. For example, a qualitative survey of student attitudes will enhance quantitative measures such as those provided by standardised tests. In addition to these, teacher anecdotes or a curriculum analysis can deepen understanding. The idea is to seek multiple and diverse sources of information. In this process your questions of data become more specific. For example, you might decide to examine teaching practices further, through a round of peer observation. Engage another data-gathering cycle in order to test your emerging theories about what is going on for students, and to gain confidence in their validity as explanations.
Decide what solutions you might explore for what you have identified as the potential cause of the data story and determine an action plan. Begin with the end in mind. For example, plan backwards from your intended outcomes, to work out the necessary knowledge, skills and dispositions required in students. Then work out the desired actions of teachers and their necessary knowledge, skills and dispositions, as well as structural requirements, etc.
Keep the focus on outcomes rather than activities or processes. Be wary of deciding too quickly on a potential course of action. Set SMART (specific, measurable, attainable, results-oriented and time-bound) goals for achievement, for example, by being clear on intended outcomes, the necessary steps of action, and the criteria for success. Anticipate barriers to the implementation of your plan. Agree on what data you will use to measure the success of your planned actions (and adapt if necessary).
10. ACT and ASSESS.
Share the data and action plans with parents, school boards and colleagues in order to generate partnerships. Respect confidentiality.
Use multiple measures to assess whether student achievement is improving.
- Look at data with an inquiry mindset of openness and curiosity. Seek a deep understanding of issues.
- Reserve judgement, tolerate ambiguity and consider a range of perspectives.
- Work in a group. Groups can more thoroughly mine the data for possibilities in order to develop collective meaning and shared understanding.
- Have a critical friend with which to debate potential data stories.
- Recognise and put to use many kinds of data.
- Become knowledgeable about statistical and measurement concepts.
- Allow time – making sense of data is not an overnight process.
- Consider teaching students to examine data about their own learning and use it to set learning goals.
Three common mistakes in using data for inquiry
Are you aware of your cognitive biases when you view data? Do you focus on action over understanding? Does your ego interfere with your ability to inquire?
The skilful use of data to improve teaching and learning is not easy. Being aware of some common mistakes and pitfalls that can hamper your ability to use data to transform your teaching and learning inquiry is very important.
Which of these common tendencies for working with data might affect your data work?
VIEWING DATA ACCORDING TO COGNITIVE BIASES
We all have cognitive biases which influence the way we see and interpret the world, including how we analyse and interpret data. When we engage with data, we rely on habits and short-cuts for meaning-making (called “cognitive biases”) that might blind us to potential meanings in the data that fall outside of these habitual understandings. When viewing data it is critical that we challenge or interpret cognitive biases, otherwise data use is unlikely to lead to changes in either beliefs or practice. The use of data for inquiry can be more powerful if teachers engage in ‘intentional interruption’ of these cognitive biases.
Here are five biases that frequently influence our thinking and decision making. Understanding these biases can help you to become a better user of data.
- Confirmation bias: making data fit your existing view of the situation
We have a natural tendency to transform information to fit what we already believe to be true about the world, rather than using new information to restructure our understandings. This is called the “confirmation bias”, which occurs when we have an existing theory or hypothesis about something, and tend to look for things that confirm existing beliefs rather than challenge them. In fact, we easily avoid evidence to the contrary. For example, we might read an article about teaching and focus on the parts that confirm what we already do and overlook the parts that challenge these practices. We might look at data and pay more attention to that which supports our hypotheses than that which challenges them. This is a cognitive bias that serves to preserve the status quo. It prevents learning, as without awareness of contrary information there is no need to learn and change.
- Recognition bias: valuing the known over the unknown
We tend to place greater value on things we recognise, than the things we do not – this is called “recognition bias”. We might quickly make a decision or come to a conclusion based on something that is easily recognisable, while failing to notice other less familiar messages in the data.
- The competency trap: interpreting current events and data through past experience
We tend to choose solutions based on what has worked in the past. However, these may not always apply to a new situation. This is the ‘competency trap’. It can also occur when we have limited information or limited expertise with an issue, forcing us to look to precedents and past solutions.
- Vividness bias: over-focusing on dramatic data results
When something is striking, or conjures up particularly vivid images, it tends to be over-emphasised in our minds. This is the vividness bias. An example is the way in which we are more worried about being in a plane crash rather than a car accident, even though the latter is more likely. This is due to media coverage of plane accidents being far greater than that of car crashes, making them more vivid and more memorable.
- False correlations: assuming two variables are related when they are not
Sometimes because certain events are more vivid or stand out in our minds, we make illusory correlations between them. An examples is the belief that “it always rains on weekends”, which develops because we tend to focus on all the times when these things have happened, and not on when they did not. We tend to believe there is a relationship because of the vividness of the examples when it has happened. It gets noticed when it happens, but not when it does not.
How to overcome these tendencies towards bias:
- Develop an awareness of cognitive bias; this can help to interrupt it.
- Don’t cherry pick your data but work systematically through all the data.
- Purposefully seek and pay attention to disconfirming evidence.
- Spend time considering all possible directions in the evidence, rather than quickly proceeding with initial thoughts and feelings.
- Be open to what the evidence is suggesting, and to the need for change.
- Work with data in groups where members are comfortable to challenge each other.
FOCUSING ON ACTION RATHER THAN EVIDENCE
Many people relate inquiry to engaging in a course of action, which results in spending limited time gaining an in-depth understanding of the problem in the rush to “do something”. This focus on action can negatively affect the success of the inquiry in making sustainable and effective changes to practice. There are two types of action that need to be guarded against
i. Jumping to solutions
We like to feel that we are accomplishing something, so we often take shortcuts in problem analysis and immediately start thinking about solutions. However, experts often spend a long time mapping out the requirements of a problem before taking action. A lack of time spent understanding the problem leads to a superficial understanding of problems and time wasted pursuing less effective or wrong solutions. Choosing activities before understanding the problem creates an “activity-trap” in which activity, rather than understanding, is the focus.
How to overcome this tendency:
- Follow the steps of an inquiry cycle.
- Use data to unpack and describe the problem under inquiry.
ii. Avoiding action
We can often be afraid to take a risk and change practice in case we make things worse for our students, rather than better. Somehow, it seems as if doing nothing and continuing with usual practice is better than making mistakes. It is common to believe that the harm caused by taking action is worse than the harm caused by taking no action. It is also a mistake to believe that doing nothing is doing nothing – of course you are still doing something, just doing nothing new.
How to overcome this tendency:
- Data can to help us see the consequences of continuing with the usual practice.
- Make comparisons between preferable futures: what you would like to happen, and the probable future, which is what you can expect if you maintain the status quo.
LIMITING INQUIRY IN ORDER TO PROTECT RELATIONSHIPS AND STATUS
For some people, the need to belong to and be esteemed by a group interferes with their capacity to engage in robust inquiry activities, such as asking probing questions, suggesting alternative ideas or offering hypotheses that contradict those of others in the group. However, a focus on preserving group harmony leads to superficial definition and exploration of problems, cognitive bias in reviewing the evidence, and inappropriate and ineffective solutions.
Another danger is to view problem solving as a competition and seek to prove yourself as the group member with the correct hypothesis or best solution. This means that valuable energy is spent advocating for proposals and opinions, and criticising those of others, and not in exploring problems and issues in depth and from a range of perspectives.
How to overcome this tendency:
- Spend time building relationships to support risk taking and honest dialogue in the group.
- Use clear processes to manage group behaviour, ensure equal participation and encourage professional critique.
What kinds of data can I collect and how?
Teaching through inquiry requires a great deal of evidence collecting so that you can evaluate what the needs are for your class, what aspects of teaching and learning are more and less effective, and how changes you are trialling are impacting on student learning. A range of data from varied sources will give you a fuller picture of the teaching and learning in your class, and you can compare the information you get from each kind of evidence to ensure that it is consistent, and therefore measure the reliability of your information. Here we define data broadly, to include all forms of data that relate to educational activity.
Here are some key kinds of data you can collect and how to do it.
You might ask students to talk through their thinking while working through a maths problem, and write down what they say. You might observe a student’s strategies for decoding an unfamiliar word while reading, make a note of the substitutions they use and what these tell you about students’ reading skills (phonetic, syntactical and semantic awareness). You might observe students’ engagement during particular lessons by measuring time spent on task, or find out which self-choice activities are most popular by mapping student position at given intervals. You might make a record of the feedback that you give during a lesson and to which students, by making notes or videoing yourself.
You can collect students’ perceptions and summaries of lessons on index cards or post-its as they leave the classroom. Pose a question or ask students to list three things they most remember about the lesson. Do these match to your learning intentions?
You might explore the numbers and types of books that students are selecting, or review student portfolios to see the number of reflections added. These findings can be represented numerically making them easier to compare. However, this might miss out understanding why books are chosen or how students feel about writing reflections. You can also access your school’s numerical information about students’ attendance, for example.
Results of tests
You can use the results of standardised tests (e.g. e-AsTTle, PAT, STAR Tests) as evidence for your inquiry into the effectiveness of learning, or of tests that you administer in class yourself. For example, you can record not only the total score of each student in a test, but which questions were generally answered correctly by a given group of students, and which students struggled with. Are marks consistently lost in a particular area of the test, for example? Are students confident in a mathematical operation in number problems but not in word problems?
Samples of work
You might gather a range of students’ work and use the New Zealand standards information to grade achievement. In addition to using these standards as a measure of students’ current understanding, you can also see whether there are particular aspects of each standard that are missing across the range of students’ work.
Surveys and scaled surveys
You might survey students, parents or other teachers. A scaled survey asks respondents to quantify their answers, so the information gathered can be easily organised in numerical form, for example, in a graph.
Scaled survey question example
On a scale of 1 – 5 (1 being low and 5 being high), how would you rate this unit of work?
The use of scales limits the scope of the survey however, and there is no way to ensure that different respondents mean the same thing when they grade the question with a 4, or what it was about the unit of work made them grade it so. Unscaled surveys allow personal responses to open-ended questions, but are more complex to organise for analysis.
Unscaled survey question example
In what ways did this unit of work meet your expectations?
It can be a good idea to pilot your planned survey to see if your questions are easily understood and supply the information you are seeking.
Interviews (students, parents, other teachers)
Interviews might be useful for investigating perceptions, beliefs, attitudes and values, or for exploring complex issues that do not have a finite set of possible responses. Interviews might be informal and conversational, which means they flow through spontaneous questions, or they may be guided by a set of questions or points to discuss. Collecting data through interview can be time-consuming, and the data generated may be difficult to compare and analyse. Also remember that interviews give a great deal of useful information about perceptions, feelings and understandings, but self-reports are limited in accuracy.