National Centre for Research Methods: NCRM EPrints Repository

Browne, William

What are Statistical E-Books?

Park, Alison

Intro to session

Olsen, Wendy

QCA and Fuzzy Set Goodness-of-Fit Tests

Topouzi, Marina

Measuring health and wellbeing using MET scores and wearable camera information

Morey, Richard

The mismatch between statistical thinking and statistical practice

Nind, Melanie

Lewthwaite, Sarah

 
 
Introduction: This paper is the result of a journey in which Sarah and I have been involved in both 
disability studies, inclusion and accessibility research, and also pedagogy and methodology 
research. As researchers involved in the NCRM study of the pedagogy of methodological learning 
we are well aware that the research council or NCRM strategy to deliver advanced social science 
research methods courses may not be enough to achieve the desired outcome of building 
methodological capacity. Educationalists, we argue, can make a critical contribution by focusing 
attention on the pedagogy of methodological learning. This involves understanding that 
pedagogy in its social, cultural, historical and political context. All this led us to the question for 
this paper: How are methods teachers positioning learners and to what extent do methods 
teachers assume a stance of pedagogy for everyone – reaching out to all their learners. 
 
We begin by sharing our understanding of pedagogy as not just teaching, or even teaching and 
learning; we use Alexander’s 2000,  540) definition that pedagogy ‘incorporates theories, beliefs, 
policies and controversies that inform and shape it’. Thus, we are not just interested in the 
traditionally more explicit dimensions of pedagogy such as the specified curriculum, the learning 
outcomes sought, the assessment approaches;  we are interested in the tacit, the often hidden 
from view, the silences, the absences and the usually invisible aspects of pedagogy (Nind, Curtin 
& Hall 2016). Like Bruner (1996, 63) we do not see pedagogy as ‘innocent’, but as carrying its own 
messages, and to understand the messages carried in research methods pedagogy we have been 
looking at:  
• pedagogy as specified 
• pedagogy as enacted 
• pedagogy as experienced 
• pedagogy as hard to know 
• multiple pedagogies 
 
Design: This has involved a multi-component design over a study spanning 4 years in total. The 
study is explained in more detail at  
https://pedagogy.ncrm.ac.uk 
 
Today we cannot address the study in its totality – nor would you be interested in it! The 
concerns within this SIG and for our common interest in inclusive pedagogy are: 
• What do methods teachers know about addressing barriers to learning?  
• What inclusive practices have they developed and adopted?  
• Is it helpful to bring in theories of inclusive pedagogy when theorising methods pedagogy? 
 
Thus, we are looking at the values, that ‘spill out untidily at every point in the analysis of 
pedagogy’ (Alexander 2009, 18). One example might be the perceived fear of methods (especially 
statistics) among methods learners, which has led to a deficit discourse in which ill-prepared, 
fearful learners are blamed for making them difficult to teach. An inclusive pedagogical lens in 
this example is helpful ‘for shifting from deficit to asset perspectives’ (Hattam et al. 2009, 306).  
 
 
 
 
 
 
 
Through our various research methods in the project stages we have been looking at the doing, 
knowing and believing of methods teachers. This follows the observation from Rouse (2008) that 
in the context of inclusive classroom practices, there is a close  interrelationship between what 
teachers do, know and believe. Thus, the way in which learners and pedagogy are positioned 
depends not just on our values and ways of knowing, but the cultures we have been immersed 
within. In the arena of research methods education, unlike traditional schooling, there is no 
history of a special needs culture, rather one of developing and valuing criticality. 
 
Data and analysis: The data used in this paper come from our interviews with national and 
international experts, or pedagogic leaders in the teaching of research methods, and from out 
dialogue with methods teachers and learners about the issues arising from both the expert panel 
and from their own practice and experience. We have pre-coded the transcripts/data to highlight 
the interesting bits, coded the data independently by hand and using CAQDAS; we are ultimately 
working towards shared interpretations but allowing contrasts in the data and in our 
interpretations to provoke fresh insights. We have focused on identifying pedagogic themes 
including pedagogic challenges and cultural practices and used our own values base to apply the 
lens of inclusive pedagogy to the data. In our analysis shared here in the table below, we focus on 
three strands arising from applying this lens: barriers to learning and workarounds; holistic 
approaches to connecting learners and learning; and the role of standpoints. In particular we 
share some data on the first of these, the barriers to learning or inclusion, being a very familiar 
concept in inclusive education. The challenges relate to (i) the learner, (ii) the content, and (iii) 
the teachers’ responses. 
 
Holistic approaches are similarly located in terms of ‘what teachers know as a result of their 
experience as teachers’ (Fenstermacher 1994) – knowledge that is practical, personal, situated, 
local, relational, and somewhat tacit. But, compared with workarounds these approaches are 
more wholesale, joining up years’ worth of workarounds, aided by a commitment to a pedagogic 
principle or passion, to develop an approach that becomes coherent or holistic. An example from 
Chris Wild, developing quantitative methods pedagogy in New Zealand is managing the cognitive 
load of learners. He talks about: ‘making it easier for people to see things and then being able to 
see data-related things quicker’. This has led him into making software to show ‘what happens at 
each stage of the process and the relationships between them’. Paul Vogt, teaching quantitative 
methods in the USA described a verbal rather than visual approach: i.e. ‘verbal ways of teaching 
research methods, so teaching regression analysis in terms of the questions that people doing 
regression analysis would be trying to answer, and teaching that in a class or two, and then 
having people read real research output and teaching them how to read it’. In each case the 
emphasis is very much on a coherent approach to the overall challenges. 
Finally, and also a holistic approach in a way, is the way in which some methods teachers are 
doing their reaching out to all learners by foregrounding standpoints in the methods teaching and 
learning. This might involve finding out the learners’ standpoints, connecting the learners’ and 
teacher’s standpoint, and critiquing Western-centric or other dominant standpoints. Sharlene 
Hesse-Biber, a leader in teaching mixed methods, for instance, spoke of ‘the importance of 
stressing reflexivity; knowing not only your own standpoint as a researcher but where your 
students can understand their own standpoint … They take themselves as an object of reflection’. 
Similarly, Bagele Chilisa, teaching research methods in Botswana, talked of ‘recognising that 
methodologies are not neutral and that every methodology that one takes assumes a certain 
standpoint. … we therefore have to look closely at how the research was carried out, the 
methodologies that were used, and the standpoint that was used in doing the research. … 
students or groups taking standpoints and debating their standpoints and trying to convince 
others, who are also coming from different standpoints, that the first standpoint is appropriate, 
it’s worthwhile...’ (Bagele Chilisa). 
Discussion: While methods teachers are hugely challenged by the groups and material that they 
teach, we found a strong desire to empower (would-be) researchers to conduct ethical, 
worthwhile research. The response to this did not show any evidence of bell-curve thinking or 
simple differentiation. Mostly methods teachers were using their own agency to build craft 
knowledge and pedagogic leaders were critically reflexive in developing their strategic approach. 
Working through data provides a bridge for connecting people, evidence and ideas – a particular 
kind of engaged pedagogy (hooks 1994) or connective pedagogy (Corbett (2001). 
Conclusion: Many aspects of enacted or experienced pedagogy are hidden and hard to know; 
therefore it is unsurprising that the discourse, culture and evidence base around both inclusive 
pedagogy and research methods pedagogy are under-developed. Neither inclusive pedagogy nor 
methods pedagogy are particularly well-understood. Both are being generated on the ground and 
through enhancing pedagogic dialogue, we are beginning to get to know the pedagogies a little 
better. 
References: 
Alexander, R. 2000. Culture and Pedagogy: International Comparisons in Primary Education, 
Oxford: Blackwell. 
Alexander, R. 2009. Pedagogy, Culture and the Power of Comparison, in H. Daniels, H. Lauder 
and J. Porter (eds), Educational Theories, Cultures and Learning: A Critical Perspective , 10 – 
26, Abingdon: Routledge. 
Bruner, J. 1996. The Culture of Education , Cambridge, MA: Harvard University Press.  
Corbett, J. (2001) Supporting inclusive Education: A Connective Pedagogy, London: 
RoutledgeFalmer. 
Fenstermacher, G. D. (1994). Chapter 1: The Knower and the Known: The Nature of 
Knowledge in Research on Teaching. Review of Research in Education 20(1), 3-56. 
Hattam, R., Brennan, M., Zipin, L. & Comber, B. (2009) Researching for social justice, 
Discourse: Studies in the Cultural Politics of Education, 30: 303-16. 
hooks, b. (1994), Teaching to Transgress: Education as the Practice of Freedom, New York: 
Routledge. 
Lewthwaite, S. & Nind, M. Teaching research methods in the social sciences: Expert 
perspectives on pedagogy and practice.  British Journal of Educational Studies 
Nind, M., Curtin, A. & Hall, K. in press, Research Methods for Pedagogy. London: Bloomsbury. 
Rouse, M. (2008) Developing inclusive practice: a role for teachers and teacher education? 
Education in the North, 16, 6–11. 
 
 
 
 
 
 
 
  
Pedagogic challenge 
Surrounding the learner 
Workarounds  for challenges  
Surrounding the learner 
Diversity of the learner group  
‘you’re going get a real range of skills’ (teacher, narrative methods, 
focus group) 
‘You’ve got some people who’ve got mathematical background and 
then all the way through to some people who might have stopped 
doing maths at sixteen’ (teacher, quantitative methods) 
Learner unknowns and making decisions about how to pitch the 
teaching/training 
‘different people come in with different levels of ability or prior 
knowledge or experience [and] in fact in the case of today, interests, in 
terms of research questions or specific problems’ (teacher, 
quantitative methods, video stimulated recall) 
Learner deficits  
‘There are deficits or big gaps in the kind of skills set that particularly 
social science undergraduates emerge from higher education with … 
Those gaps are all around the use of numbers in different contexts’ 
(MacInnes, pedagogic leader, quantitative methods, UK)  
 
Using more experienced learners as a pedagogic resource in co-teaching 
 ‘I found that useful … it’s good to have, because it’s another perspective in the room, 
somebody who’s used it.’ (learner on peer as co-teacher, video stimulated recall) 
Continually finding out about and from learners, monitoring and making adjustments 
‘So trying to provide a general idea of what’s important, and capture some of the very 
specific issues and questions that people have, and then try and keep an eye on whether 
people have lost the flow or have not and are still keeping up, I think that’s the challenge 
with it. It’s trying to get that sort of balanced view’ (teacher, quantitative methods, video 
stimulated recall) 
Learner introductions, Group work and dialogue 
‘I try to find themes that may cut across as many different audiences as possible.’ 
(Saldana, pedagogic leader, qualitative methods, USA)   
‘So on the first day we asked people to sort of, just to give us a little bit of a brief intro 
about themselves, … keep asking any questions about it and looking at people [to see] if 
people look confused or sad.’ (teacher, systematic review) 
Stimulate the statistical imagination 
‘anything that gets students interested in data’ (MacInnes, pedagogic leader, quantitative 
methods, UK)  
‘first try to get people to a realisation that data can tell you something interesting’ (Wild, 
pedagogic leader, quantitative methods, New Zealand) 
  
Pedagogic challenge 
Surrounding the content 
Workarounds  for challenges  
Surrounding the content 
Teaching with, through and about data 
 
‘you try to ground what you’re doing in what they’re doing, so that 
they’re thinking it through in relation to their own work, rather than 
just in the abstract, but that’s quite challenging to do.’ ‘sometimes the 
issues about data aren’t contained within one short transcript’; ‘they 
often bring massively inappropriate amounts’ of data’; ‘sometimes 
there are ethical issues in what they bring’ (teacher, qualitative 
methods, focus group) 
‘practically it [using students’ own data] would be a complete 
nightmare. Because most data needs a bit of organising to get it into 
the right shape for use with whatever packages you’re using, and 
there’s always little niggles when you go from one computing 
environment to another, and to expect one or a couple of instructors 
to cope with all that in a class of ten or twenty people I think is a bit 
Utopia’ (MacInnes, pedagogic leader, quantitative methods, UK) 
 
See participants’ data in advance 
‘I’ve invited people to bring their own, but to let me know if they want to do that. … 
Because it requires a discussion in advance, about what they’re bringing, and also 
whether it’s something that other people can look at.’ (teacher, qualitative methods, 
focus group) 
Whole learner group to work with the most suitable data 
‘what works best, actually despite what I said about wanting them to relate to their own 
work, is where you bring data and give data, because you can choose the data and you 
can choose what kinds of challenges and messages there are in that, and so you could do 
something like that and then get them to have a session where they somehow or other 
relate that to their own work, maybe in groups they talk about how they might, … 
similarities and differences with their own data or something.’ (teacher, qualitative 
methods, focus group) 
‘existing datasets … sometimes those datasets are not always realistic, they’re not 
interesting very often. … Now there’s some wonderful datasets … that they could get 
their teeth into. So actually getting them to work practically on those datasets, initially 
just doing data exploration, so just you know looking at how the datasets are constructed, 
just running frequencies, cross-tabs and then gradually building up (Williams, pedagogic 
leader, quantitative methods, UK) 
Highly resourced data workshops 
 
 
  
Pedagogic challenge 
Surrounding the teacher 
Workarounds  for challenges  
Surrounding the teacher 
Knowing where to start. Having methodological knowledge and 
needing to communicate in ways that learners can understand and 
use it. 
Embedding the method in a familiar context, being authentic 
‘people will always, and it’s a good thing that they want to bring their own research to 
bear in their teaching.  I mean that is the last thing you’d want to lose, because that’s 
what makes, often makes the teaching very valuable’ (Brannen, pedagogic leader & 
mixed methods specialist, UK) 
‘my role would be to describe in as you know kind of vivid a detail as I can, some of my 
own ethnographic for example experiences’ (Leonard, pedagogic leader, UK) 
Because I’m a theatre and drama educator, my profession demands that we be on our 
feet for studio work, and so again I transfer that same pedagogical practice into my 
research methods classroom.  (Saldana, pedagogic leader, qualitative methods, USA) 
Finding pedagogic hooks 
‘nothing works better than hands-on work on something they’re interested in’ (Vogt, 
pedagogic leader, quantitative methods, USA) 
‘start from where people are’ using ‘methods of everyday life’ (Coffey). 
 
 
 


Inclusive pedagogy meets research methods pedagogy - Reaching out to learners

Mr Mathew Quaife makes a presentation on  economics and choices.He emphasizes that choices must be made amidst scarce resource

Quaife, Matthew

What are discrete choice experiments?

Collins, Debbie

 
 
Teaching Research Methods – Building research to develop expertise 
Workpackage 5 Pedagogy of Methodological Learning Event 30 Nov 2016, RIBA, London 
Workshop 4: Using digital technology to enhance the teaching of research methods 
 
Resources 
Koehler, M. J., & Mishra, P. (2009). What is technological pedagogical content knowledge? 
Contemporary Issues in Technology and Teacher Education, 9(1). Retrieved from 
http://www.citejournal.org/volume-9/issue-1-09/general/what-is-technological-
pedagogicalcontent-knowledge 
(Useful theoretical text for thinking about what you need to know to make use of digital technology 
in teaching). 
 
Silver, C., Woolf, N.H. (2015) From guided instruction to facilitation of learning: the development of 
Five-level QDA as a CAQDAS pedagogy that explicates the practices of expert users. International 
Journal of Social Research Methodology. 18 (5): 527-543 
http://dx.doi.org/10.1080/13645579.2015.1062626 (Useful example of technological pedagogical 
content knowledge in action in the development of a teaching approach) 
 
Manifesto for Teaching Online from Digital Education at the University of Edinburgh  
https://onlineteachingmanifesto.wordpress.com/the-text/ (An example of what a digital pedagogy 
looks like) 
 
In this session we are going to look using digital technology to enhance the teaching of social 
research methods. 
By digital technology I mean the use of computer technology (devices & software applications), the 
use and or storing of data & information, the semantic web, managed learning systems and social-
media systems. 
The workshop involves three activities: 
 Looking at what the literature and findings from the NCRM Pedagogy of methodological 
learning research tells us about the use of digital technology and how it is being used 
 Reflecting on our own experiences of using digital technology in our teaching of research 
methods, how we made use of it (or might do so), and for what purposes 
 Thinking about what resources would be useful to support research methods teachers in 
discovering and exploiting the affordances of digital technologies in their teaching 
What might we learn from the research evidence? 
The literature on the role that digital technology in the teaching of research methods is limited. The 
following is based on my review of the literature, published since 2005. 
© Debbie Collins, Sarah Lewthwaite & Melanie Nind - Using digital technology to enhance the teaching of 
research methods 
 
2 
 
 
What digital technologies are being used in the teaching of advanced social research methods? 
Generic e-learning tools Social research methods-specific 
Virtual Learning Environments (VLEs) e.g. 
Blackboard, Moodle 
Data collection software (e.g. online 
questionnaire tools) 
Assessment software e.g. TurnItIn Data analysis software (e.g. NVivo, R) 
Discussion forums Data visualisation software 
Collaboration tools (e.g. WiKi’s) Online data sets and resources 
Social Media   
 
How digital technology is being used to support pedagogical goals 
Examples of how digital technology is being used to support pedagogical goals1 are found in the 
literature, and are summarised below. 
Pedagogic goals Use of digital technology 
Active Learning - making the research process 
visible by actively engaging students in aspects 
of the research process and highlighting the 
connections between theory and practice  
Teachers and students use actual survey data to 
explore topical issues and make connections 
between theory and methodology  
Teacher uses an interactive white board to 
demonstrate  quantitative analysis technique, 
then students and teacher work through an 
example together (students using analysis 
software on a computer) before the student 
does one on their own  
Learning through doing - facilitating learning 
through the experience of conducting research  
Students undertake their own research project 
(individually and or as a group) using digital 
research tools, such as writing their own web 
questionnaire using software like 
SurveyMonkey, analysing qualitative data using 
a software package such at Nvivo, or creating a 
Wiki to support the writing up of a group 
research project.  
Software may also be used so that students can 
experiment, e.g. finding out what happens if 
they change the parameters of a statistical 
model. Students may develop their research 
proposal using online peer review software, 
where students gain experience in reviewing 
their peers’ research proposals  
Critical reflection - encouraging critical 
reflection on research practice  
Online peer review or online collaboration tools 
are used, with students reviewing their own 
research proposal in light of comments 
received from their peers and tutor   
Students critique the data they collected as part 
of a group research project  
                                                          
1 Earlier work by (Kilburn et al. 2014) identified three overlapping pedagogical goals expressed by teachers of 
advanced social research methods: active learning; learning by doing and critical reflection.  
© Debbie Collins, Sarah Lewthwaite & Melanie Nind - Using digital technology to enhance the teaching of 
research methods 
 
3 
 
 
Examples of how digital technology is supporting and developing pedagogic goals 
In the literature I found some examples of how digital technology is being used to support 
pedagogical goals. These examples are summarised below. 
Ways in which digital technology is 
supporting/ developing ASRM pedagogic 
goals 
Activities & approaches 
Builds students and teacher confidence 
 
• developing students understanding of data 
analysis software architecture & skills in 
using its features 
• creating web interfaces that allow students 
to explore survey data with minimal 
guidance 
• using interactive/collaborative tools to 
create  collective, safe learning spaces  
 
Helps students achieve their learning 
outcomes 
 
• practising ASRM skills & exam technique 
through use of online quizzes  
• receiving regular, fast feedback 
• identifying problematic threshold concepts 
 
Facilitates communication • between students, students & teachers, 
students and the outside world  
Assists with breaking down barriers to 
learning 
 
• facilitates access to learning resources 
• can learn at a place and time of student’s 
choosing 
 
 
 
Consider: how complete a picture does the literature paint of the uses of digital technologies in 
the teaching of social research methods? 
What we have learned from our Pedagogy of Methodological Learning research so far 
In our research we have spoken with research methods teachers who are using digital technologies 
in their teaching, as the medium through which they teach (online) and as part of their teaching. 
Some additional digital technologies were mentioned, that the literature review had not identified: 
creating and making use of video; interactive online documents; and software development 
practices such as hackathons and sprints. 
Our research is highlighting a range of different approaches, strategies and tactics to teaching 
research methods in online environments 
The technology can be seen as liberating. 
 “I think working online means you don’t have all the baggage of … ‘now we have to write 
the lecture, that’s going to take an hour and it’s going to happen on a Wednesday 
afternoon’.  You can dispense with all the expectations around how you deliver Research 
© Debbie Collins, Sarah Lewthwaite & Melanie Nind - Using digital technology to enhance the teaching of 
research methods 
 
4 
 
Methods, and think about, well, what would be a creative way to think about this issue 
around transcription and what it means to do that, for example.”  
It can introduce new ways of thinking, for example, what Richard Rogers calls the ‘digital turn’ which 
involves social researchers making use of approaches such as ‘hackathons, skill share workshops and 
sprints, these sort of traditions’ to develop digital research skills and methods.  
However, it can also bring challenges. The institutional infrastructure may not be set up to support 
online courses or teaching (for example, one of our participant’s talked about how library books only 
existed in physical form rather than as e-books, reflecting the university’s preconception that 
students would visit the library in person to borrow resources. This was not practical for the 
students of this online Masters course, who were spread across the globe). Some other challenges 
mentioned by our research participants are presented below, along with the way(s) in which they 
have attempted to tackle them. 
Challenge How have teachers/ course leaders attempted to tackle it 
How to engage students in 
research methods (in an 
online setting) 
“In our course, we don’t have any lecturers at all.  Each week is 
structured around a topic, or topics, and students do some core 
readings and some additional readings.  We have a discussion 
area, where tutors and students talk together about the issues 
that came up in their readings.  And then we have a set of 
activities…to get a real feel for what some of the stuff involves 
and what it feels like to do it…So for example in the first week of 
the course, we have about six or seven short interviews that I did 
with people doing digital research, in a bunch of different 
contexts, and we asked students to transcribe them…”,   
Getting to know your students 
in an online setting (to 
support a student-centred 
approach) 
“I make assumptions obviously to start with, about where, what 
their [student] background is and so on…  But most of all I guess 
it comes from dialogue with them. In talking to them you 
suddenly realise they’ve got that point, move on to the next one. 
Or that issue was easy, what about something a bit more 
challenging? That kind of thing. So it’s very much something that 
gets adapted and adopted as I go through the course … within 
the constraints of the topic for that week or that session. 
Adopting an experiential 
learning approach in a MOOC 
environment 
“I make them do an observation…, and I tell them to go out and 
observe people waiting, and observe their behaviour and 
describe it, very detailed.  Then they have to post that online, or 
they have to hand it in, and other people have to peer review it. 
…  I make the others think about it …later on, they have to hand 
in their reworked assignment again, and then the others have to 
use the peer review to code it.  So I ask them to code… read 
what’s on the observations, and … make them write a little piece 
about it.  And that was really, really, really hard at the beginning 
of this MOOC: to make people understand that this was not just 
peer reviewing, this was using a peer review because we didn’t 
have a coding tool.”   
Teaching students how to use 
software online settings 
“I’ve taught NVivo for many years now in lab type situations, and 
one-to-one as well, and I’m quite used to the fact that students 
get stuck, they do things you couldn’t imagine they could do … or 
they don’t understand some kind of concept that they’re 
struggling with, that you want to try to explain to them.  And 
© Debbie Collins, Sarah Lewthwaite & Melanie Nind - Using digital technology to enhance the teaching of 
research methods 
 
5 
 
being there alongside them, over their shoulder or whatever, it’s 
a really easy way to help them.  It’s very hard to do that online.  
I’ve tried it.  We use Adobe Connect as our, within our software, 
and I have run NVivo through it.  It’s not wonderful, there are 
problems … but at least you can do something [...] You have to 
watch the ChatBox - there’ll be something in a chat or the hand 
will go up,  in  Adobe Connect you can do that as well. So that’s 
how you know when someone’s got a question or are stuck or 
whatever…When I have done it, it’s been just a few people… 
one-to-one, or in one case it was three people online.  It worked 
just about okay.  But doing the twelve, yeah that’s, I’ve never 
gotten that far yet.”  
Learning to talk to students 
through the lens of a video 
camera 
“And when I started doing these [video] clips… I had to write the 
scripts, and I’m not the best writer in the world, especially not 
English … so it took me a lot of time to write these scripts and 
then these were written scripts rather than spoken language.  So 
at one point I simply ditched them and I told the story to a 
camera, and then it turned out that that was working much 
better, much more informal and I acted as if there was someone 
really small inside that camera…, who was my friend, and that I 
was explaining stuff to, and that worked.”   
How to use video in teaching “…so there’s activities… focusing on doing things, focusing on 
reflecting at times, and there’s content, which is often the video, 
but then ways of getting students to interact with the content, 
interact with the doing bits and interacting with, you know, us 
online in ways as well…”  
 
Do these challenges sound familiar? What other challenges are there, particularly when using digital 
technology in a blended learning environment? 
Your own experiences 
What are your experiences of using digital technology in your teaching of research methods?  
What motivates you to (want to) use it? 
What are the challenges you have faced (or think you will face) in using it? 
How have (might) you use it in your teaching? What does (could it) help you achieve? 
Resources 
What pedagogical resources are you aware of / do you use when thinking about making use of 
digital technologies in your teaching of research methods? 
What (additional) resources would be helpful to you on making use of digital technology in your 
teaching? 
 
 


Workshop 4: Using digital technology to enhance the teaching of research methods

The Qualitative Expertise at Southampton (QUEST) seminar titled 'One post said...': Curating comments as data in digital contexts. www.quest.soton.ac.u

Ashton, Daniel

YouTube comments and contesting participatory promises

Durrant, G

Moore, Jamie

Correa, Solange

Smith, Peter

 
Assessing risk of nonresponse bias and dataset 
representativeness during survey data collection 
Gabriele Durrant 
 
Joint work with Jamie Moore, Solange Correa and Peter W.F. Smith 
University of Southampton  
 
RSS Social Statistics Section, 3 March 2016 
  
The Research Project  
 
 • Was funded by  
– ESRC research grant on “Paradata” 
– ESRC National Centre for Research Methods, 
Workpackage 1 “Data Collection for Data Quality”  
– ESRC Administrative Research Centre for England 
(ADRCE).  
 
2 
Introduction   
 
 
• Focus has shifted from nonresponse rate to 
nonresponse bias 
 
• Key question: How to monitor, assess and minimise 
(risk of) nonresponse bias? 
– Post or during data collection  
 
• Questions from survey practice: when to stop calling? 
 
3 
Introduction   
 
 
• Fully observed information on both respondents and 
nonrespondents necessary  
• Sample frame information from  
– register / Census  
– administrative data 
– previous wave 
 
• Datasets (face-to-face surveys): 
– ONS Census nonresponse link study 
– Understanding Society  
4 
How to assess the risk of nonresponse bias? 
 
 
• Main idea: measure similarity between sample data 
obtained and frame data in terms of variation in response 
rates 
• Use of a response propensity model to obtain estimated 
response propensities 
• Representativeness indicators: estimate variation in these 
response propensities (SD = Standard deviation of the 
response propensities) 
• Low variability in response propensities imply high 
representativeness  
 
5 
Representativeness Indicators 
 
 
• R indicator:                 𝑅 = 1 − 2𝑆𝐷 
 SD= standard deviation of response propensities   
 Ranges between 0 and 1  
 Close to 1 indicates high representativeness 
• CV (Coefficient of Variation):  𝐶𝑉 =
𝑆𝐷
𝑟
 
 r = response rate 
 CV close to 0 indicates high representativeness 
• Here computed at each call (visit to a household by 
interviewer) 
6 
Applying these Methods –  
Key Research Objectives 
1.  Visualise trends in dataset representativeness 
2. Are trends in representativeness generalizable 
across surveys (of the same population)? 
3. Can we derive stopping points for an adaptive data 
collection strategy – can these be generalised?    
 
 
7 
Data 
Data 
• ONS 2011 Census Non-Response Link Study (CNRLS) 
• Links response indicator from three UK social surveys to 
survey call record data  and census household (HH) 
information on sample frames 
• 3 (cross-sectional) face-to-face surveys: 
 - Labour Force Survey (LFS) (wave 1) 
 - Life Opportunities Survey (LOS) (wave 1) 
 - Opinions Survey (OPN) 
• Up to 20 calls to a household 
9 
Application and Results 
R indicators 
 
 
 
 
 
 
 
 
 
 
 
 
       final response rate:  LFS = 65.7% 
             LOS = 70.1% 
              OPN= 64%. 
 
R indicators 
 
 
 
 
 
 
 
 
 
 
 
 
• In case of low response rates (as is the case early on in data 
collection) small response propensity variation, limited potential for 
response propensity divergence  
• R indicators close to 1, falsely indicating high representativeness  
• R-indicator can be misleading in this case  
 
CV (Coefficient of Variation) 
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20
r 
/ 
O
v
e
ra
ll
 C
V
Call number
LFS r LOS r OPN r LFS CV LOS CV OPN CV
.
.
.
.
.
.
.
.
.
r 
/ 
O
v
e
ra
ll
 C
V
      
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
• CV standardises SD by r; overcomes the problem of the R indicator 
• CV decreasing, close to 0 indicating high representativeness  
 
(Unconditional) Partial Indicators 
 
• Aim: estimate the extent to which response is 
representative with respect to a covariate or a particular 
category 
• We found similarities across surveys, some variables 
improve across calls, some remain the same (but do not 
improve) 
 
Phase Capacity or Stopping Points 
Stopping or Phase Capacity Points 
• When to change a survey data collection method?  
     (Phase capacity point) 
 
• When to stop calling? 
     (Stopping point) 
16 
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20
r 
/ 
O
v
e
ra
ll
 C
V
Call number
LFS r LOS r OPN r LFS CV LOS CV OPN CV
Stopping or Phase Capacity Points  
• Adaptive Strategy: stop when indicator within 0.02 of 
minimum value  (points later when threshold decreased) 
• Responsive strategy: stop when indicator within 0.02 of 
previous value 
 
Stopping or Phase Capacity (PC) Points 
• Overall: 
 
 
 
 
 
 
• Also possible by variable 
 
 
 
 
 
Survey PC point  
(adaptive) 
% calls 
saved 
PC point 
(responsive) 
% calls 
saved 
LFS 6 8% 5 12% 
LOS 8 15% 7 18% 
OPN 6 13% 6 13% 
18 
Further Evidence from 
Understanding Society 
20 
 
Understanding Society Data 
• Longitudinal study 
• Assess (risk of) nonresponse bias at each call for wave 2 
for a range of survey variables as measured at wave 1 
 
 
21 
 
Further Data Quality Indicators 
• Proposed approach 
– Dissimilarity indices (e.g. Delta index) 
– Basic idea: compare two distributions (those for 
respondents and those if everyone had responded)  
• Comparison to 
– Coefficient of Variation (CV)  
 
 
 
 
 
 
22 
 
Dissimilarity Index: Categorical  
• Delta index 
∆𝑧=  π 𝑧,𝑘 − π𝑧,𝑘 /2
𝐾
𝑘=1
 
 𝜋 z,𝑘  observed proportion in category k of survey variable z 
 πz,𝑘  corresponding expected  proportion 
- ranges from 0 to 1 
- the higher the delta index the more dissimilar is the estimated 
distribution to the true distribution 
- values below 0.03 may indicate similarity (negligible nonresponse bias) 
- no model required 
 
23 
Delta Index 
Binary and Categorical Variables 
24 
Response Rate, R-indicator and CV  
Summary 
• Representativeness increases similarly in the surveys  over call 
records  
– Sources of non-representativeness are under-representation of 
economically active HHs, HHs located in London / SE, and single adult HHs   
• CV preferred over the R-indicator  
• Data collection stopping points differ (slightly) between surveys  
• Dissimilarity index:  
– Can monitor categorical variables with several categories 
– Allows monitoring of several variables in the same graph 
– Does not require the fit of a model at every call 
• Results for CV very similar to Dissimilarity Indices – reassuring 
 
 
25 
Implications for Survey Practice  
• Number of calls could be reduced (no more than 8 calls)  
• Implications for cost savings without potentially much 
loss of data quality 
 
26 
 
Thank you. 
 
g.durrant@southampton.ac.uk 
27 
Acknowledgements 
 
This work contains statistical data from ONS which is 
Crown Copyright. The use of the ONS statistical data in 
this work does not imply the endorsement of the ONS 
in relation to the interpretation or analysis of the 
statistical data. This work uses research datasets which 
may not exactly reproduce National Statistics 
aggregates.   
 
28 


Assessing risk of nonresponse bias and dataset representativeness during survey data collection

National Centre for Research Methods

Not a member yet