National Centre for Research Methods: NCRM EPrints Repository

Browne, William

Using StatJR’s Statistical Analysis assistant to assist in automating statistical analysis

Lowe, Sarah

The Value of Safe Settings in Evidence-Based Policy Making

Hodges, Duncan

Digital Identity

Janz, Nicole

ESRC Research Methods Festival
5 July 2018 | University of Bath 
Research Reproducibility 
in Political Science
Dr Nicole Janz
School of Politics and IR
Replication Human RightsCorruption
Political Science Replication Blog
Dr Nicole Janz
Assist. Prof. in International Relations
@polscireplicate

4
5/2013 Stanford - Miguel 5
How	can	we	
establish	
trust in	science?	
Providing	full	
access	to	data	
itself
Information	
about	data	
analysis	
Process	of	data	
collection
Data
transparency
Analytic	
transparency
Production	
transparency
Data	Access	&	Research	
Transparency	(DA-RT)	
www.dartstatement.org
Provide or describe 
raw data & variable 
codings
Code for models 
(SPSS, STATA, R)
Good methods 
section in paper
(see Lupia/Elman 2014; Moravcsik 2014)
Upload datasets 
used for analysis
Providing	full	
access	to	data	
itself
Information	
about	data	
analysis	
Process	of	data	
collection
Data
transparency
Analytic	
transparency
Production	
transparency
Quantitative research
University training
Replication
exercises
10
Why should you replicate?
Learn Statistics
Publish
Reproducibility 
routine
• Real life data
• Author decisions
• Bugs included
• More fun than textbook
• When are published 
results really 
reproducible?
• Add value
• Publish faster
Use terminology accepted in your field
Duplication Replication
Verify	research	results	
exact	same	data	set
exact same	methods
Test	the	robustness	of	
the	original	research	
results
new	data
new	models
Political	Science	(see	King	2003)
Practical steps in a replication study
1 Select paper
2 Access data & code
3 Identify each variable
If you got to this point, you completed a duplication.
4 Reproduce tables, figures
5 Compare
2-3 weeks
3-4 weeks
Practical steps in a replication study (II)
6 Add value 
• new data
• new variables
• new model specifications
• theoretical contributions
7 Compare
8 Get feedback from peers
9 Journal submission
4-6 weeks
months
You now completed a full replication!
Comparing your results with the original study
• Exact same data and methods: results cannot be duplicated
• New data, experiments, models, methods: 
describe exactly at which step the results changed and why
• Different measures of a concept can naturally yield 
different results
• Different lab conditions may lead to different results
Clarify with precision the extent to which you 
were able to replicate the author’s results.
Gary King (2006)
Communicating failed replications
Be professional!
What replicators write
“We … find that coding errors, selective exclusion 
of available data, and unconventional weighting 
of summary statistics lead to serious errors” 
(Herndon et al. 2013)
“If we cannot even reproduce the original results 
using the same publicly available data, there is 
no need for further commentary.” (Miller et 
al, 2001)
How original authors often respond
“less realistic”, “inconsistent with the substantive 
literature,” and “of limited utility” (Mansfield, 
Milner, and Rosendorff 2002)
“fundamentally flawed” 
(Peffley, Knigge, and Hurwitz 2001)
“statistical, computational, and reporting errors 
that invalidate its conclusions” (Gerber and 
Green 2005:301). 
5/2013 Stanford - Miguel 19
Our	estimation	approach	builds	off	of	the	
methodology	and	data	used	by	Gomez
et	al.	(2007)	…,	adding	measures	of	electoral	
closeness	in	order	to	focus	on	how	the	
randomly	assigned	cost	(rain)	has	a	different	
impact	depending	on	the	electoral	
environment.
… we analyze a dyad-year data set (used by
Rauchhaus 2009) to examine whether existing
findings on the effect of symmetric nuclear weapons
possession on conflict are robust to the
improvements noted above. We find that once pre-
nuclear dyadic conflict is controlled for, symmetric
nuclear dyads are not more likely to experience low-
level conflict.

Transparent	
workflows
4 Steps to Transparency
Fixed Folder 
Structure
Comment your code
Clear methods 
section in paper
Share materials
Fixed folder structure
Decide on a template structure for each project
Never alter raw data
Main
Readme.txt
Paper
/Data
/Code
On	the	fly….
/Results
/Materials

Comment your code
# loading data
# variable transformation
# merging tables
# models for table 1



Methods section in paper
Describe methods clearly
Name exact models with citations for statistical 
choices
Footnote should contain software versions
If space is restricted: Appendix

Share your materials
Readme file
Data, code, variable codebook
Information to reconstruct data from original sources
Data sharing platforms
Github
OSF
Inter-university 
Consortium for Political 
and Social Research
https://dataverse.harvard.edu/dataverse/NicoleJanz
Code
Data
Variables



Provide or describe 
raw data & variable 
codings
Code for models 
(SPSS, STATA, R)
Good methods 
section in paper
(see Lupia/Elman 2014; Moravcsik 2014)
Upload datasets 
used for analysis
Providing	full	
access	to	data	
itself
Information	
about	data	
analysis	
Process	of	data	
collection
Data
transparency
Analytic	
transparency
Production	
transparency
Quantitative research
Replication 
Exercises
Transparent 
workflows 
5/2013 Stanford - Miguel 42
Five	Selfish	
Reasons	to	
Share Data	
5/2013 Stanford - Miguel 43
Quality Establish	trust	&	credibility	in	your	
work
Reputation Be	known	as	a	transparent	
researcher	in	your	field
Citation Your	data	will	be	cited
Consistency Transparent	workflow	makes	it	
easier	to	re-use	your	own	data	later
Practicality Meet	journal	&	funder	standards
(Markowetz 2015)
nicole.janz@nottingham.ac.uk
@polscireplicate
Political Science Replication Blog
P
ho
to
: S
am
 M
at
he
r
Thank 
you!

Providing	full	
access	to	data	
itself
Information	
about	data	
analysis	
Process	of	data	
collection
Data
transparency
Analytic	
transparency
Production	
transparency
Provide (partial)
• Interview 
transcripts
• Field notes
• Videos…
Describe which
evidence supports 
which claims 
Discursive 
footnotes / 
supplement
Explain how data 
were collected:
• Interviewee 
selection
• Participants
• Documents…
Qualitative research
Literature on Replication
• King, Gary. (2006). How to Write a Publishable Paper as a Class 
Project,copy at: http://gking.harvard.edu/papers
• Janz, N. (2015) Bringing the Gold Standard Into the Class 
Room: Replication in University Teaching, International Studies 
Perspectives, Article first published online: 9 March 2015. Copy at: 
http://tinyurl.com/q2qnrvn
• Brandt et al. (2014) The Replication Recipe: What makes for a 
convincing replication? Journal of Experimental Social 
Psychology, Vol 50, pp. 217-224. Copy at: 
http://tinyurl.com/poe474k
• Markowetz, Florian (2015), Five selfish reasons to work 
reproducibly. Genome Biology 16:274. 
47
Literature on Transparent Workflow
• Christensen, Garret (2016). Manual of Best Practices in 
Transparent Social Science Research 
https://github.com/garretchristensen/BestPracticesManual
• Open Science Framework. Transparency and Openness 
Promotion (TOP) Guidelines. https://cos.io/top/
• TIER Documentation Protocol 
https://www.haverford.edu/project-tier/protocol-v2
• Janz, Nicole & Figueiredo, Dalson (2017, March 13). 
Workshop: The Gold Standard of Reproducible Research. 
Retrieved from https://osf.io/2fqnw/ (slides, handouts)
Literature on Political Science Debate
Moravcsik, A. (2014). Transparency: The Revolution in Qualitative 
Research. PS: Political Science & Politics, 47(1), 48-53. 
doi:10.1017/S1049096513001789 
Lupia, A., & Elman, C. (2014). Openness in political science: Data 
access and research transparency. PS - Political Science and 
Politics, 47(1), 19-42. doi: 10.1017/S1049096513001716


Research Reproducibility in Political Science

Haaker, Maureen

Some practical protocols for transparency for qualitative research

Chirstina, Silver

Session 2: Integrating the Five-
Level QDA® method into 
CAQDAS teaching: examples in 
ATLAS.ti and NVivo
Dr Steven Wright
Lancaster University
Presentation Overview
Background and Context of Teaching and Support
Integrating 5LQDA: 
pre-emptive work
Further development
Student Responses, Experiences and Feedback
Teaching Reflections
Limits and Opportunities – where next?
Quick Intro
Who am I?
Learning Technologist at Lancaster University in Faculty of 
Health and Medicine
Independent CAQDAS consultant www.caqdas.co.uk
PhD In Technology Enhanced Learning (Oct 2014) – used 
ATLAS.ti
Senior ATLAS.ti Trainer
NVivo Certified Expert
Undertaking research project on CAQDAS use
Course Delivery
Courses designed and delivered 
in “Fully Blended” Mode
Taught sessions in classroom with a 
live link for remote participants 
Recordings available afterwards
Bookable one-to-one sessions 
either face-to-face or via Zoom
Focus on individual project, 
planning, tasks, components
Audience allowed + segments of 
session recorded and shared
Remote screen share / 
presentation enabled
Remote control of desktops via 
Zoom.
Background and Context: The Course(s)
Pilot workshop series used free online worksheets 
and datasets from Silver and Lewins (2014) together 
with sections from software-specific books >>>
Developed into taught sessions for PhD Research 
Training Programme within different faculties on:
Making an informed choice of software (what they do, 
transcription, ATLAS.ti vs NVivo – differences / SWOT 
analysis)
Intro, Next Steps and Advanced (individual project 
consultation) for ATLAS.ti and NVivo
Developing my own model for working with the packages 
effectively
My model for the doing (and avoiding 
traps) POETS
Workshop 1
POETS
Preparation
Preparing data for import
Organisation
Laying foundations for analysis
Creating a framework for writing, 
annotation and coding
Considering Cases
Exploration
Annotation
Visualisation
Workshop 2 + Consultations
POETS
Tagging
Code structuring
Applying Codes
Synthesis
Exploratory queries
Co-occurrence trees and tables
Exporting reports
Initial stages – starting a project 
(Workshop 1)
Prepare
•File naming
•Formatting data
•Lit Mgmt
Organise
•Memoing
•Structural auto-
coding
•Creating document 
groups
•Creating Code 
Groups
Explore
•Read and Annotate
•Link
•Use Reflexive 
Codes
Tag
•Applying codes 
with definions
•Code renaming and 
structuring
•Reviewing (with 
merging and 
splitting)
Later stages of project development
(Workshop 2 + Consultations) 
Prepare
Organise
Explore
Tag
Synthesise
The missing bit/s… 
Why are you doing things?
Advantages
Students like structure and want to avoid 
pitfalls and have effective tricks
Sequential but not restrictive
Majority of projects have very similar 
research designs (12 +/-2 interviews, 
thematic or GT analysis) 
Based in experience and expert 
recommendations
Emphasises the things that are often 
skipped over (preparation, exploration, 
annotation)
Disadvantages
Mechanistic? Simplistic?
Doesn’t connect practices clearly to 
theoretical or conceptual framework
Can appear to be a recipe
Doesn’t really explore or focus on “why”
Shows and Tells rather than encouraging 
exploration and critical consideration.
Lacking:
Something “higher” (conceptual)
Something ”before and after” (thinking
about “what next?”)
Something to frame it all 
Integrating 5LQDA: Pre-emptive work
Initially, in 2015, 
introduced the 
outline model with 
publication of: 
Silver, C., & Woolf, 
N. H. (2015) article –
and using concepts 
and emphasising 
model > 
Explored using 
worksheet …
Good idea – NEEDS 
context!
Integrating 5LQDA: As Core Course Design
With publication of books and revisions to a course chose to move 
to a more thorough integration for 2018 
KEY BENEFIT: A common framework for exploring different CAQDAS 
packages
Avoids beauty contest / disciplinary affiliations from example projects
Set up course to begin with APW and build from there
Worked *really well* with those who engaged
Wonder if it intimidated those who planned less/felt broadly confused
Required a lot of reading – 3 chapters for p/t DL working students non-trivial
Caught some issues early and provided framework for providing advice 
Student Feedback
my only feedback would be how useful it is to first have an 
understanding of the really basis practical aspects of using NVivo 
before trying to then utilise the 5 Level method. So I would 
recommend having both books to hand (Woolf & Silver and Bazeley
& Jackson) and cross referring between them as you go along with 
your project. What is really useful is having access to the booking 
system for one-to-one sessions with you 
Student responses contd.
Here's my first go with the APW. Have found the general process 
really helpful in making me think about what I want to get out of my 
literature review. Also have found the book and videos very useful -
even though I think I have read a couple of the chapters about 3 
times each!
Limit or Opportunity? Selecting packages,  
not just components within them. 
However have ground to a halt on the more 'advanced' step of 
comparing and creating assciations between my sources. What I 
want to do is to be able to say Research A (as documented in a 
journal article) is related to Reseach B (as documented in another 
journal article) for reasons such as follow-on study, influenced the 
methodology, is used to support latter's argument, is argued with 
etc. I think I might be heading to a PROJECT MAP but am not sure 
how to make the connections?
BTW - I am using a Mac which I think slightly restricts a couple of 
options.
Teaching Reflections
APW – when engaged with - helps students to clearly set out there 
aims and thereby guide focus of instruction and support and 
providing advice on choices
Needs to be shown as a benefit – prep for this workshop is helping with that 
by identifying student quotes and comments to re-use in the future (and a 
demo project too)
If used prior to software selection then can inform that choice (with 
support and advice)
However the detail that can come in APW may obfuscate this….
But the framework applied and common books support transition
Demonstrates that thinking and planning are essential!
Some Challenges
Can provide framework for “what to consider when choosing best 
package for you” but doesn’t particularly guide that. 
However- that is good because localized expert knowledge still has a place!
Students + researchers are always time-poor and want to know how in 
terms of:
What’s the best way?
OK, if not the best then foolproof way?
• OK, how about lowest risk?
• What does everyone else normally do / what is the convention? 
These don’t map well to the “consider all the components and what they could do”
Instruction needs to hold these tensions in balance: to be non-restrictive yet 
supportive and context-sensitive
My approach has been and will continue to be to try and combine 5LQDA for the 
higher-level and exploratory with the POETS approach as a low-risk sequence
Next steps
Integration at an earlier stage: do some planning before you choose 
package…
“Conceptual Framework” has caused confusion – often FAR too high 
a level
Need to develop clearer strategies and activities to clarify and differentiate 
this from high-level “theoretical framework” and to support explication of 
this
Additional, New and Ongoing Challenges
Constant change of software: terminology, interface etc
Component Orientation Videos and Screenshots out of date
Opportunity to create alternatives locally
Potential to contribute collectively?
References
Silver, C., & Woolf, N. H. (2015). From guided-instruction to 
facilitation of learning: the development of Five-level QDA as a 
CAQDAS pedagogy that explicates the practices of expert 
users. International Journal of Social Research Methodology, 18(5), 
527-543.
Silver, C., & Lewins, A. (2014). Using software in qualitative research: 
a step-by-step guide (2nd ed. ed.). Los Angeles: SAGE.
Woolf, N.H., & Silver, C. (2018) Qualitative Analysis with ATLAS.ti / 
MAXQDA / NVivo: The Five-Level QDA® Method. Routledge.


Harnessing CAQDAS packages

This video explores how academics can work with communities in research to bring about more resilient collective futures, drawing on action research methods, and a communities of practice approach

Cameron, Josh

Collaboration in research: harnessing the creative power of boundaries | Dr Josh Cameron

This video explains what survey measurement error, its effect on total survey error, and how we can estimate it

Oberski, Daniel

Survey measurement error and how we can estimate it.

Qualitative data on the move: using qualitative data diary apps to collect multi-modal dat

Radcliffe, Laura

Leighann, Spencer

Qualitative data on the move: using qualitative data diary apps to collect multi-modal data

What is methodological pluralism? Graham Crow was presenting at the 8th ESRC Research Methods Festival, 3rd - 5th July 2018 at the University of Bath. The Festival is organised every two years by the National Centre for Research Methods (NCRM

National Centre for Research Methods

Not a member yet