In this blogpost I’m sharing the process we went through in choosing to use the software package MAXQDA to analyse the writings of volunteers in relation to the MOP Directives “Social Divisions” and “Life Lines”. I hope this will help others make their own informed choice between the range of software available.
Software designed to allow researchers to manage and analyse qualitative data are collectively known as CAQDAS packages. CAQDAS stands for Computer Assisted Qualitative Data AnalysiS. These packages have a range of features including data management and organisation, content searching, code and retrieve, writing and visualisation, querying (e.g. Boolean, proximity) and output. For more information see the CAQDAS Networking Project.
There are many different qualitative software packages to choose between (see here for some reviews). There is no “best” package; each has its pros and cons and therefore the decision-making process is best made in relation to the characteristics of particular projects. For some it might not be a given to use software at all. If that’s the case, I’d still recommend going through a process like the one described here because it will help rationalise whether using one of these software packages would be useful.
How to choose
The first step is to list and prioritise the needs of the analysis, then compare products in terms of how their features can be harnessed for those needs. The aim is to choose the program that will enable you to fulfil your prioritised needs in the most streamlined way. There are usually some compromises – because not all the programs have the same features. Prioritizing the project needs will help you see which compromises you can live with and which you cannot.
The first step is to be clear about the broad objectives of the study. For the Defining Mass Observation project we have a number of overarching objectives relating to the whole project which were important to consider in choosing a CAQDAS package:
Overarching project objectives
- a) to illustrate the value of MOP data for social scientific enquiry;
- b) to facilitate its opening up for the purposes of secondary analysis; and
- c) to contribute to the representativeness debate around the use of MOP data.
You can read more about these objectives here.
In addition are the specific research questions we are starting out with. In choosing a CAQDAS package these relate to analysing the writings provided by volunteers in response to the two MOP Directives we are focusing on in this part of the project. There will be a blogpost about the research design of the project as a whole later on.
It’s worth saying at this point that qualitative research design is emergent. That means that the research questions you start out with likely change as the analysis proceeds, which is why those that informed our choice of software are called “initial research questions”. We have three, two of which have sub-questions:
Initial research questions
- How do writers describe their perceptions of their own and others’ identities, in relation to class, politics, race and gender?
- how to writers perceive their educational trajectory to have influenced their class status and social mobility?
- do the ways MOP writers reflect on identities reflect socio-economic classifications used by social researchers?
2. How do writers perceive the structure of their lives?
A. how and why are certain events significant?
B. how and why are meanings are attached to events and decisions?
3. Of the writers who have responded to both Directives, how have their perceptions of their identities and their lives changed between 1990 (Social Divisions directive) and 2008 (Life Lines directive).
In order to fulfil the overarching project objectives and the research questions, we listed our key analytic and practical needs:
– we need to be able to combine numeric information about MOP writers’ characteristics with their actual writings;
– we need to undertake an ‘inductive thematic analysis’ of the writings (see Braun & Clark, 2006 for a nice overview of the different types of thematic analysis );
– we need to be able to track the contributions of writers who responded to both the Directives;
– we need to be able to access keywords and phrases used by MOP writers that indicate the concepts we are interested in;
– we need to be able to map concepts within writings in response to each Directive in isolation, as well as compare across both Directives;
– we need to be able to interrogate the ways writers discuss Directive themes according to their characteristics.
– at least 3 researchers need to be able to work with the qualitative software;
– the software needs to be intuitive and easy to learn;
– we need the software project we create to be accessible to non-expert users.
Having thought these needs through in more detail, we prioritised them by putting them into a table in order of importance. This allowed us to rank the needs, as shown in the table below. As a result we chose to use MAXQDA. You can find out more about MAXQDA from their website.
Note that the table shows the results of our evaluation of MAXQDA only. I’m not showing in this blogpost the comparison with other CAQDAS packages, because the point is to outline the nature of our decision-making process rather than to compare products. In reality we constructed a table with several other columns – one each for the programs we looked at. I find it useful when doing this to colour the cells to show visually in which dimensions the different programs are evaluated positively. Here I’ve made the text bold here to highlight the reasons why MAXQDA was seen as a good choice. In the larger comparison chart, the cells were highlighted green for positive evaluations, red where a particular program does not enable a requirement, and left white where any of the programs had the required features – resulting in a heat-map type matrix of requirements by software program features.
|Requirement – in priority order
||Evaluation – of MAXQDA features in relation to requirements
||There are three researchers on the project who will contribute to the analysis and use the software. They have different responsibilities and roles and therefore we need the ability to isolate and integrate aspects at different times.
||Although MAXQDA does not allow concurrent work by multiple analysts, whereas other packages do, its teamwork import/export features are sufficient for our needs.
Two of the researchers have prior experience of working with MAXQDA in team projects and therefore are familiar with the protocols we need to put in place to enable systematic and streamlined team-work in MAXQDA.
||Intuitive and easy to learn
|This project runs over 15 months and therefore timescales are tight. When making the decision we did not know if the researcher we are recruiting would have experience of using any package and therefore a program that could be familiarised with quickly was important.
The overarching objectives of illustrating the value of MOP for scientific enquiry and opening it up for secondary analysis also necessitates an accessible means of communicating our analytic process and findings to other researchers who may not conversant with CAQDAS packages.
|Intuitiveness of software and ease with which individuals learn to harness it for sophisticated analysis are subjective. We therefore used our experience of teaching CAQDAS packages as a proxy measure for this requirement: MAXQDA is amongst the most straightforward to teach.
||Assessable to non-expert software users
||An intended output is to make the software project we develop available for other researchers to access.
||That MAXQDA has a free Reader-version that enables those without software licences to open projects and view analysis was an important factor in our choice, as it will enable us to share our database with others easily.
||Linking of quantitative information with qualitative texts
||This is a key analytic priority of this project. Quantitative information about writers’ characteristics is being collated and cleaned for the purposes of statistical analysis in order to contribute to the debate around the representativeness of MOP writers and the value of these materials for social science research.
||All of the leading CAQDAS packages allow numeric information to be imported and linked with the qualitative materials to which they correspond; although this is a fundamental need of our research, it was thus not a determining factor for the choice of software.
||Data-driven qualitative analysis
||We are adopting an inductive thematic analysis of MOP writers’ responses to two Directives. We therefore need software features that we can use for this purpose.
||Any of the CAQDAS packages would enable in-depth qualitative analysis so this was the least important requirement in our prioritized ranking and not a factor that impacted upon our choice.
||Track responses of writers contributing to both Directives
||An important aspect in opening up MOP materials for research purposes is to illustrate how the writings of individuals over time can be tracked.
||All of the leading CAQDAS packages will allow for tracking via combining or grouping data contributed by the same individuals. So this was not a determining factor in our choice of software.
However, MAXQDA’s Document Comparison chart provides an alternative way of comparing application of (groups of) codes at the level of data files, not available in other packages. We considered this will be of benefit when considering whether and how individuals perceptions differ when responding to different MOP Directives.
||Map concepts within and between Directives
||The nature of the data (i.e. that it was generated in response to open-ended Directive questions) means we have no overview of content at the outset, as is the case when generating data through customary qualitative data collection methods, such as interviews or focus-group discussions. An initial analytic task will therefore be to map out the content of writings at a high level, in order to identify the prevalence of concepts so that we can focus analysis.
||All the leading CAQDAS packages enable high level mapping of the application of codes in data files in tabular (numeric) format with access to the corresponding texts for qualitative interpretation.
MAXQDA’s Code Matrix Browsers, however, are particularly easy to generate and provide clear and accessible visualisations. The requirements of 2) and 3) make these features attractive.
||Interrogate Directive themes by the characteristics of writers
||Exploring the extent to which the perceptions and experiences of MOP writers differ according to their characteristics is one way in which we hope to contribute to the debate about their representativeness.
This means we also need an accessible means of showing other researchers, who may not be familiar with the software, the results of our work.
|There are several “mixed methods” features in MAXQDA that we can use to present joint displays of dimensions in the data. The ones we evaluated as being particularly useful for this project are Crosstabs, Configuration Tables, turning codes into Categorical Variables and constructing Typology Tables.
The Quote Matrix feature enables a joint display of quantitative characteristics and qualitative texts to be directly outputted which will allow us to share findings easily.
Udo Kuckartz (2012) paper on mixed methods in MAXQDA gives a clear overview of these features.
||Access and auto-code for keywords and phrases
||This is particularly important for the Social Divisions Directive because we want to consider the extent to which established socio-economic classifications are understood by and reflected in writers’ accounts and to focus on evocative language used in the texts.
||All of the CAQDAS packages have word search type tools which we could use for this purpose. In MAXQDA we can create and save our own dictionaries to locate multiple keywords and phrases which is attractive but not thought to be a major aspect of our decision.
Choice of software is always a combination between analytic and practical priorities. The outcome of a process like the one discussed here might prioritize analytic needs over practical ones. For Defining Mass Observation the practical needs outweighed the analytic because of the overarching project-level objectives.
It’s important to say that we are not claiming that this project could not be undertaken using a different CAQDAS package; far from it.
But when in the position of being able to make a choice for a specific project it is important that the choice is guided by the needs of the project. That requires both an understanding of the differences between programs as well as being clear about the overarching project objectives and the specific research questions.
Braun, V., Clarke, V., 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 77–101. doi:10.1191/1478088706qp063oa
Kuckartz, U., 2012. Realizing Mixed Methods Approaches with MAXQDA.