Here are a few thoughts I intend to share with my students on the first day of class...
How to Lead a Critical Discussion
Several students will be asked to volunteer to lead discussion each session. Everyone will have to lead discussion multiple times during the semester. If you are not leading discussion you are still expected to come well prepared. If you are not willing to participate substantively, actively, and regularly in class discussion, this class is not for you.
The purpose of leading class discussion is to practice thinking critically. This is not an easy task; it takes a lot of preparation before each class session. The authors whose work you are reading are experts in their field and they are very smart. They will rarely make simple and obvious mistakes, but that does not mean that their arguments are flawless, or that they don't cut corners.
If you work hard in this class, you will learn how to critique any argument. As you master this craft, you will also learn how to help make an argument stronger and more persuasive.
Here are some guidelines:
Read the assignment carefully and actively several days before the next class session. This means that you should be taking notes in the margins of the text while you read. Often it is a good idea to write a small phrase next to important paragraphs and passages that you know you will have to revisit. Underline key parts of the argument, particularly the thesis statement in each chapter and sub-section. You should summarize important points in a few sentences at the end of each chapter so you will remember what you have learned when you revisit the text. Don't hesitate to write all over your books, it will save you time in the long run. Give yourself time to digest the reading. Take time to think about what you have read over a day or two.
Note: Use a pen not a highlighter. Highlighters are not useful when you have to return to the text days or weeks later and you will have forgotten why you chose to highlight a particular section. As students get tired they tend to start highlighting everything in sight. If you use a pen you will have to stay alert.
Outline the author's argument on a piece of paper. A well written book is relatively easy to outline just by following headings, sub-headings, and the first sentence of each paragraph. The outline is for your own understanding. You may not summarize the readings in class. You should assume that everyone has done the reading and they are only asking you to lead off the discussion with critical and substantive comments.
When you are outlining the argument, ask yourself the following questions:
- Against whom is the author arguing?
- What political ideology informs this author's position? How do you know?
- What political ideology informs the adversaries of this author? How do you know?
- What does the author not cover? What issues are covered in haste?
Ad hominem arguments will not be useful in your critique, but it may facilitate your comprehension if you know how to situate an author along a political spectrum.
List all of the variables used in the author's argument and see if you can sketch a map of the argument. If the author is making a causal argument, you need to answer the following questions:
- What is the author trying to explain; what is the dependent variable?
- Which factors influence the dependent variable?
- Which factors are the ultimate causes and which factors are merely intervening or modifying variables?
Unless you can demonstrate its relevance to the argument, your personal feelings about the style of the author's prose are not useful. The emotional impact of the text on your worldview is also not relevant for class discussion. Please do not make comments about the style, length, jargon, font size, etc.
Assess the logic of the argument and sub-arguments. Here are a few basic logical flaws to look out for:
- Implausible Assumptions: All arguments are based on certain assumptions about how the world works. It is important to identify the explicit and implicit assumptions of an author and assess their plausibility. If the assumptions are not plausible, then the argument is flawed and speculative.
- Faulty or False Inferences: A faulty or false inference occurs when an author ignores alternative interpretations of the meaning or significance a particular fact or set of facts. False inferences may also occur if an author skips over important steps in constructing their argument, or if the data used is not directly relevant to the question.
- Argument from Authority: Arguments which rely on the authority of particular experts are admissible, but one should still scrutinize the expert to ensure that their credentials are relevant to the issue at hand. There are no objective sources, but some sources are more credible, scholarly, and balanced than other sources.
- Straw-man Argument: A straw-man argument is one in which the author intentionally presents a weak version of their rival's position in order to easily dismiss objections.
- Manichean Logic: Authors who operate in dichotomous frameworks are often guilty of essentializing and/or over-simplifying (e.g. East vs. West; Capitalist vs. Proletariat; rational vs. irrational). In most cases, a complex social phenomenon requires a study of the shades of gray and the figures at the margins of a social order. The social world is complex and nuanced. There are times when a Manichean approach might be an appropriate rhetorical strategy (e.g. a political speech, a manifesto), but it is generally not a good framework for social science analysis.
- Anecdotal Evidence: Anecdotes and vignettes can be powerful ways of persuading readers about the validity of a particular argument. However, anecdotal examples that serve as the basis for a generalization require careful scrutiny.
- Overstated Arguments: It is common to see a carefully constructed, rigorous survey or experiment be used to draw general conclusions on a much wider issue area. Authors often seek to make their work relevant, particularly in their conclusion, by overstating the relevance and generalizability of their work.
- Narrow Timeframe: When examining evidence always ask why the data presented begins and ends when it does and whether there are important events which are ignored or hidden by the timeframe. It is not acceptable for to simply use data from a narrow timeframe because that data is readily available, if the question posed requires a longer time horizon. It is incumbent on researchers to find and code the data that answers their entire question.
- Force Coding: Did the author code their own evidence? What were their guidelines? Is ambiguous evidence forced to fit into a rigid code? A good author will generally acknowledge and discuss the data points which were difficult to fit into a set coding scheme. Authors may claim to be objective because they rely on published datasets that have been coded by other authors. Nevertheless, a reliance on public data sets does not remove bias.
- Excluded Exogenous Factors: Does the author's causal model account for relevant exogenous factors? Is there a prior causal factor or an important intervening variable which is not being factored into the explanation? Is there important evidence which is being ignored or left unexamined?
- Interdependence of Independent Variables: Are the factors or variables which the author uses to explain the outcome genuinely independent of one another? If variables are not independent, then it is more difficult to sustain causal claims. Social science arguments which emphasize accurate description over prediction do not need to establish rigorous independence of variables.
- Unvarying Variables: Do the variables in the author's causal model actually vary? If the variables do not vary or if they are yes/no (i.e. "monotonic") variables, then the model may not capture a range of nuanced outcomes. If the variables vary too widely along a given metric, the model may be attempting to explain too much.
- Ignored Outliers: It is a common practice to exclude "outliers" (i.e. one or two data points that distort a general pattern) from a model. If the outlier is insignificant, then this practice is justified. However, if the outlier cannot be ignored, then the model needs to explain the outlier.
For example, many arguments about economic requisites of democracy exclude India. However, India is the largest democracy in the world and it cannot be dismissed from a general theory of democracy.
- Non-interchangeable Data Points: Some data points which occur only once have far greater significance than other data points which occur frequently.
For example, one could argue that the US has prevented 99.99% of major terrorist attacks inside the US every day since 2000. Of course, this would overlook the singular importance of 9/11/2001. (This argument is also guilty of a counterfactual fallacy, as we will discuss, since we the absence of terrorist attacks does not imply that the US government prevented those attacks).
- Logical Validity without Empirical Evidence: A logically consistent and valid argument is not necessarily a true argument. It is important to test proofs with empirical evidence. Pay careful attention to sentences where the author asks you to accept an argument based on reason or common sense.
For example, Neo-liberal economists have argued that people will only value insecticide treated mosquito nets if they have to pay for it. This sounds self-evident but recent empirical studies show that proper use of mosquito nets is relatively the same regardless of whether the nets were sold at a nominal cost or distributed for free.
- Inverted Causality: Is it possible that the causal arrows in the argument could be reversed? Is there a feedback loop between the effect and the cause? Did the author test this possibility?
- Non-Falsifiability: A strong causal argument generally specifies the type of evidence which would invalidate the author's hypothesis. In other words, the author should state what kind of evidence would force them to withdraw or restate their argument.
- Counterfactual Arguments: An argument which asks the reader to imagine the outcome of a scenario if a particular event had not occurred is speculative. We cannot know the answer to counterfactual arguments in social science. A counterfactual exercise can be useful and productive, but one cannot draw strong conclusions from such an exercise.
- Ceteris Paribus Clauses: Ceteris paribus means "all other things being equal." In other words, it is a way of predicting outcomes if one holds all other variables constant. In the social sciences, all other things are never equal. Ceteris paribus clauses should be treated as speculative assertions.
- Grandma/Grandpa Test: Is the author telling you anything which is not unknown to your grandmother or grandfather? In other words are the results of this research non-trivial?
Always evaluate the evidence presented in support of the argument. Do not gloss over charts and graphs. Take time to read through footnotes that present evidence in support of an argument. In the social sciences, footnotes, charts, and graphs are data. It is important for you to determine if the evidence presented is admissible and adequately supportive to the argument. Often times, the weaknesses in an argument are hidden in the footnotes. Scrutinize the sources used by the authors and you will be richly rewarded.
Social scientists do not "prove" things; they find evidence that correlates or strongly influences a particular outcome. You should be highly skeptical of any author who claims to have proven a particular causal relationship that is also a non-trivial finding.
Be prepared to make multiple points in each class session. As several students will be presenting each week, you will need to be prepared in case others say what you were going to point out. The more critical interventions that you come prepared to share the better it is. It is not acceptable to say that someone took all of your points.
At the end of your presentation, please provide the class with three questions that you believe we should discuss. The questions should be designed to spur debate or further inquiry.
How to Listen to a Critical Discussion
Listening in an academic context is neither an intuitive nor a passive activity. Listening requires active participation and reasoning. In my classes, students are graded as much on how well they listen to their colleague's presentation as they are on their own presentations.
Students who are not presenting are expected to come to class with the same level of preparation as the student who is leading the discussion. This means that students must have read, outlined, and thought about the material actively. Students are expected to bring the assigned reading to class.
Students who are not leading discussion should take notes on their colleague's comments. It is important to note the type of logical flaws in the author's work which have been identified by your colleagues.
While it is important to maintain a polite and civil environment for class discussion, students should never hesitate to interrupt a speaker to ask clarificatory questions or to ask a presenter to re-state the point of their comment. If the presenter is simply summarizing the material, it is not at all rude to ask the presenter to stop summarizing and state their argument.
While the student is presenting it is important to anticipate counter arguments. As you are listening, ask yourself:
- Does the presenter seem to have a grasp of how to situate the author's ideology?
- Is the critique based on a fair, thorough, and accurate reading of the text?
- Is the presenter analyzing all of the evidence or only selective portions?
- Did the author already anticipate the critique being presented by your colleague in another section of the readings?
- If the author did not anticipate these critiques, how would the author most likely respond?
- Is there an alternative position or explanation which has not been articulated?
Sometimes students will make critiques of an author based on their personal experience or material from another class. These types of arguments should not be dismissed, but they do deserve scrutiny:
- How familiar is the student with the topic at hand?
- Is the outside evidence they present pertinent?
- Did the student examine the issue in question in a systematic manner?
Write out at least three questions that you would like the presenter to answer.
Labels: teaching
Last fall I struggled to get my students to adopt proper citation protocols in their research papers. I don't know why it was so difficult to get students to use the Chicago Manual, maybe I should have devoted a class session to reviewing this topic. Only a few of the papers had proper citations; one paper had multiple styles including redundant parenthetical citations and footnotes on the same page. 
