Phone Home?

A Wall Street Journal post highlighting the results of a Pew Research Center religious attitudes poll is making the rounds on social media today. The Pew Research poll results were released and published on their own site yesterday under the heading, "How Americans Feel About Religious Groups: Jews, Catholics & Evangelicals Rated Warmly, Atheists and Muslims More Coldly." Although I claim no particular insight into the veracity of their data or claims, I find frustrating a few things about this survey. Overall, both the WSJ snippet and Pew's own highlights remove so much context and methodology from the survey that it is difficult to get a feel for the significance of the results. Pew's methodology of differently categorizing respondents' professed religions and query religions seems awkward. Furthermore, omitting significant categories (according to their own categorization and numbers) certainly must skew the poll results.

Omitting significant categories results in questions a bit like: "When did you stop kicking your dog?" There is no good answer. The Pew report inquired about the following categories of religions, which I list here (and annotate) by group-size as a percentage of all those surveyed. Lest we compare apples and oranges, the numbers affiliated with a given category are also self-reported numbers according to Pew, available on their website.

Faith (% of adult adherents)
Evangelical (26.3)
[Roman] Catholic (23.9)
Mormon (1.7)
Jewish (1.7)
Atheist (1.6)
Buddhist (0.7)
Muslim (0.6)
Hindu (0.4)

But if we are doing comparisons, why these? Why, for example, are what Pew refers to as the "Mainline" Protestants (18.1%) not included? Or Orthodox Christians (0.6%)? Or Protestants affiliated with "Historically black churches" (6.9%)? For that matter, why are Evangelicals singled out among Protestantism? Evangelicalism is only one of the three groups into which Pew subcategorizes Protestants (51.3%), which, according to Pew, comprise Mainline, Evangelical, and Historically black churches. Looking more closely into the full survey results posted shows that Pew used "White evangelical," "White mainline," and "Black Protestant" categories to group respondents' declared affinities but did not use these same categories in the queries. In most surveys, the respondent categories and query categories are, indeed, different. When gathering data about religions according to religion, it would seem that having the same level qualification would be appropriate. As one small example, I am an Orthodox Christian. If you are not immediately familiar with Orthodoxy, sometimes referred to as "the best kept secret in Christendom," think: "My Big Fat Greek Wedding." Were I surveyed, how could I answer? I would question every query: Does the query include Orthodoxy as part of Roman Catholicism (which it is not, but with which it shares much), include it as one of the Protestant sects (which it is not, but with which is sometimes is associated by by folks familiar only with the break from Roman Catholicism of the Protestant Reformation and not the earlier Great Schism, which split Christianity into Roman Catholicism and Orthodoxy), or ignore it out altogether? My answers would be affected by not knowing what the questions intended. Perhaps my incessant querying of queries may be why I have never been surveyed by Pew. Or perhaps it is because these types of surveys repeatedly use the same limited population.

The Pew survey was based on a population of 3217 respondents in its "American Trends Panel" (ATP). Pew recruited ATP members based on a 2014 telephone survey of 10,000 "nationally representative" respondents. Pew points out in their materials that utilizing the ATP approach allows them to better track trends and changing views over time and predict behavior. They mean, of course, that they may be able to predict the behavior of the ATP members and that, with any luck, that behavior is generalizable to the larger population. Perhaps Pew has never surveyed me because (like many of my colleagues in technology and many young, single people) I have not owned a landline phone in a decade.

For comparison, the American Religious Identification Survey (ARIS) is a survey of religious adherents in the U.S. referenced by the U.S. Census Bureau. The 2008 ARIS included 54,461 respondents, and both the categories and the numbers differ from the Pew survey. One would assume differences over six years, of course--changes influenced by such factors as population increase, changing attitudes about religion, differences in generational religious affiliation, changing immigration patterns, and so on. While calculating percentages and comparing the two surveys, one huge discrepancy struck me. In 2008 (ARIS), only 0.94% of the population declared itself Evangelical Christian. According to Pew, the percentage is now 26.3. Did Evangelicalism really grow by 2786% in six years? Or are these kinds of surveys simply inherently flawed?

It is not merely the small population or the apparently mismatched categories that makes me question these findings. A very small population can produce great survey (or usability) results if that population truly is representative for the purpose, and most data findings have outliers. Pew claims that the ATP is nationally representative. However, based on my comparison with U.S. Census, CIA Factbook, and Gallup Poll data of their population demographics, the ATP appears to be un-"representative" of the U.S. population in almost every regard. With a potential population of almost 3000¹ respondents, it seems that Pew could shape any given ATP survey "round" (Pew's term for a particular survey instantiation) to be significantly more representative for the stated survey goals than this one appears to be. When asking about Mormons, Buddhists, and Muslims, for example, perhaps it would be appropriate to actually include Mormons, Buddhists, and Muslims. (Oh, my!)

¹ Only 2941 responses are recorded for religious affiliation (or no affiliation). The only demographic that reflects Pew's stated 3217 responses is whether the respondent is male or female.