It's always interesting to see which data analysis or surveys get covered by mainstream media outlets and which don't. A bunch of media outlets chose to highlight a study this week comparing 289 teens (average age of 17 in 1996) who took abstinence pledges at that time with 645 teens who were otherwise similar but didn't take abstinence pledges. That is, the two groups shared, among other things, similar religious, sexual and political views. The study did not in any way compare either of these groups with the 11,000 or so teens from the same data set who didn't take pledges and weren't similar in their political views or religious views. So the study, whatever one thinks of it, is comparing two very specific and relatively small groups of people against each other -- teens who took pledges with teens who were similar but didn't take pledges. It may not make for as sexy of headlines but the best social science tends to have such limited scope. It enables you to control for variables much better and retain statistical significance. And it's a great idea for a study -- and one that has some pretty striking results. Research Janet Rosenbaum, who has done numerous studies in this field, found that there was no difference in the rates of sexual activity between these two groups of individuals, except that the pledge takers were somewhat less likely to use birth control. To me, this is exciting and significant enough for a news story. While there is much dispute over the whether this is the right way to measure such significance (some say that you must wait for 10 years of prolonged education to study effectiveness), the author is saying that her measurement determines the effectiveness of abstinence pledges when compared to teens with similar socio-religious outlooks.

Unfortunately, it wasn't exciting enough for the media. Much of the coverage erred by intimating that the study compared those who took abstinence pledges with all other teens, which is not true.

Here's the Daily Mail lede, for instance:

Teenagers who vow to remain virgins until they marry are just as likely to have pre-marital sex as other young people, a study has found.

BEEP. Wrong. MSNBC headlined its summary of the analysis this way:

Study: Teenage 'virginity pledges' are ineffective

Misleading. This study did not compare teens who take abstinence pledges with those who don't and if it did, it certainly wouldn't show abstinence pledges to be ineffective. Instead, the researcher attempted to compare highly similar groups who did and didn't take virginity pledges -- "apples and apples" as she has said.

My only problem is that while she was able to control for many variables, there was no reported control for the type of sex education the two groups received. It's impossible to know, for instance, whether any member of either group received any sex education, much less formal abstinence education.

And yet that didn't stop other media reports from saying that the study is an indictment of abstinence education. This was, in all fairness, probably helped along by the study's author blaming abstinence classes for the difference in birth control rates.

Here's Fox News, for instance, saying that this very limited study actually proves that "Abstinence-only programs do not delay the onset of intercourse." This study didn't even look at abstinence-only education programs. It looked at abstinence pledges, which may or may not be a part of abstinence-only education programs. Do reporters not understand what abstinence education is? As defined federally, at least, it doesn't even mention pledges. And, again, this study only dealt with pledges. I'm not even sure -- based on the reportage or study -- where the 289 pledges back before 1996 came from. Were they taken as part of a school curriculum? Were they self-reported from the Baptist youth group the teens were part of? I have no idea.

Anyway, here's the Washington Post (which used the same lede as most other outlets) saying that the study raises questions about abstinence education:

The study is the latest in a series that have raised questions about programs that focus on encouraging abstinence until marriage, including those that specifically ask students to publicly declare their intention to remain virgins.

If you're going to indict abstinence education, it would be helpful for the reader to know whether abstinence pledges are part of formal abstinence curriculum, how much of a part, how little the study looked at types of sex education teens received, etc. There's simply no discussion of that in the article.

The Post report does do a good job of explaining aspects of the study in laymen's terms, but look at how quickly it broadens out to something completely beyond the scope of the study, which I'll stop repeating:

The findings are reigniting the debate about the effectiveness of abstinence-focused sexual education just as Congress and the new Obama administration are about to reconsider the more than $176 million in annual funding for such programs.

The article quotes numerous people saying the study shows that abstinence education, which I believe fewer than a quarter of students receive, should be stopped. We also hear from someone who supports abstinence education:

Proponents of such programs, however, dismissed the study as flawed and argued that programs that focus on abstinence go much further than simply asking youths to make a one-time promise to remain virgins.

"It is remarkable that an author who employs rigorous research methodology would then compromise those standards by making wild, ideologically tainted and inaccurate analysis regarding the content of abstinence education programs," said Valerie Huber of the National Abstinence Education Association.

I do appreciate the inclusion of this quote, although I'd like to hear more about the other complaints she has about the study -- other than that abstinence programs aren't about one-time pledges.

While according to the study there were no significant differences in the sexual activity of pledgers and their very similar peers -- they both significantly delayed sex compared to other teens but eventually had sex at the same age, had the same number of sexual partners, engaged in the same type of sexual activity, etc. -- the percentage of the 289 pledge takers who did have sex before marriage were less likely to use birth control. The study's author attributed the difference to "what youths learn about condoms in abstinence-focused programs." While the story got a counterpoint from someone disputing Rosenbaum's anti-abstinence education charge, it would be more helpful to know if there was any difference in sex education between the groups studied and, if so, what that difference was. Without knowing this information, the opposing allegations don't mean much.

The author herself has some interesting discussion of media coverage of the report at her blog and this Time Q&A with Rosenbaum is also helpful.