Kipping is showing misleading snippets on his videos. Thank you Res Ipsa for pointing that out!

[Marcus]

Well, he might. He has a practical bent to his thinking that borders on obsessiveness— something that, imnsho, more than a few scientists could benefit from!!

That's very kind of you Marcus, but I don't think I understand papers better than their authors do. I just do my very best to understand what the authors are saying in a given paper. It just so happens that the skills needed to understand what a judge is saying in a legal opinion also apply to understanding what the authors of a scientific paper are saying. And, once I dig into a paper, I am obsessive about doing what needs to be done to understand what the authors are actually saying as opposed to what I'd like them to say.

I've found there are a few handy rules of thumb that help us non-science folks understand an academic paper:

1. Don't rely on just an abstract. Find and read the whole paper if you can. If you can't, be extremely cautious about drawing conclusions from the abstract alone.

2. Understand what the paper is about -- what is the question the authors are asking?

3. Make sure you understand the definitions of important terms. Don't assume that key terms are being used in their ordinary sense.

4. Understand the author's assumptions, which are commonly found in the introductory section of the paper. "Assumptions" include reliance on other published papers. Sometimes you need to take a look at those papers to understand the context of the paper you are trying to understand.

5. Understand what the authors are not saying. Sometimes the paper has a separate limitations section, which is really handy for lay folks like me. Other times the limitations aren't in a separate section, but appear in or following the conclusion.

6. Understand the conclusion. If you are going to use a paper as an authority for a proposition you are arguing, you need to understand what the authors actually conclude. Using a paper as an authoritative source using something other than the author's actual conclusion creates a significant risk of being flat-assed wrong.

All those steps get you is a good shot at understanding what the authors are saying in the paper. And I think that's as far as I've gone in the referenced thread. Critiquing a paper is an entirely different process. The papers discussed in the referenced thread us methodologies that are far beyond my capacity to make any sort of independent judgment. The methodology section of that Bayesian model paper made me break out in a cold sweat. For me, determining whether a paper is "good" or "bad" involves reading lots of other papers or critiques of the paper I'm interested in.

Excellent points, and yes, I was being somewhat facetious which is why I said “you might…”!

Another piece of advice i would add is to start by perusing the discussion section of these types of papers, typically found right before the conclusion. It’s a quick way to get a handle on how the researchers themselves felt their experiments or analyses went, and typically adds a good layer of understanding to the information found in the abstract.

I also prefer to look at the raw data rather than the conclusions made about such data, but that’s pretty time consuming. It’s necessary, however, because the range of assumptions people can make, and subsequently get published, about data is frequently astonishing.

(Oh and one last thought— if it’s a mopologist paper, NEVER trust a footnote to be accurate, honest, academically legitimate, or even related to the sentence in the paper preceding the footnote number.)

[Morley]

I've found there are a few handy rules of thumb that help us non-science folks understand an academic paper:

1. Don't rely on just an abstract. Find and read the whole paper if you can. If you can't, be extremely cautious about drawing conclusions from the abstract alone.

2. Understand what the paper is about -- what is the question the authors are asking?

3. Make sure you understand the definitions of important terms. Don't assume that key terms are being used in their ordinary sense.

4. Understand the author's assumptions, which are commonly found in the introductory section of the paper. "Assumptions" include reliance on other published papers. Sometimes you need to take a look at those papers to understand the context of the paper you are trying to understand.

5. Understand what the authors are not saying. Sometimes the paper has a separate limitations section, which is really handy for lay folks like me. Other times the limitations aren't in a separate section, but appear in or following the conclusion.

6. Understand the conclusion. If you are going to use a paper as an authority for a proposition you are arguing, you need to understand what the authors actually conclude. Using a paper as an authoritative source using something other than the author's actual conclusion creates a significant risk of being flat-assed wrong.

All those steps get you is a good shot at understanding what the authors are saying in the paper. And I think that's as far as I've gone in the referenced thread. Critiquing a paper is an entirely different process. The papers discussed in the referenced thread us methodologies that are far beyond my capacity to make any sort of independent judgment. The methodology section of that Bayesian model paper made me break out in a cold sweat. For me, determining whether a paper is "good" or "bad" involves reading lots of other papers or critiques of the paper I'm interested in.

7. Recognize that any given paper probably does not provide an overview of the consensus opinion in the field. This is because if the paper is talking about something that's already known and agreed upon, then it's unlikely find a publisher.


I think it's fair to say that anyone who has done research, and published on a given subject, recognizes just how narrow and cloudy even the best of research is. Sometimes we think, Oh my God, this paper has numbers, the authors must know what they're talking about! Unfortunately, that's not always true. Data, like concepts, can be false friends.

[Doctor Steuss]

When a literature review is included in a paper, it makes me want to send the authors a Christmas card.

[Res Ipsa]

When a literature review is included in a paper, it makes me want to send the authors a Christmas card.

When I want to figure out what the state of the science is in a field, I look for literature reviews. They're generally not overly technical and are written in a way that ordinary folks can understand.

[Res Ipsa]

I've found there are a few handy rules of thumb that help us non-science folks understand an academic paper:

1. Don't rely on just an abstract. Find and read the whole paper if you can. If you can't, be extremely cautious about drawing conclusions from the abstract alone.

2. Understand what the paper is about -- what is the question the authors are asking?

3. Make sure you understand the definitions of important terms. Don't assume that key terms are being used in their ordinary sense.

4. Understand the author's assumptions, which are commonly found in the introductory section of the paper. "Assumptions" include reliance on other published papers. Sometimes you need to take a look at those papers to understand the context of the paper you are trying to understand.

5. Understand what the authors are not saying. Sometimes the paper has a separate limitations section, which is really handy for lay folks like me. Other times the limitations aren't in a separate section, but appear in or following the conclusion.

6. Understand the conclusion. If you are going to use a paper as an authority for a proposition you are arguing, you need to understand what the authors actually conclude. Using a paper as an authoritative source using something other than the author's actual conclusion creates a significant risk of being flat-assed wrong.

All those steps get you is a good shot at understanding what the authors are saying in the paper. And I think that's as far as I've gone in the referenced thread. Critiquing a paper is an entirely different process. The papers discussed in the referenced thread us methodologies that are far beyond my capacity to make any sort of independent judgment. The methodology section of that Bayesian model paper made me break out in a cold sweat. For me, determining whether a paper is "good" or "bad" involves reading lots of other papers or critiques of the paper I'm interested in.

7. Recognize that any given paper probably does not provide an overview of the consensus opinion in the field. This is because if the paper is talking about something that's already known and agreed upon, then it's unlikely find a publisher.


I think it's fair to say that anyone who has done research, and published on a given subject, recognizes just how narrow and cloudy even the best of research is. Sometimes we think, Oh my God, this paper has numbers, the authors must know what they're talking about! Unfortunately, that's not always true. Data, like concepts, can be false friends.

Excellent addition. Thanks.

[Res Ipsa]

That's very kind of you Marcus, but I don't think I understand papers better than their authors do. I just do my very best to understand what the authors are saying in a given paper. It just so happens that the skills needed to understand what a judge is saying in a legal opinion also apply to understanding what the authors of a scientific paper are saying. And, once I dig into a paper, I am obsessive about doing what needs to be done to understand what the authors are actually saying as opposed to what I'd like them to say.

I've found there are a few handy rules of thumb that help us non-science folks understand an academic paper:

1. Don't rely on just an abstract. Find and read the whole paper if you can. If you can't, be extremely cautious about drawing conclusions from the abstract alone.

2. Understand what the paper is about -- what is the question the authors are asking?

3. Make sure you understand the definitions of important terms. Don't assume that key terms are being used in their ordinary sense.

4. Understand the author's assumptions, which are commonly found in the introductory section of the paper. "Assumptions" include reliance on other published papers. Sometimes you need to take a look at those papers to understand the context of the paper you are trying to understand.

5. Understand what the authors are not saying. Sometimes the paper has a separate limitations section, which is really handy for lay folks like me. Other times the limitations aren't in a separate section, but appear in or following the conclusion.

6. Understand the conclusion. If you are going to use a paper as an authority for a proposition you are arguing, you need to understand what the authors actually conclude. Using a paper as an authoritative source using something other than the author's actual conclusion creates a significant risk of being flat-assed wrong.

All those steps get you is a good shot at understanding what the authors are saying in the paper. And I think that's as far as I've gone in the referenced thread. Critiquing a paper is an entirely different process. The papers discussed in the referenced thread us methodologies that are far beyond my capacity to make any sort of independent judgment. The methodology section of that Bayesian model paper made me break out in a cold sweat. For me, determining whether a paper is "good" or "bad" involves reading lots of other papers or critiques of the paper I'm interested in.

Excellent points, and yes, I was being somewhat facetious which is why I said “you might…”!

Another piece of advice i would add is to start by perusing the discussion section of these types of papers, typically found right before the conclusion. It’s a quick way to get a handle on how the researchers themselves felt their experiments or analyses went, and typically adds a good layer of understanding to the information found in the abstract.

I also prefer to look at the raw data rather than the conclusions made about such data, but that’s pretty time consuming. It’s necessary, however, because the range of assumptions people can make, and subsequently get published, about data is frequently astonishing.

(Oh and one last thought— if it’s a mopologist paper, NEVER trust a footnote to be accurate, honest, academically legitimate, or even related to the sentence in the paper preceding the footnote number.)

I haven't tried starting with the discussion section, but I will. If the paper has a separate limitations section, I sometimes start with that. The CDC does something that I really like in write-ups on papers. They have text boxes for "What we already know," "What this research adds," and "What more do we need to know." Or something like that. Of course, one must also understand what "know" means in that context.

I'd love to be able to interpret the data, but I just don't have the necessary math and stats chops, which is why I love having you and the others who do have the chops as resources. It's one of the things I really enjoy about this particular forum -- there are lots of participants with expertise in all kinds of fields who are willing to answer questions or address issues in a way that I could never hope to do myself.

[doubtingthomas]

Res Ipsa doesn't understand probabilities and scientific papers better than the researchers, Thomas, he just understands them better than you do.

Step back, my friend.

Carroll said, "Probability of life starting could be 10^-100 per planet. We just don’t know". Can you please tell me what's wrong with his statement?

[Morley]

Res Ipsa doesn't understand probabilities and scientific papers better than the researchers, Thomas, he just understands them better than you do.

Step back, my friend.

Carroll said, "Probability of life starting could be 10^-100 per planet. We just don’t know". Can you please tell me what's wrong with his statement?

As you already know, that's not the statement I'm asking you to step back from.

It's this bit of snark:

I want to congratulate Res Ipsa for brilliantly understanding probabilities and scientific papers better than the researchers. Res Ipsa understands Kipping's paper better than Kipping himself. Just WOW!

edit: Thomas, for what it's worth. consuming research intelligently is sometimes not as straight forward as one might hope.