Friday, February 23, 2007

Peer Review is changing

Public Library of Science (wikipedia) journal PLoS ONE will use an Editorial Board instead of three luminary boffins (the traditional peer-review model) to Peer Review articles. PLoS ONE is an Open Access Journal - anyone can annotate and discuss articles source: SciAm.

Wired Magazine gave PLOS a rave award in 2004.
PLoS posts new research online, making it available to everyone from high school students to scientists in the developing world. Authors agree to let anyone annotate, excerpt, link, and otherwise add value. And that's not all: Online readers pay nothing so its "open access." Funders of research - usually government agencies - cover the cost of publication up front.

my
highlights / emphasis /
comments

Japanese TV Show Admits Faking Science

According to a story published today in Nature, the Japanese public's insatiable appetite for "infotainment" coupled with the outsourcing of the creation of TV shows to independent production houses may have encouraged Japan's most popular science show to fake results, overdub interviews with researchers and mislead viewers.

After an outcry from the Japanese media, the company has cancelled the series and is reviewing all 520 episodes....

reposted from: SciAm
my highlights / emphasis /
comments

Knowledge, belief, and what counts as good science: More thoughts on Marcus Ross - by Janet D. Stemwedel

I reposted the original debate here.

Posted on: February 22, 2007 5:56 PM, by Janet D. Stemwedel

Following up on my query about what it would take for a Young Earth Creationist "to write a doctoral dissertation in geosciences that is both 'impeccable' in the scientific case it presents and intellectually honest," I'm going to say something about the place of belief in the production of scientific knowledge. Indeed, this is an issue I've dealt with before (and it's at least part of the subtext of the demarcation problem), but for some reason the Marcus Ross case is one where drawing the lines seems trickier.

reposted from: Adventures in Ethics and Science
my highlights / emphasis /
comments

First, for the sake of argument, I want to set aside all questions of Marcus Ross's actual motivations in pursuing a Ph.D. in geosciences and plans for using that Ph.D. now that he has it. To get to the issue I'm after here, I'm going to assume a Ross-doppelganger who is not just "trying to get through" the process of satisfying a thesis committee, and who is committed to arguing in good faith. (I do not know what the case is with the actual Marcus Ross, but we're going to set it aside as irrelevant to my question, so please don't email me with personal testimonials on either side.) Also, let's stipulate that this Ross-doppelganger has made no special effort to conceal his religious beliefs (which include a belief that the Earth is no older than 10,000 years), but that he may not have felt any particular need to bring his religious beliefs up in the context of his scientific studies or research. (Whether intellectual honesty would require that he call attention to particular of his religious beliefs in a scientific context is something we'll get to.)

Are we clear on the character in my thought experiment? Good.

Our Ross-doppelganger wants to study geoscience. He is interested in the sorts of phenomena geoscientists study, the features of these phenomena they observe, the theories they construct to explain the phenomena, the tests to which they subject these theories -- the whole ball of wax. He applies himself to learning how to make good observations, how to use instrumentation and analytic methods of various sorts to generate further data, how to do good calculations, how to use statistical methods to get good measures of the statistical power of the results and the sized of the error bars. He has a thorough acquaintance with the geosciences literature and a firm grasp of the theories guiding research in his field (as well as keeping up to date with the new approaches described in the current literature).

His efforts make him someone who can make excellent observations in the field, work up data exactingly, and develop explanations that stand up to rigorous testing (which he can also perform well to evaluate his own explanations and the explanations of others).

It seems reasonable that this level of competence in a field requires mastery of a number of empirical and analytic techniques, a thorough understanding of the theoretical structure of the field, and a good grasp of how scientists draw inferences from data and justify those inferences.

Does it also require that he believe the theories of his field are true? Does it require that he believe that the patterns of inference at work in building geoscientific accounts of the world necessarily result in true claims?

The answer to the second question is pretty clearly "no". Scientists are well aware that their reasonable inferences can go wrong. Sometimes this is a matter of drawing inferences from a necessarily incomplete set of data (what with the problem of induction and all that). And sometimes it's because the theoretical structure within which the inferences are being drawn is not precisely right -- possibly because it's missing some important feature, or a little off on another. Of course, this means that scientists can draw perfectly good scientific inferences without having full faith in the truth of their theories.

Rob Knop has an excellent post about using theories that can't all be true in physics. In it, Rob writes:

All the time in science we have to behave as if we believe something is true, even though deep down we don't believe it really is true. Here's a concrete example: in Physics, we have two very excellent, very well-tested fundamental theories. For gravity, there is General Relativity (GR). For everything else, there is Quantum Mechanics (QM). Unfortunately, the two are inconsistent; if you try to do quantum mechanics where gravity is significant, you get nonsensical results.

This means that GR and QM can't both be right. And, yet, we soldier on, using GR every day to do gravity calculations, even though it probably isn't completely correct. We learn the rules and play the game so that we can get the results out. ...

[A]lthough we know that either GR or QM isn't the most fundamental description of reality-- most physicists assume it will be GR, rather than QM, that needs to get modified-- we do believe, and indeed know, that GR is an excellent approximation to what is going on for a wide range of situations. GR may not be "The Truth," but it does work for predicting the orbit of Mercury or the gravitational lensing of light around a cluster of galaxies. ... GR may not be the fundamental truth, but we really believe that there is mass there when gravitational lensing measurements tell us that it is there.

(I should note that Rob uses this example to set up a contrast with the real Marcus Ross. We're considering what to say about my Ross-doppelganger, so we'll have to see whather it's possible for him to avoid the pitfall into which Rob sees the actual Marcus Ross falling.)

In the current state of affairs, you can't simultaneously believe that GM gives a true account of the physical world and that QM gives a true account of the physical world. At most, only one of these theories can be true. And, it's possible that neither is true. Nonetheless, good physicists can work with both of them to make sense of data, to explain various phenomena, etc. This would seem to say that, strictly speaking, belief in the truth of a theory is not a requirement for use of that theory to generate good science. Philosopher Larry Laudan points out (in his book Progress and Its Problems) the useful distinction between accepting a theory and pursuing a theory.* Scientists can pursue all manner of theories that they take to be pretty far out, even unlikely to be true -- to see whether anything useful could come from working with that theory. Arguably, the willingness of scientists to explore theories that they don't accept (at least at the outset) can be very productive for science. Isaac Newton was not inclined to think action-at-a-distance was a good way to model reality, but pursuing what seemed like a nutty idea got us to a theory of gravity (whatever we mean by theory) that made sense out of Kepler's laws.

Here's another consideration: If scientists are serious about testing their theories, belief in those theories could be an impediment. This is part of why people like at least the spirit of Karl Popper's picture of the scientific attitude: scientific testing is looking for evidence against our theories, not evidence for them. We may love those theories to bits, but we cannot let our acceptance of them be unconditional -- they must prove themselves worthy of our love by standing up to a barage of tests. (Imagine an adaptation of Mr. Jealousy in which Annabella Sciorra plays the theory of Quantum Mechanics.)

Put another way, what makes scientific knowledge scientific is that believing in the truth of your claim contributes exactly nothing to whether other scientists will accept your claim. Scientific claims are supported with evidence of a certain sort (including empirical evidence, possibly fit with theories that are well-supported by empirical evidence, etc.). The insistence on testable claims is not just a step away from arguments from authority ("It's true beacuse I say it's true!") but also a step away from relying on your gut-feelings to make the judgment. Empirical science elevates the evidence of our senses over the deliverance of our gut. In a sense, this means that, as a matter of methodology, scientists have good reason to be cautious in their regard for what they're inclined to believe. That they believe it surely doesn't win the argument.

Now, back up a step. Our Ross-doppelganger has showed his skill in the pursuit of theories that other scientists in his field accept. Perhaps the Ross-doppelganger even accepts these theories in a Laudanian way -- he recognizes their problem-solving prowess compared to all the alternatives currently in use or development. But, he doesn't believe these theories. What he believes is what his religious instruction has taught him on this matter.

Assume for the sake of argument that the Ross-doppelganger knows his religious beliefs on things like the age of the Earth have no scientific credibility -- that they don't have empirical support, don't fit into the inferential structure in the right way, etc. Thus, he's not going to hold up his religious conviction as persuasive evidence that the science must be wrong. Let's also assume that the Ross-doppelganger will happily allow that the account of things that fits best with the empirical evidence is the account from the geosciences. In other words, from the point of view of offering natural explanations for natural phenomena, the geosciences are doing a good job.

Are you inclined to view the Ross-doppelganger as a good scientist? Can we trust the scientific knowledge he builds?

You might object, "How can he call what he's producing 'knowledge' if he doesn't believe it's true?" But if you're going to challenge the Ross-doppelganger on these grounds, you may have to challenge the physicists as well.

Here, return to Rob's analysis. When Rob writes,

GR may not be "The Truth," but it does work for predicting the orbit of Mercury or the gravitational lensing of light around a cluster of galaxies. ... GR may not be the fundamental truth, but we really believe that there is mass there when gravitational lensing measurements tell us that it is there.

I take it what he's getting at is that science involves a certain commitment to the reality of the observational data -- that there is a planet Mercury, that it does have an orbit, that there is light, there is gravity. Without some kind of acceptance that empirical data are real -- that we get them through certain kinds of interaction with the physical world -- there would be absolutely no reason to think it problematic to just make up data.

What seems less clear cut is what kind of commitment science requires to the causes behind the empirical data. There are some (Bas van Fraassen comes to mind**) who suggest that the task of science is accounting for the empirical data with empirically adequate theories, but that the "hidden causes" you might infer lay beyond those data are not the kinds of things you can establish with certainty. In other words, there are some scientific claims you can support with empirical data, and other scientific claims that fit really well with the data -- maybe better than any of the competing claims we've cooked up to date -- but about which it is possible, or even proper, to maintain a healthy agnosticism.

If the Ross-doppelganger acknowledges that the empirical data are what they are -- that this is what the world presents to our senses (and to the instrumentation we use to extend our senses) -- is he on solid scientific footing? Remember that he knows what to do with those data to draw proper scientific inferences, that he knows how to test his inferences against possible scientific objections, and so on.

Does the fact that he entertains as a possibility, albeit one that he acknowledges as scientifically untestable, that the empirical data are produced by a God who also made the world within the last 10,000 years -- even if this possibility plays no role at all in his scientific work -- disqualify him as a proper scientist and disqualify his work as properly constructed scientific knowledge?

If so, why?

________
*Acceptance in Laudan's account is similar to belief, but it's worth noting that, in contrast to some other kinds of belief we might have in everyday life, acceptance of a scientific theory is something for which Laudan thinks we can have rational grounds (on the basis of the theory's current overall level of problem-solving power compared to the available alternatives). Note also that as other theories are developed, it's perfectly possible that we will be presented with rational grounds for accepting a different theory over the one we had accepted.

**The full articulation of van Fraassen's view is given in his book The Scientific Image.

Comments

The only flaw with the GR/QM scenerio and the YEC/current-geological-science scenerio is that BOTH the GR and the QM are scientifically valid and mathematically accurate to within acceptable degrees of certainty.

Neither are "believed" in so much as supported by the evidence available at the time at which they are applicable. That they contradict at certain points is not, in itself, a problem of belief. Physicists who's fields reflect one or the other continue on as if the contradiction didn't exist, because at acceptable levels of certainty it doesn't. They're both science, and extremely accurate sciences in spite of the contradictions and funnyness that goes on when the very small works around the very big.

However,

YEC is not only NOT VERIFIABLE by any standard at all (unlike, say, "angels" or supernatural gods), it CONTRADICTS all evidence utterly.

This is not a case of believing in one while ignoring the other merely because at acceptable scientific levels of accuracy, "it works". This is believing something that totally contradicts ALL evidence at hand, even without the philosophical hand-waving of "God made it *look* like it was 4 billion years old, but its really young".

The comparison is inapplicable.

Posted by: Joe Shelby | February 22, 2007 06:34 PM

Joe, the point wasn't to suggest that YEC is like either GR or QM. Rather, the point was to ask whether using geological theories (which deep down, one doesn't believe) might be like working with GR (which deep down, one doesn't really believe).

I take it there's no case to be made here that YEC can be construed as scientifically supported belief. The worry is whether "good science" requires that you have a certain level of belief in the scientific theory you're using.

Posted by: Janet D. Stemwedel | February 22, 2007 06:50 PM

Does one really "not believe" in GR, or does one merely accept that GR is the best available explanation for the phenomena THAT ACTUALLY WORKS within that acceptable certainty. One can accept that GR is not "right" while at the same time accept that the better explanation eludes us, "but we're working on it" (which we are).

That's totally different from suggesting that one can accept that GR is not right by *believing* that "planets and stars spin on crystal spheres", the astronomical equivalent to YEC.

When a scientist says "I don't believe in GR" its completely different from when some "faithful" chap says "I don't believe in everything anybody has ever said about modern cosmology". They really are two different meanings to "believe" and the context remains key. The comparison has not resolved that contextual difference that changes the weight behind "believe".

Posted by: Joe Shelby | February 22, 2007 07:13 PM

Well, Janet, you're a philosopher, so you know that there are all kinds of tenable possibilities for empirical observations. The only thing that you can prove to exist is your own mind. Aside from that, there are all kinds of completely logical, if not incredibly likely, possibilities. The world could have been created 10 seconds ago. There is absolutely no way to disprove this hypothesis.

But as a scientist, you're committed to the idea that the most parsimonious explanation is likely the truth. There's all kinds of evidence that the world has been around a very long time. I can look at a sample of uranium and see that, on average, one atom decays into thorium about every ten seconds (and then through all the chains to lead). Then I can look at a piece of rock that should be all uranium, but 50% of it is lead. The most parsimonious explanation is that this piece of rock is billions of years old.

That's how scientists decide how the world works. Looking for empirical evidence to explain how things are the why they are. If I honestly believed that the world was created 10 seconds ago, and I would continue to do so no matter how many radiodating studies I did, I'm not being a scientist. I'm not actually looking for evidence about how the world really works. And as such, I have no motivation to be truthful about what I find, because it's all just a game anyway.

I think that's the real problem here. Science really relies on people being utterly open about their observations. That means being personally motivated to report the truth.

Posted by: Brian | February 22, 2007 07:31 PM

Joe, what does a belief matter to the work done? Surely what goes on inside your own head only becomes a problem if it goes beyond that and influences your work and writings. If I was to deny GR because I honestly accepted the Crystal Spheres hypothesis (don't worry, I have never even heard of this one before) would it really, honestly matter, so long as I didn't allow that prejudice into my work?

Lets be honest with ourselves, all scientists will have some prejudices. The whole point of peer review and standards of ethical conduct (including intellectual honesty in all its forms) is to minimise the impact of those prejudices.

In the case of the hypothetical here, the belief is kept entirely detached from the work produced (otherwise there would have been no way any PhD, or science fair sticker for that matter, could have been awarded). It would be no different to an atheist making an argument to Christians that referred to the bible. You may not believe it is true, but that doesn't stop you understanding the others viewpoint and using it to make arguments.

In this case you can see the paper written as almost a thought experiment. The same sort of view taken by the church with Galileo when his model made beautiful predictions but didn't fit with dogma. While such a stance is in no way scientific, it doesn't make the science done any less viable to those who do accept it.

There were no grounds to deny the PhD on the work done or the general religious beliefs held by this man. There may have been other reasons, such as straight up dishonesty in presenting data that he believes is false. But if he believes the data is real and his reading of it is the best that exists materially, then there is no real way to avoid admitting he did as much as any other PhD student in the way of science.

Posted by: Paul Schofield | February 22, 2007 07:39 PM

One thing I haven't seen brought up in any of these discussions is "Last Thursdayism". Last Thursdayism is the recognition that God could have created the world last Thursday with all of the historical elements required to make the Earth (and Universe) look like it is 13 billion years old.

It is entirely possible that Ross could believe that the Earth really was created by God 6000 (or so) years ago, but with all the attributes to make it look like 13 billion years. In that case, accepting that 13 billion year age "for scientific purposes" would be a necessary part of learning more about the mind of God.

Posted by: Ahcuah | February 22, 2007 07:53 PM

1) A basic part of being a scientist is being able to suspend your beliefs. Not your disbelief -- that's easy -- but your beliefs, and especially the ones you actually like!

2) The GR/QM confict is vastly overhyped. It applies solely under conditions which are not directly observable (even in principle, AFAIK), and in which one or the other theory may reasonably "forced out" by new findings or boundary conditions.

3) In the same vein, the ultimate test of a physical theory is technology -- that is, given the theory, can you use it to create devices or conditions which did not previously exist? Both GR and QM pass this test with flying colors. Besides the astronomical observations, relativistic effects become directly relevant when, say, building a GPS system. The system we're using explicitly accounts for those effects, and it works. QM becomes directly relevant when building very small electronic devices, such as those in the computer you're using to read this comment. That works too. QED.

Posted by: David Harmon | February 22, 2007 07:54 PM