Trends in Cognitive Sciences recently published a provocative letter by a pair of MIT researchers, Ted Gibson and Ev Fedorenko, which has been causing a bit of a stir in the language camps. The letter - "Weak Quantitative Standards in Linguistic Research" and its companion article - have incited controversy for asserting that much of linguistic research into syntax is little more than - to borrow Dan Jurafsky's unmistakable phrase - a bit of "bathtub theorizing." (You know, you soak in your bathtub for a couple of hours, reinventing the wheel). It's a (gently) defiant piece of work: Gibson and Fedorenko are asserting that the methods typically employed in much of linguistic research are not scientific, and that if certain camps of linguists want to be taken seriously, they need to adopt more rigorous methods.
I found the response, by Ray Jackendoff and Peter Culicover, a little underwhelming, to say the least. One of the more amusing lines cites William James:
"Subjective judgments," they claim, "are often sufﬁcient for theory development. The great psychologist William James offered few experimental results."
Yes, but so did "the great psychologist" Sigmund Freud, and it's not clear whether he was doing literary theory or "science"... More trivially, James was one of the pioneers of the fields and didn't have access to the methods we now have at our disposal. That was his handicap - not ours.
We can contrast that (rather lame) response with what computational linguist Mark Liberman said about corpus research last week in the New York Times:
"The vast and growing archives of digital text and speech, along with new analysis techniques and inexpensive computation, are a modern equivalent of the 17th-century invention of the telescope and microscope."
Here, here, Mr. Liberman. I couldn't agree more.
Last month, Michael Ramscar and I published a seven-experiment Cognitive Psychology article, which uses careful experimentation and extensive corpus research to make something of a mockery of one piece of "intuitive" linguistic theorizing that has frequently been cited as evidence for innate constraints. Near the end of the piece, we take up a famous Steve Pinker quote and show how a simple Google search contradicts him. After roundly (and amusingly) trouncing him, Michael writes - in what must be my favorite line in the whole paper -
"Thought-experiments, by their very nature, run into serious problems when it comes to making hypothesis blind observations, and because of this, their results should be afforded less credence in considering [linguistic] phenomena.”
No doubt, this one-liner owes some credit to a brilliant P.M.S Hacker quote (actually a footnote to one of his papers!):
"Philosophers sometimes engage in what they misleadingly call 'thought-experiments.' But a thought experiment is no more an experiment than monopoly money is money."
Let's stop mistaking 'thought experiments' for science.