Wednesday 29 April 2009

Race hazard

The other week I was visiting friends in Edinburgh. A great time was had by all (I hope - I certainly had a whale of a time). Aside from the joys of seeing all-too-distant friends and moseying around a beautiful city, we had a little time to play on mountain bikes at Glentress. I've never been mountain biking before. I never realised what I was missing. Oh my word, it's so much fun. I'm reasonably sucky at it (hey, first time, remember), and spent the first day vacillating between terror and refusal to attempt stuff that really shouldn't have worried me at all, and stupidly tearing down trails faster than I was capable of handling because I knew no better. And then falling off. Given that they were carting Mr. Liability around all day with them, many many thanks are in order for the folks I was with - they encouraged, scraped me up, and tried to teach me enough to stop being such a danger to myself all in the best of spirits. As a result, despite the wealth of bruises I'm still sporting, I'm totally hooked and can't wait to go again. The second day we went, by the way, I got padded up to try to stop me from getting bruises on bruises. And managed not to fall off once. Ho hum. But another glorious day of epic fun for me.

...and between those two days, we get to the point I want to do writing about: Racing games. Edd, Ruth and I convened to introduce Ruth to the Way of Game (which she picked up and ran with startlingly quickly and well). In our session, we played two entirely different racing games: Pitch Car and Formula D. The former is sort of like racing subbuteo, or shove hapenny maybe (if that means anything to you). You have a wooden track, with little fences on the outside of the corners to help keep your cars on, and you flick your little wooden disc-shaped cars around the track. That's it. If it sounds simple, that's because it is - but that doesn't stop it from being brilliant fun. The main problem with it is the amount of time you have to spend crawling around under the table tracking down the car you just blasted into the stratosphere by mistake.

Formula D is a totally different kettle of fish; it's a dice-rolling based game with a nifty gear shifting mechanic (based on the gear your car is currently in you roll a different dice). It is, however far more complicated and drier than pitch car. Having said that, we almost killed ourselves laughing when Ruth took a hairpin bend at 300 miles an hour. While being shot at. Unsurprisingly, she didn't quite make it.

Pitch car would probably be fun no matter who you played it with - anyone who'd be happy to flick a wooden car around a racetrack is going to be in the right frame of mind to enjoy the game anyway - but I definitely got the feeling that Formula D relies, like so many games, on the personalities around the table. It could be dull, tedious play if everyone was going to carefully analyse every move to pick the statistically optimal gear each turn. Or, as it was for us, it could be a riot as you wing it round every corner slightly too fast hoping not to launch yourself into a building. I guess that's why I enjoy playing games so much: it's not just the game (though I do enjoy them for their own sake), it's also a prism to direct the attention of a group of friends together into a shared experience. And how is sitting around a table with good friends having a whale of a time ever not going to be a good way to spend a few hours?

Definitely looking forward to doing all that again. If only they didn't live so damn far away.

Monday 13 April 2009

Peering in from the outside

So here's a thing: It seems an awful lot of people don't get what peer review is for, and how it works. Keeping the standard issue "I may not know what I'm talking about" disclaimer in mind, I thought I'd take a look at it since I've been both the reviewer and the reviewee in the past. And the perception that some people have seems to bear little relation to my experience of it. So, let's take a look at the strange idea of peer review that people have.

As far as I can tell, this is how the people that've bothered me with this think the process works:
1) A scientist writes a paper to prove their theory is the new theory of awesome
2) Some other scientists read said paper and then vote on whether or not it's the new theory of awesome
3) Depending upon what these other scientists vote, it's either an official New Theory of Awesome (and hence Is Science), or it's not (and hence is either junk or One Of The Great Ideas The Establishment Have Turned Their Back On).

There's a weird misunderstanding of the process of scientific endeavour in there, and it's one that is most likely linked to the way the media cover scientific research (with all the talk of "breakthroughs" and scientists "challenging" each other when they disagree). I think it's fundamentally leaning towards seeing science as received wisdom, where Experts who Know snipe at each other until the ones who win the most points get to have their idea exalted as Science and the losers are... I dunno exactly. I suspect that people think this is where the nutjob theorists who claim to have been rejected by The Establishment come from.

The funny thing is that this is almost exactly not what happens. Although the majority of scientists within a field will tend to share a consensus view of many aspects of that field, they didn't vote for it. There's not an organisation that neatly divides theories and empirical data into "science" and "not science" based on the votes of scientists. What happens to form this consensus then? Well, in my experience this:-

A researcher, working in some field or other, will to begin with spend many, many hours trying to find every paper which might possibly have something to do with what they're working on. Why? Well, for a start, how do you think researchers decide what to research? And how do you think they decide how to go about it? And how do they figure out what might already have been done many times before so they can choose whether or not to do it again (this bit's actually quite important)? By trawling through huge numbers of papers and reading bits of them.

Hold on a moment.

Bits of them?

Yes. Bits of them. Right now, they've got an epic stack of research papers on their desk/hard drive. They've pulled everything they can find which might be relevant. Most of it won't be. So now, they read through the abstracts and maybe some other bits of the papers, and throw out everything that was likely looking, but turned out to be irrelevant. Note irrelevant. Not "inconsistent with what they want to read". I mean stuff like, you were looking for papers on face recognition because you're doing a study on how people recognise faces. Unless you're planning to also look at how automated face recognition works, any papers you might have picked up which deal purely with automatic face recognition probably aren't relevant to you at all. If you find a paper which inconveniently seems to show exactly the opposite of what you hope to find - well, that's not only relevant, that's gold dust. Put that on the top of the pile. You'll be needing to read that.

Right, we've manageed to shrink our pile of papers to something manageable. Now we're going to go through it all in detail and figure out what each paper says, what the researchers did, how they did it, and if they did it sensibly. Once we've done that we can decide if we need to replicate any of their results before we carry on.

This is the important bit I mentioned earlier.

Now, if you're going to be directly building on the work other people have done, it's a good idea to (a) make sure their theoretical work holds water (b) make sure their maths isn't squiffy and (c) make sure that you can empirically demonstrate what they said they demonstrated. After all, best will in the world, they could've just got (un)lucky and fluked it. They may have messed up their experimentation. Probably best if you verify that you can pretty much do what they say you can do with their technique, right?

Obviously, you can't always do this. Sometimes you don't even want to. But in these cases, you can really rely on their results only when you have lots of other papers which have been independently produced by other people who've been in your position but actually done the verification work. If fifty research groups have already demonstrated a paper to be reliable. you're not adding much by re-running the experiment. It may still be helpful for you as a learning exercise or a baseline - in which case you'll run it anyway. Sometimes, of course, you simply can't reproduce the experiment. You don't have the resources. So you do the best you can. You check their methodology. You make sure you can reproduce any derivations. You take a look at their data and verify that what they claim it shows is, indeed, what it shows. And you keep in your mind - and likely note in any publication - that you're basing this on work that you couldn't verify and that no one else has. It'll be a worry. And it'll mean your work has less weight to it because it's built on a shakier foundation.

OK, so, you've done all your work and you write up your paper detailing what you did, how and why, what your results were and what you think they mean. And you reference all the papers you found which were relevant to your work. So, anyone really interested in what you did can go back and see the work that your work relies on.

Now, multiply this up by ALL THE RESEARCHERS IN THE WORLD.

What you see is that useful, reliable work is being checked all the time by being used. Scientists aren't voting on what they believe to be right, they're just reporting back on what they tried out, and what seemed to be reliable because they used it themselves, or because they wanted to verify that someone elses work ...well... worked. And what they're doing the whole time is criticising the work that came before - pointing out experimental flaws, errors in reasoning, over-stated conclusions. And that all gets fed back round the system. On top of that, you have to remember that scientists aren't trying to prove themselves right - you take your hypothesis and you try to prove it wrong. So, all those papers are trying to show that their central idea is false, and then reporting back on their failure to do so. (Before anyone points out this isn't always the case - I know, ideal world. But the scientific process is fundamentally based on this type of negative feedback. Even if you're trying to prove something is the case, you do this by trying to demonstrate that it isn't, if you get what I mean).

OK, so, that's science. And personally, I wouldn't consider any of the above to be "voting". And at no point does anyone decide what is or isn't science, you just have models which have proven to be useful and ones which haven't. Where does peer review come in? And why is it so important?

To answer the second question first: It isn't all that. Peer review is a very, very poor approximation to the above (which happens after publication). Just because something's appeared in a peer reviewed journal doesn't make it right. Independent verification is the thing. So what's the point of peer review?

As far as I can see, it comes down to basic quality control. It's an attempt to weed out the papers that are nonsensical, unreadably badly written, patently fraudulent, misleading, blatantly poorly designed, have obviously fudged their results, etc. It doesn't pick out the absolute best papers, it filters out the garbage that no one would (or could) ever apply the above process to in the first place. And why do we need it? Partly to reduce the expense of publishing, though I have little sympathy for that argument, but also partly to cut down on the volume of crap that researchers would have to wade through to get to the well-written papers about well-constructed experiments that are of any value at all. It's a blunt instrument, and it's far from ideal.

But don't be mistaken by the fact that something is peer reviewed. It's a start - but it doesn't make it valuable, reliable or right. Read the paper. See if it makes sense. If you have the resources, reproduce the work. If you don't, find other papers that have. Build an evidence base for its reliability. Build an evidence base for its unreliability. Compare the two. That's science.

Friday 3 April 2009

Ignorance

I'm not entirely sure where this is going, but bear with me. Hopefully a point will start waving and shouting at us as we proceed, and I can skid to a halt at its feet and pretend that's where I was trying to get to all along.

Now, a couple of weeks or so ago I got into a bit of an ill-tempered argument with a good friend of mine. It was a stupid argument, not really worth having. One of those that comes from both parties being tired, not understanding the other properly and having the added confusion of conducting the conversation through the medium of little typed words in between doing work. Standard issue falling out because what you wrote in your email isn't what the other person read, even though both of you were looking at the same words in the same order. I suspect, moreover, that no small part of the problem was that I was trying to make a general point, but doing it very badly and ended up coming across like I was bludgeoning a specific issue I knew very little about to death from a position of complete ignorance.

Which, as it turns out, is deliciously ironic because the point I was fumbling towards was this: There's a fascinating world view shift between three groups of people - those who actually know what they're talking about for a given subject, those who think they know but don't, and those that are fully aware that they don't know jack.

I'm talking about something a little more subtle than the persons own perception of how good they are at something (although, a little aside here, I know there's an interesting paper about this, and I'd love to link it here. I think it's this one that I've seen referenced before but, ho ho ho, it's pay-to-view only and I don't have a whole world of spare cash to throw at checking whether or not I'm right. Chances are I'm wrong and I'll need to go on a merry chase through several references and more pay-to-view articles to find it. Nice one, Science. Just you go ahead keeping your findings under lock and key. No wonder people turn to mentalist woo-merchants because they seen no difference between their statements of absolute truth and What Scientists Say, since joe public can have a hell of a time trying to track down the papers to read for themselves. Anyway, I digress.). Rather, the perception they have of the entire field that they're talking about.

Here's an example that I haven't been recently fighting with my friends over. Now, I happen to know, on account of spending quite a long time looking up the details and poring over them, that for lots of applications you don't need to use Windows. Especially for embedded or server based applications. In fact, I'd go as far as to say that for many applications you would be better served by using an alternative OS. I don't think that's a particularly contentious position to take. I'm not saying that you should never use Windows, just to be clear, simply that sometimes it's not the best choice for a given system. Point in case, I've recently worked on a system that needs to be running a tiny, fast, secure, kiosked OS that pretty much just exists to get a single application running and pass hardware control through to it. Now, you can do that with Windows. But you're paying for software you just don't need, and that (from the position of sitting there and trying) is much harder to cut down to the core that you want. But I've had conversations with people - recently - that have flatly stated, variously, that "no one uses anything but Windows. Ever" and "our customers insist that everything connected to their network must run Windows. Everything". The former statement is patently false (just check the statistics for the number of *nix web servers, the number of people in the media that use Macs, etc, etc) and the latter is highly debatable. So, they insist that ip enabled security cameras run Windows? They'll have a job. Such devices don't even have an operating system.

But the interesting thing, I think, the really interesting thing is that the people making the above statements Knew They Were Right. They honestly, and without any hint of one-upmanship or nonsense like that, thought that they were in a position to make these firm statements because they believed they were in the position of knowing all about that field. Now, talking to other people about the same situation shows that people who simply don't know anything about it - and know they don't - will tend to pretty much come out and say that they don't know but will often express surprise, or even outright incomprehension, that something is the way it is. But then, from the opposite side, people in a position of immersion in a field will be bemused and often unable to understand the decisions and world view that people less immersed hold. So, for example, I'm generally bemused at the way that most people seem happy enough with proprietary, closed-source software. I can't really understand why anyone would want to put up with the restrictive world of Apple. But then, I'm perfectly happy to spend an evening hacking around compiling a new kernel with debug outputs to find out why a new bit of hardware isn't working (and I suspect most people aren't). I like to know that I can have a poke under the hood if I want to (and I suspect most people couldn't care less).

So, to get back to what started this, which provides a somewhat less techy and perhaps more accessible example, I was trying to explain why I found the world of people who really know about cycling weird. A bit of backstory here - for the past ten years, my primary mode of transport has been a bike. Until this year, it's been either a very crappy bike, or a reasonably OK if you know no better bike. The best I could afford when I bought them - and, given I was a student at the time, that should paint a relatively clear picture. Even so, I've racked up many, many miles on them. Our holidays tend to involve getting on the bike and heading off into the wild blue yonder for a week or so. Our weekends often require significant pedalling to get us where we're going. Until we moved to our current house two years ago, we would transport all of our groceries through the medium of panniers. All of my commuting would be done on a bike. And now, with a one and a half year old son, we have a kickass trailer which I haul around the local countryside as often as we're able. So, in summary: I would say that over the past ten years, I've cycled quite a lot. But I can only do the most rudimentary maintenance, and have never been in any kind of cycling club or anything like that. And I certainly don't know much about bikes.

On the other hand, I have friends who know an awful lot about bikes. They have multiple bikes costing (to me) eye-watering amounts of money. And, frankly, I just plain don't understand why they'd want them. And I'm not going to - I use my bike to haul stuff (and me) around places. I can't possibly afford to have another bike, and what would I use it for? I could just about understand having a mountain bike as well as my general purpose workhorse bike, but... well... there're no mountains here. And I can't imagine why I'd ever want to ride a singlespeed bike. It looks like a world of pain for no good reason to me. Of course, from their point of view, I don't know what I'm talking about. I freely admit this. But then, that's the difference: They do cycling. It's a big part of their life. I, on the other hand, ride my bike sometimes. Of course we're going to see the entire world of bikes in totally different ways, see different things as reasonable, etc. But what I didn't realise before I started asking them about bikes (because I wanted to buy a new one that was a little more efficient than the old one) was just how little I knew. I went from a position of thinking that I knew a reasonable amount about a subject to realising that I really didn't know that much at all. And, thinking about it, you have to worry because...

...what if the things I think I know, I actually don't? How does one tell that one know nothing about a subject? You have to know enough to know that you don't know much... but if you don't know enough to know you don't know much then you might think you know lots but actually know next to nothing. And the view you have of that aspect of the world, that you know is reliable, might actually be a complete fantasy based upon your current ignorance. But you don't know that, because you know so little that you think you know a lot. I think. Maybe.

And now I need to go and have a lie down in a darkened room until I manage to un-knot the above.