Difficult question. I know that Matlab has some very strong Machine Learning libraries which make it extremely convenient to experiment. As much as I like GNU R and FOSS in this case I would say Matlab has the upper hand.
For one it has a huge userbase which posts code examples and solutions to various problems. Another being that Matlab has nice Machine Learning libraies in my opinion.
I find that programming in R is much nicer. I find that the language is more powerful and supports closures which is something I've always missed in Matlab. Extending R in C is also quite easy.
Its probably worth pointing out though that R is primarily for statistics. It walks all over Matlab in this domain (because most statistics researchers use R). However, for machine learning, signal analysis, system identification and working with ODEs I wouldn't be surprised if Matlab is easier and better supported.
If anyone had machine learning startup ideas that _would_ implement themselves, Eliezer Yudkowsky would probably be interested.
Tangentially, there seem to be a lot of computer vision startups--there's a company called Cortexica with an unreleased iPhone wine label identifier, and Evernote's nonpareil handwriting recognition makes me wish its algorithm were open; open-source alternatives seem to focus only on printed text.
I've only worked with Matlab and it's great for putting together library functions but horrible for writing any functions yourself. Is R any better?