You're moving goalposts. You denied language mattered:

> then the reality is that no big effect has been found for the choice of programming language on software quality

Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room.

Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence.

...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post.

However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper.

"Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code."

and from that same paper:

"As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets."

> Sure. Here's code review...

I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does).

> process matters a lot

And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.

> You're moving goalposts.

Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC).

> It's not proof, but it is strong evidence.

I don't think it matters if it's strong or weak evidence, as it's not even about software development.

> the C sets can be up to an order of magnitude higher than the mean values for the C++ sets.

As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined.

If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric.

> I just said it didn't deliver quality

But those papers show that, unlike language, process does have a big impact on quality.

> With enough process you can close any hole in a language, but it gets exponentially expensive.

You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.

> I'm talking about software development

And the value different languages may add is entirely about software development.

> I don't think it matters if it's strong or weak evidence, as it's not even about software development.

Programming languages are not about software development? Oh do go on.

> As I mentioned in another comment, C is not a good example.

A study of C vs C++ strongly indicates your claim about languages is false and suddenly C is not a good example? I'm trying to believe I'm just misunderstanding you but it's getting more difficult.

> The original prediction of that theory was that no 10x improvement would be made by a single language improvement

This isn't what you said. Let me remind you "If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality"

The goalposts aren't being moved, you've just bought them plane tickets to barbados.

> If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric

Another claim. Show me the study that says that.

> But those papers show that, unlike language, process does have a big impact on quality.

It can if done properly; it does not do so automatically. You can have code reviews that are of little use because you have no spec to review the code against. I have been in that very position. I don't dispute the value of process but it's a necessary but not sufficient condition to bring about quality. I also agree code reviews are good, if done properly.

ME >> With enough process you can close any hole in a language, but it gets exponentially expensive.

YOU > You are now making an unsubstantiated claim that contradicts a substantiated claim.

If I use C I have to worry about garbage leaks, double-freeing pointers, out-of-bounds accesses etc that can be detected with code reviews. If I use python, I never have any of these problems so a code review need not check for these. That's a lot cheaper.

(Edit: weird stuff happening to my post, may appear twice)

> This isn't what you said.

I can't repeat all the caveats every time I say something. I assume readers believe my comments are at least reasonable, even if they disagree with them. For convenience, I restated the current state of knowledge more precisely in my previous comment. If you're interested in what I have to say, assume I'm not an idiot, and perhaps we could have an interesting discussion, and if you think I'm an idiot, then there's no point in arguing at all.

> Show me the study that says that.

Show me the study that says otherwise. I don't need to provide evidence for the lack of an effect (although I did). The lack of an effect is, at the very least, the default hypothesis in this case (as in many others) as there is no theory suggesting we should believe otherwise (while a theory that explains why languages increasingly have smaller effects has made correct predictions).

> It can if done properly

That's your hypothesis. We've been trying to find evidence of that, or others like it, for a long time -- both in academia and in industry -- without success. We simply do not observe that the choice of language today (among reasonable ones etc.) has a big effect, either in studies or in industry practice. But if you've found a language that can drastically reduce software development costs and/or increase quality at scale, that discovery can be easily translated to billions of dollars. Go ahead and make them.