I think I'm almost entirely with you except for these two
statements:
>It's premature to take for granted that a paper brain emulator can exist in principle.
I think it is premature to say we could simulate all of quantum mechanics with sheets of paper, so on that technicality, I totally agree. However, I think it's quite unlikely to be the case that we couldn't in principle simulate the requisite components required for a faithful replication of consciousness. But you're correct that it's probably not a given.
>Finally there's something of a category error in making the claim that pieces of paper can be sad. A paper emulation of a bar magnet would not itself be magnetic. In the world of the emulation it would produce an output that would model magnetism, yes, but that's the extent of it. Our paper brain emulator, if it were constructible, would produce output that would be a model of sadness in the emulation, but it would not actually be sad.
I think I disagree with almost all of this. The bar magnet is a bit of a bait and switch because the system, the input, and the output weren't well defined. In consciousness, you need to define the system, system input and system output properly. If the system has persistence of thought/computation, and the inputs and outputs can be defined in a way identically to those of your own consciousness, then it would actually be sad, in every way that you are. In particular, if you replace bits of paper with "neurons", your argument should absolutely still hold. So the logical extension of this argument is that either YOU also aren't capable of being sad, or your consciousness does not originate in your brain. Both of those I think are pretty close to sufficiently absurd that we can accept them as givens.
>It's premature to take for granted that a paper brain emulator can exist in principle.
I think it is premature to say we could simulate all of quantum mechanics with sheets of paper, so on that technicality, I totally agree. However, I think it's quite unlikely to be the case that we couldn't in principle simulate the requisite components required for a faithful replication of consciousness. But you're correct that it's probably not a given.
>Finally there's something of a category error in making the claim that pieces of paper can be sad. A paper emulation of a bar magnet would not itself be magnetic. In the world of the emulation it would produce an output that would model magnetism, yes, but that's the extent of it. Our paper brain emulator, if it were constructible, would produce output that would be a model of sadness in the emulation, but it would not actually be sad.
I think I disagree with almost all of this. The bar magnet is a bit of a bait and switch because the system, the input, and the output weren't well defined. In consciousness, you need to define the system, system input and system output properly. If the system has persistence of thought/computation, and the inputs and outputs can be defined in a way identically to those of your own consciousness, then it would actually be sad, in every way that you are. In particular, if you replace bits of paper with "neurons", your argument should absolutely still hold. So the logical extension of this argument is that either YOU also aren't capable of being sad, or your consciousness does not originate in your brain. Both of those I think are pretty close to sufficiently absurd that we can accept them as givens.