I spent a little time working at Pearson on a textbook project, and yeah, they're pretty profitable, but the up-front costs are enormous. The production budget for the biology book I worked on was $10 million. And that was for a book that would be widely used at the high-school level.
For a math book like this, I'd guess they spent more like a million getting it done. I wouldn't be surprised if it was $50-100K just for the typesetting.
There is something more than just high production costs going on, though, for math books. Consider Apostol's "Calculus". Volume I new is $150-270 on Amazon. Volume II new is $160-270.
These were first published around 1961. A second edition of Volume I was published in 1967, and a second edition of Volume II in 1969. There have been no further editions. The schools that use Apostol today are still using those late '60s second editions.
They were about $20 per volume when I bought them in 1977, which is equivalent to about $80 per volume in today's money.
A book that cost $20 in 1977 and was made to the same standards today--some paper and binding quality, etc.--would probably have to cost at least $100 today retail. Unfortunately, costs for paper and printing have outpaced inflation generally. Most trade books now are printed very cheaply--the paper and binding are crap.
I estimate the "fair" price of a textbook to be around $50-80; math (particularly calculus) is a textbook price I consider to be more naturally expensive, since they tend to cover more material. Your estimation of decades-old textbooks costing about $80 in today's money seems to vindicate that viewpoint.
For a math book like this, I'd guess they spent more like a million getting it done. I wouldn't be surprised if it was $50-100K just for the typesetting.