If your construction cost is significantly greater than your resale value, and you intend to borrow a large portion of this cost, then there is not enough collateral for the lender to recover their investment.

A house which you will treasure, but is considered undesirable by the masses will be a problem.

normally you couldn't get a loan because banks would be worried about the resale value of a "underground" house. But maybe recently you could have because they were just giving loans to anybody. But now you probably wouldn't have a chance. You'd have to build it with cash, and they cost about 20% more than a conventional house, last I checked.