Single tile gray graphic gets subdued into the cave floor. But building more adjoining ceiling&floor; will look different. Also, if built in the dark parts of the cave, they’ll get illuminated when looking at their direction.
You’ll need a torch (or campfire) to initiate the build in the dark, but once going: you can take a break, put out your torch, then facing the in-progress tile, select Make | Building | continue Paused process. (It might be an oversight/glitch, as if you’re not facing the in-progress tile and select the direction with NumPad number; that’ll fail prompting “pick a direction that you can see”)
One of my currently-on-hiatus characters builds his hideouts in caves. If/when he spots adventurers, robbers, Njerpez in the wilderness, he packs his belongings and moves on.
I think he has built in 5-6 caves. Returning to some occasionally. Some have been a single stay only.