Mathematicians do care about how much "black magic" they're invoking, and like to use simple constructions where possible (the field of reverse mathematics makes the central object of study). For example, Wiles' initial proof of Fermat's last theorem used quite exotic machinery called "inaccessible cardinals", which lie outside of ZFC. Subsequent work showed they weren't needed.
Another good example of mathematicians caring which 'house of cards' their results are built on is the search for an "elementary" proof of the prime number theorem (i.e. showing it doesn't rely on complex analysis).
Edit: here's a great related discussion on MathOverflow, bringing in analogies from CS: https://mathoverflow.net/questions/90820/set-theories-withou...
most mathematicians act as though they are Platonists, even though, if pressed to defend the position carefully, they may retreat to formalism https://en.wikipedia.org/wiki/Mathematical_Platonism
Mathematicians begrudgingly retreat to formalism and foundations when pressed because its easier to defend, but the day-to-day of contemporary mathematics is much more an explorative process of a "real" mathematical landscape. They aren't concerned with foundations because it "feels" self-evident that the mathematics they are discovering is true (because their means of discovery, rigour and proof, "guarantee" it to be so).
A lot of the comments here are making false assumptions like "but surely mathematicians all know that their field is ultimately justified as a symbol-pushing game from some axiomatic system right?" in the same way one might say "surely all computer scientists know that every language ultimately compiles down to 1s and 0s processed by a CPU" but that is not at all how most mathematicians think about doing mathematics.
Architects know and care that they are building things made of atoms. And then ... pretty much don't think about atoms because the objects and relationships they are working on are abstractions well above the fine details of atoms.
And having designed many structures with architectural methods, and seeing those buildings built and stand, it doesn't worry them to hear physicists arguing that maybe atoms are different than they thought. They figure, their experience with architecture and its artifacts has proven to be reliable, so there is no realistic threat of some new quantum theory undermining their work.
On infrequent situations, where their work needs to deal with some special property of some material, they don't have any issue dipping down any number of levels of abstraction. But as a practical matter, that is infrequent for most math.
I know TFA says that the purpose of foundations is to find a happy home (frame) for the mathematicians intuition. But choosing foundation has real implications on the mathematics. You can have a foundation where every total function on the real numbers is continuous. Or one where Banach–Tarski is just false. So, unless they are just playing a game, the mathematicians should care!