I did the first ragdoll physics system about 20 years ago, when we had less compute available.[1] GJK, which the author mentions, is the preferred algorithm for convex hull collisions. GJK is a hill-climber. You start with two points, one on each object, and walk them towards each other along edges, picking the direction that produces the most improvement. This is O(sqrt N) on the number of vertices per object.

That's with two random starting points. If you're doing this repeatedly, as in an animation, you can start from the winning points of the previous round. If the objects are not rotating and moving very fast, the recheck is constant time. If they move a little, it takes slightly longer, of course. Many modern programs don't use the incremental form, but it's much faster.

GJK is a very fussy algorithm numerically. The optimization involves subtracting large numbers and caring about small differences. Loss of significance due to underflow can be a problem, and can cause the termination condition to not terminate, with the optimization cycling between a few near-optimal solutions. The loss of significance problems show up as objects settle into parallel-face contact, which is why this algorithm tended to loop when put inside a physics simulation where objects settled. Passed tests with randomly oriented convex hulls all day.

This was hard to fix. I had a hack solution that detected cycling, and Prof. Stephen Cameron at Oxford finally fixed the numerical math to behave.[2]

(You can get his code from his web site, there's a substantial license fee for commercial use, and he died years ago. I licensed the code back in the 1990s, but can't sublicense or release it. If someone really wants it, they'll need to find his heirs and negotiate.)

You also need "good" convex hulls. Your options are hulls with polygonal faces that won't be perfectly flat due to numerical precision limits, or hulls with triangular faces that are sometimes coplanar. Using a convex hull generator which can enforce a minimum break angle of about 1 degree, with polygonal faces, seems to work best.

Now there's OpenGJK.[3] Haven't looked at that. Hopefully they've dealt with all these problems. I was doing this back in the stone age of game physics, when nothing worked out of the box.

[1] https://www.youtube.com/watch?v=5lHqEwk7YHs

[2] https://www.cs.ox.ac.uk/stephen.cameron/distances/

[3] https://github.com/MattiaMontanari/openGJK

You then have the terrain chopped up into these 8x8s too, and can then do a collision test with at most 4 of them with the 1 character/entity mask.

Unfortunately it was created before Unity could use vectorization and other C# engines were just starting out as well.

The source, a great resource for learning C# SIMD, even though its a few years old by now: https://github.com/bepu/bepuphysics2