I have to second the mention of Embree. Intel poured several man years of optimization into this code. As long as you are fine with being restricted to x86 platforms, you won't find faster open source code for general ray/triangle intersection.

Unreal Engine 4 uses Embree in its lightmass static lighting builds. They got a big speedup from moving to it.

I want to make simple FDTD mesher, maybe with non rectangular grid. For this I would need ray-object intersections and point to object shortest distance. Thanks for pointing out Embree, I was not familiar with that library.