Gnu CL also had problems as a Common Lisp implementation. It works (like ECL, which is related) by compiling to C, which is then compiled with a C compiler and the object code loaded into the image. This approach generates inferior code to that which an implementation like SBCL can achieve.
All the Lisp implementations needed more memory than C programs would, which limited their impact before 32 bit machines became standard. And by that time, C was strongly in place.
There's arguably also the part where it was, AFAIK, mainly driven by needs of Maxima and it's declaration (or so I recall) of focusing on CLtL first didn't help things either.
In comparison, ECL which forked from the same family seems to work out much better over time.
All the Lisp implementations needed more memory than C programs would, which limited their impact before 32 bit machines became standard. And by that time, C was strongly in place.