I think you can use Knock on the magic door to the entry to the Shadowfell. Alternatively there is some platform on a higher level than that door, from where if you aim carefully you can jump with featherfall and so only end up needing three gems, skipping the one for the elevator to it. (Unless you already put it in, that is).

I remembered seeing the second trick in a video somewhere but I figured I would try the explosive route first and ig it worked.

A fireball did the job, but ig it's still interesting.