well you could always just fast travel there go to bowerstone, get some people, go to oakfield ) fast travel) and get some more, go to the Temple (fast travel) It works. The fast travelling with followers. For me at least.

---