I think there was a revolt and the people of Albion destroyed both temples. If you go to the Temple of Light and press A on the statue with the hammer. There is a statement that talks about the hammer being held by the statue. I think the hammer was used in the destruction of the Temple of Avo.