After updating the wireless-driver to a more recent version (12/31/2008, I had 12/30/2008 before!) and the kernel panics are gone. Also SSH still works after 2h. Yippie!