I've never thought it was a good idea take away the right to challenge the governments ability to hold anyone indefinitely.

No kidding. If you can be held indefinitely without a judicial hearing, your other rights aren't really worth much. Isn't this why we had the American Revolution?