Low: controller: don't retry an invalid remote key

We already skip retries if the key is missing. There's also no point in
retrying an invalid key.