An environment got stuck in some bad state. First, publishing was no longer possible. Then, after deleting the environment from admin center the environment was still there but Delete button was not available.
Dropping to console, issuing
pac admin delete --environment <your-guid-here>
and waiting a few minutes helped.