This is literally a whole "The Good Place" bit.
Over a year ago I met an Anthropic employee at a conference and at one point she mentioned that one of the things she was working on was whether or not it was cruel to ever turn off an AI model. I thought she was joking. She was not.