Can I turn drift concerns off after they have been created in OpsCompass?

Yes, you can disable/enable concerns at any time if you want to limit the noise for a certain alert at a given time.

In the alert concerns tab you have the option to disable/enable any already configured concerns in OpsCompass.