proper use of conjunctions

It seems to me that our grammar teachers of yesteryear always stressed the fact that sentences cannot begin with a conjunction, yet the use of such is becoming the norm.
As an example, the sentence,”We had fish for supper. And, it was delicious.” In order to be grammatically correct, shouldn’t it read, “We had fish for supper, and it was delicious?”