At least three fashion retailers are leaving Towson Town Center, with one cleared out and two on their way out the door.