Definitions of downside:

• noun:   a negative aspect of something that is generally positive

  Example: "There is a downside even to motherhood"