The apostrophe exists for a reason. It doesn't exist just for show, to place on a page indiscriminately just because it looks good there, or just because one feels like it. For example, it's used to replace letters, such as in the term "rock 'n' roll". The apostrophes in this term have replaced the "a" and the "d" in the word "and". It's not "rock 'n roll" or "rock n' roll", although many write it those ways. Another example is regarding the word "little". You can write it in colloquial short form as Lil or LIL, but if you add an apostrophe and write 'Lil, that's just wrong. Not that anyone cares, but as a career ESL teacher I notice these things, get annoyed by them, and sometimes comment on them on social media forums. If even one person benefits from my Grammar Nazi or Punctuation Nazi rants, that's good enough for me.
Write well, appear smart. Appearance matters.