I have NEVER heard any jokes about adopted people.
There are bigger problems in your life then being made fun of for being adopted, fix those first.(I know there are... Being made fun of is one of the least of problems) Learn it form someone experienced.
Your fine, and let us get back to it. You are the one creating the problems. Don't like it, ask him to remove it(politely), and move on. If he keeps it, ignore it. I have done that for SOOO many years.