When getting back with someone I think the important things to consider would be: why did we break up in the first place? Is this issue(s) resolved? What were some problems we had in the relationship? Are these resolved? Do we have a way of resolving them? If no then they will likely persist. Put a bit more emphasis on what the issues were and bit less in the memories.

What are your boundaries concerning him seeing others while seeing you? If you are not ok with this (which it understandably seems like you aren't) then I would suggest to stick to your boundaries. If you give in despite your hesitation then it can lead to resentment later on.