There are two ways to do this. One requires a bit of foresight when merging (and is best practice) and the other involves a lot more work.

The best way to handle things like this is to actually make a new branch whenever you are merging. Since branches in git are literally just a text file with a commit hash string in them, creating a new branch is fast and cheap. Instead of merging into your target branch, create a new branch off the target branch and then merge your commits into the new branch. If you are satisfied with your changes, then you can repeat the process on the target branch and delete the test branch (or do a git reset --hard test-branch on the target branch). This allows you to check and see if a merge accomplishes what you want it to, and gives you an easy reset switch in case things go bad.

It's best practice to always use a temporary branch when doing anything that modifies a branch history unless you are absolutely sure what you're doing is correct and wanted.

If you have already made the merge using the target branch without a test-branch, then you'll have to fix the branch manually. Again, I recommend using a test-branch for this. Copy the branch and then you can use reset and cherry-pick to fix the history how-ever you want it. Once you're happy with the result on the test-branch, then you can use git reset --hard test-branch on the branch you wanted to fix, and your new history from the test-branch will be instated.

Also another note, your example is a bit off. If you have B1:(A->B->C->D) and you merge it with B2:(A->F->G->H) you should get the following result: B3:(A->B->C->D->M, A->F->G->H->M). Merge commits are special because they actually have two parents. This is why git is described as a unidirectional non-cyclic graph rather than a tree. So, undoing the merge is actually as easy as doing a git reset --hard $commit where $commit is the commit hash right before the merge commit.

Hope that answers all your questions.