Ignore the threats he has a contract he needs to fulfill.

The Citizens Advice Buru has a series of letter templates that can be used. Basically you need to write down your complaint and give him a reasonable timeframe to complete the job to an acceptable standard. Then get quotes from others to rectify the work. Give hm another deadline with an estimate of the cost you will claim from him for breach of contract before terminating the contract with him.

You may not be able to recover this money but it shows you were reasonable in terminating the contractor and gave him ample opportunity to rectify the problems. Again Citezens Advice Buru has an excellent templates and guides for dealing with this.

Source: home owner who has had issues with buyers in the past.