Prior to 1947, Sylhet, a Bengali speaking, Bengali majority district was a part of Assam. It used to be a part of East Bengal, but the British decided that they needed educated and trained people to rule Assam after its annexation and hence, decided to make Sylhet a part of Assam.

Sylhet has oscillated between Assam and East Bengal several times. People were given jobs in Assam and they started to occupy the Brahmaputra Valley. Many people settled there. This was before independence. They were and still are Bengali, but are the residents of Assam. Assam was never monolingual.

Now, during partition, Sylhet went with Bangladesh. As the partition was based on religion, Hindus left their home and came to Assam. Some people had to shift just 20km to be on the other side of the border. This is the amount of proximity Assam shares with Bangladesh.

The story is different in Barak Valley. Barak Valley has a completely different history. It was never Assamese. It was a part of the Tripua kingdom and then Cachari kingdom. The Karimganj district was a part of Sylhet. In fact, Karimganj district was included in India so that Tripura would not get landlocked.

So essentially, Bengalis didn't move to Assam. They were first brought to Assam by British for administrative purpose and stayed back and later, after Sylhet referendum, they moved to the Indian part of the state. Honestly, if they knew Assamese would harbour such anti-Bengali sentiments, they would have moved to West Bengal.