There are many choices to consider here depending on what kind of things you are looking for and I'll name quite a few books here that are both readable and will give you an accurate sense of what modern historians think about the Roman Empire. I also encourage you to think about the empire as an entity that lasted well into the middle ages - I hope you don't mind some of my later recommendations!

To get a general sense of the empire, try The Romans: From Village to Empire (2012). If you are interested in military affairs, I recommend Adrian Goldsworthy's The Complete Roman Army (2003) for, well, a complete manual on what the Roman army was all about. His volumes on the Punic Wars, Julius Caesar, Antony and Cleopatra, and Augustus are all excellent too and are all very readable introductions to the complex world of Roman politics. I have to caution you against reading Tom Holland's Rubicon (2003) here, as despite its popularity it was not written by an academic historian nor written in a way that meshes well with modern academics' conclusions. To get a sense of historians' views on the empire in the fourth and fifth centuries, consider Goldsworthy's The Fall of the West (2009) and Peter Heather's The Fall of the Roman Empire (2005).

Imperial Rome however lived on after 476! The sequel to Heather's book, The Restoration of Rome (2014), is an excellent overview of what happened after Rome 'fell', which is rather important since the Roman Empire lived on in the east and the post-Roman world in the west was never as bad as what past historians, such as Gibbon, told us. Late antiquity really wasn't as cataclysmic as some people thought it was and this was really down to the amazing work done by Peter Brown, particularly in his The World of Late Antiquity (1971), which dealt with cultural history from the third century to the seventh, so I heartily recommend that as well - it literally transformed our understanding of the Late Roman Empire and is very well-written. If you are interested in Christianity as well, which cannot be separated from any discussion of the Roman Empire from the fourth century onwards, Brown's The Rise of Western Christendom, 200-1000 (2002) is the book to read. Of course, late antiquity was also an age of violence and material deprivation despite its cultural and political continuity with the earlier Roman period. This is best encapsulated in Bryan Ward-Perkins' The Fall of Rome and the End of Civilization (2006).

For the Byzantine Empire (though it really was just the Eastern Roman Empire, historians just use it to distinguish between two phases of the same empire), consider Treadgold's A History of Byzantine State and Society (1997) and Norwich's A Short History of Byzantium (1997), though both are a bit too old for my taste. For a more modern approach, I heartily recommend Judith Herrin's Byzantium: The Surprising Life of a Medieval Empire (2007), as she examined the empire through various themes rather than chronologically, and explored fields often unexplored in popular histories of the Roman world, such as on the role of women, art and the average citizen. For more specific treatments of certain periods, Mark Whittow's The Making of Orthodox Byzantium, 600-1025 (1996) and Peter Frankopan's The First Crusade: the Call from the East (2012) are both very accessible and good overviews of the specific periods they cover.