A clearance designates you a credible person in the eyes of the government to view classified information, but the clearance itself doesn't grant you access to anything. To your example, you lose access to your company's files, but you have the exclusive right to reapply for a job to see them again if rehired. Others without a clearance would not even have the chance to apply since they lack the proper credentials. The government agency or the gov't-approved contractor you work for is the one that grants you the access to the information within your job duties because of your "need to know". Overall, this system allows contractors to work on classified projects, for example Boeing, without being government officials and for gov't employees to have mobility within different agencies or to transition to the private contracting sector.

Unless Brennen was working recently for a contractor, once he left the CIA he lost access to their information. That's typical of most employees, but Brennen being former director and chummy with high-ups within the org, who knows. But never mind access, by revoking his clearance he lost the trust of the US government.