Context: An organization that must comply with specific strict security controls and has a formal engineering and GRC program in place to ensure compliance. Large slow lumbering release cycles, many sign-offs to ensure everything is done correctly etc.
Situation: Critical legacy system was migrated by third-party integrator to the cloud. System migration was completed and turned over to the org. The org worked closely with the third-party integrator throughout but there was a lot of dysfunctional communications. Integrator was a massive top-tier tech company everyone would recognize.
Discoveries:
Migration went through the entire governance process, numerous documents were generated and signed off for engineering and security reviews, yet there was no architecture diagram or details on what changes were made
Both engineering and security teams were ignorant of what was actually included in the system
After migration, an interface partner (in same org) worked directly with the integrator (after the contract had ended) to make changes to the source code (that was supposed to be deleted) in the integrator's dev area (that was supposed to be destroyed)
Integrator pushed these changes to the live test environment (bypassing the main dev environment completely -- all records of the code change exist solely in the former integrator's supposedly-deleted repo)
Interface partner insisted changes be pushed to prod to support their critical needs
Engineering & security had no clue what changes were made
Only one person in entire org (business manager type person) had access to the integrator's source code repo because he was a former engineer and had insisted on access months earlier
Had to get printout of code diff from him to hand carry to engineers and security to show them the change that was being proposed so they could evaluate (again, without understanding the technical architecture of the new system)
Changes were actually approved by engineering and security based solely on reading a spreadsheet summarizing at a high level what was changed (they would have used only that and trusted that everyone was telling the truth, if I hadn't insisted on getting a printout of the code to review beforehand)
Changes went live, were left in integrator's repo, not merged into baseline
When I asked system owner "where is the baseline currently maintained" he actually responded "yes (thumbs up) that is a good question, nobody knows." (it was his job to define it...)
Best part: Security & governance maturity for this system is considered "above average" by auditors and management and they are held up as an example to be followed.