As an Ops Manager, we always try to push the activities towards bots or Service accounts so that they can system can continiously work. Same goes true for Cloud accounts where we create Service accounts in case we need such usecases. Whenever external account access is required create a json key and go with it. We all have been there , and it all works fine from a small scale angle but here is when it gets really interesting
Assume you are managing operations for multiple projects and pleotra of service accounts and Cloud Security teams set up a policy to rotate the keys every one month.
BOOM!! Goes the Dynamite
Not sure if you know what hit you !! let me put it in prespective ; Your org has around 30–40 projects under your pervue and each had uncontroled service account creation having upto 5–6 json keys per account in each of the Dev/NonProd/Prod Enviornments and you are left standing on top of all this.
If you still did not get it !! well God speed my friend 😬
Well lets keep all the prejudices aside and lets device a plan of action !!
First things first , after you have tried whining about the security teams that have noting contructive to do and sending emails to your leaders about the work load increase and already have got responses like “I understand … <insert blah blah text> but it is for security and we have to do that”
Set up a baseline run on lower enviornments : This is because you might not know what is already broken in lower enviornments !!!
Check the Service account usage logs : To validate if accounts are been used at all
If you dont think the account is being used , disable the key : Go through this excellent article here
gcloud iam service-accounts keys disable key-id \
--iam-account=sa-name@project-id.iam.gserviceaccount.comRun a post baseline to check lower enviornment for any issues
For a better future set up automatic key rotation
Now that We can rotate the keys, we could think of a more optimal solution. One way to reduce and optimise the keys is to see if we can reduce to one Service account per application and utilising the process described here
That is one way to do things. Another right wing approach is here (Looks like the auther will punish you if you downloaded keys !! ) applicable in cases where end users use the service accounts, which is acually incorrect and they should be using impersonation.
Well we went though multiple aspects but for the impatient here is a gist
- Always reduce your surface when it comes to service accounts as you dont know went it is going to hit you
- Take a planned approach to disable keys and accounts to check impacts
- Try to to see if we can use common service account in multiple production leveraging that GCP-GCP interaction will not require json files
- See if impersonation can solve problems that you are using Service accounts
Too much of service accounts, here is a baby image to get your sanity back !!!
