{"id":115,"date":"2019-03-10T22:35:38","date_gmt":"2019-03-10T22:35:38","guid":{"rendered":"https:\/\/blog.adonlon.org\/?p=115"},"modified":"2019-03-20T13:02:39","modified_gmt":"2019-03-20T13:02:39","slug":"jenkins-and-aws-ecr","status":"publish","type":"post","link":"https:\/\/blog.adonlon.org\/index.php\/2019\/03\/10\/jenkins-and-aws-ecr\/","title":{"rendered":"Jenkins, Docker and AWS ECR"},"content":{"rendered":"\n

There’s a difficulty using Jenkins to access ECR in a security conscious environment, such as an environment requiring HIPAA or PCI compliance. Even if Jenkins is on an EC2 instance, using the standard docker login command documented by AWS utilising IAM roles and the STS to get short-lived AWS credentials, the docker login command generated by the command line puts the STS token into a command parameter, visible by tools such as ps<\/code>, but it also ends up causing the default docker engine to store the short term credential in plain-text in a file on disk (~\/.docker\/credentials.json<\/code>). The docker community have ‘helpfully’ created warnings about these things, which auditors ‘helpfully’ pick up on and then ask questions about your approach to security.<\/p>\n\n\n\n

So, let’s go through the issues, why the auditors care about them and what might be the best solution to them.<\/p>\n\n\n\n

Issue 1 – The Documented Method Leaks Via Command Line<\/h2>\n\n\n\n

According to the standard AWS documentation<\/a> on using AWS ECR, the best way to log into an ECR repository is via the AWS CLI – $(aws ecr get-login --no-include-email)<\/code>. The documentation is quite clear that this results in a docker login command which includes the short lived token as a command line parameter.<\/p>\n\n\n\n

Just to be clear, aws ecr get-login --no-include-email<\/code>generates a docker login command which looks like this….<\/p>\n\n\n\n

docker login -u AWS -p eyJwYXlsb2FkIjoiaWV0ekhtdEVWblFTTXovV3NlT2w5aWE4VlMwNGxhZm40eWd6OE9mL0VpbjNURjJrZE8wc1hWeUM5SFY2aEhYSmVmb3k5VFVZT20xU2hTS1I4RFcrdzhCY0xWOTE0TndDbi9WT1IydnhQZ2VtcEtMbzRnZW1HclIvU2pSNnNlV0tlbVZURGUyaUVJMlZoSGtuZDJXY3p1V1R0bVJ0SzA5NEMweStmTVlSV0hiMklDd3hSWXJBVkdiNmdKWFJVYjZxL2pmc2RSVkdqS2JLTjNyT094djY4N3ExUm0rTll4dFIvdHQ4OHZhOWpzWnBFQjZHbGhzekJ1TVhDMHplb0lGMVdIdXB3RGtnU0hkVm5tU3hyUU96ckRRZW1iWXBNYmxqWGFqTzF2YkhsZVlNdFBSWnVXelhzbVlKbWkvd3VhVThFNS9nZmtQVll3Q1BsWG0zcXpRY2daTjQ5WjhZZXVTT3grczFZcTdydk4rdU93cWZpQ0doTVJJTVVYdERCMCtWT2U4SGl6elpuNGhpTmdva05adXpMMGNuRXRDa1BFc0VMak5acE00UzZ4QUpxcDZjM01wdW9IMmNsMEFadHlyMmN0NmY4aHM4Yjk3cGM5MGhnSGhTL3hkR0h1RnRnZFVWZ3Y4c3QxYXJJTXYwYkhuRm9xeFgzQkFIZUFzZWVDSjhjZG5Ca2JmYUYyZkxZYzRpejJuNXdOK0lTTFNWN0NjenkyeFBobUN1VG93STNqdWVYOCtFY203aXFHY3g4ZEpYbzFQbmFaQkVXZnFwTTVGMDQ1RzhkdTlBb0F5TzFGbFRCbUcxUUhHcGViNWVSVzhoa3MzTXdYTU5zNnJ2Y0tZd01uWlAvY241aklmMWpmWWVFZXVPeHdFTVFJTDVrYzhFYkVROE0vSVlBMGUyNVVvcTVVcXhaUDBjR29xL1RRRXVpRERYWDJKYW9YNGI4NFpCZlhFa3o2djFHNThHL0xyeGp2UXJNMHFmVmt0ZXRoQ2VkdXJkWUErNmhGRnhoTVFVQ0ZselArTUZOdStvMXR3YXEvekRwRjErY1RLck1DTjJtNEdCM2xESUpnNHNob2lQVEMzMkRIS21pcXdrY1N1YmdvNlAvSzc1SVcxZjBJTUYyNEZTWDZrbkh5RWpiZlNRbE1kRit2SVNHZi9YVC9sOVAxM1JkQjNJWlNQanZlVlpBdDhXbGJ2NENlWUVsM0ZiUVdhRUlNcEl1MFJweTNWNzhORldpQmZUb1oyVGl4Y3BYSmYxT3hXZkJveE84ajFPV2xDQkRLWjdRMHJwMi9ncThSTGR4dk5IdjNaVTBUWEd1YVdMemhONHlEdnNyZm5vTlMyNmRrL3RRZ1ZmUENabTRGZER1MWE5a3k0MW0zSHoydVE0cFFJSllQWnJibEdGQ3BrSER3bW5Nbk5vNnhIay9NeTFzemtmYlkrQk9wNUk2a0hWN3ZWbmRQM0dKQnpVM3FDV3dqbUs3S01SWStBdHFpM0syUkE4cDNmWVlnPT0iLCJkYXRha2V5IjoiQVFFQkFIaCtkUytCbE51ME54blh3b3diSUxzMTE1eWpkK0xOQVpoQkxac3VuT3hrM0FBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREVRTzNzY25XQ1U1aDVNRytBSUJFSUE3MXhiRlJpWUlvRzJpQjE2aE9Sa0NjbnJtRVZ6NnE5aWNiTzlOWXpaK3d5MDRQTzd3bGc0WnNVTURtdjdncVFmVmFRMk9RRWV1Zyt1WHRjOD0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE1NTEwODgwMTV9 -e none https:\/\/xxxxxxxxxxxxx.dkr.ecr.xxxxxxxxx.amazonaws.com<\/pre>\n\n\n\n
Now, I’m pretty sure nobody’s going to be using this to shoulder surf any passwords. What they *can* do, however, is to use automated tools to snaffle the token via commands like ps -ef <\/code> or its lower-level programmatic equivalents. The thing about this is that the token is no longer tied to the EC2 instance which it was generated for – anyone can use it from anywhere and act like they were the original IAM role.<\/p>\n\n\n\n
That’s possibly not the worst thing in the world if you’re working with a system where people log in with their own IAM roles from their own laptops \/ desktops. An individual users workstation or laptop isn’t *supposed* to be used by multiple people so it’s quite likely only to have the credential owner actively using the system. For the credentials to leak the system must first be compromised by some kind of malware.<\/p>\n\n\n\n
On a Jenkins CI server, however, this isn’t quite the case. The server itself is generally accessed by multiple groups of people with varying levels of ability to affect what happens on the system. Some, for sure, will have limited ability to cause things to happen but others will have more intimate access, up to and including shell access. So now you’ve got *2* mechanisms by which credentials could leak – either a malicious intruder *or* a malicious insider.<\/p>\n\n\n\n
It’s worth remembering that Jenkins exists to allow people with a significant amount of access to the machine just to do their job. A malicious developer could craft a job which runs once a minute search for ‘docker login’ commands and mails the tokens to their secret lair. A malicious operator could install a script doing similar. The Jenkins UI could be configured to lock things down but ultimately the really determined malicious developer would just write a script which does the same thing.<\/p>\n\n\n\n
Issue 2: The Documented Method Stores Credentials In Plain Text<\/h2>\n\n\n\n
So, the AWS default mechanism for logging into a remote registry has the potential to leak credentials but that’s not the only issue. The default docker installation actually stores the credentials it gets for logging into registries in a file in ~\/.docker\/configuration.json<\/code>. The file is readable only by the jenkins<\/code> user bu basically, anyone with access to this file has access to all the credentials which that user has used to access any repository (or at least any one which they have required credentials for).<\/p>\n\n\n\n
The issue here is similar to the previous one – on a single machine used by a single developer (e.g. workstation or laptop) it may not be such a big deal for credentials to be available but when that machine is shared then those credentials are potentially available to anybody with access to the machine.<\/p>\n\n\n\n
It doesn’t really matter that the credentials are only valid for 12 hours. The reality is that they’re only really usable by automated processes anyway (given their length) so the most likely attack vector involves *some* kind of automation somewhere, even if only copy-paste.<\/p>\n\n\n\n
The other thing is, Jenkins runs jobs as jobs as the jenkins user. So the credentials which, on a single laptop, are stored in file which a directly accessible only to the logged-in user are now available to a group of users on a shared instance. Now anyone with Jenkins access can acquire the permissions associated with the Jenkins user \/ role. Typically, this is quite a lot of power given that Jenkins is often given CI \/ CD responsibilities and so can have the power to push \/ pull docker images into ECR.<\/p>\n\n\n\n
The Clue Is In The Error Message<\/h2>\n\n\n\n
When you run AWS’s version of the docker login<\/code> with a password on the command line it says this to you<\/p>\n\n\n\n
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
 WARNING! Your password will be stored unencrypted in \/xxxxxxxxxxxxxx\/.docker\/config.json.
 Configure a credential helper to remove this warning. See
 https:\/\/docs.docker.com\/engine\/reference\/commandline\/login\/#credentials-store

Login Succeeded
<\/pre>\n\n\n\n
There’s a mini-wall of text there but basically it’s got two lines which are telling you that the AWS token giving you access to the ECR will be<\/p>\n\n\n\n