Another impossible to debug AWS issue:

I ran an AWS Systems Manager Patch Manager RunPatchBaseline association on a managed instance. My managed instance uses a custom IAM Role slash Instance Profile, but I read the documentation and added the proper permissions.

I ran RunPatchBaseline and got an Association Failed status on the instance with the custom role. The Run Command Output showed the following error:

No IMDS credentials found on instance.failed to run commands: exit status 156

I connected to the instance and verified that IMDS does work fine on the instance, following the AWS documentation here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

Further details under the Run Command Output showed a completely different error:

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Further up in the logs I found that the SSM agent was trying to download the baseline_overrides.json from a dedicated S3 bucket operated by Amazon, starting with aws-quicksetup-patchpolicy-ACCOUNTID-QUICKSETUPCONFIGID. My IAM profile, even though it is setup to allow access to this bucket, and can list the baseline_overrides.json object, is blocked from downloading it.

Solution: by comparing the custom role to the Amazon generated QuickSetup role, I figured out that I have to manually add a Tag to the custom IAM role for this to work. S3 is checking if the ROLE has a special Tag associated before allowing access to the object. It is documented on AWS here: https://docs.aws.amazon.com/systems-manager/latest/userguide/quick-setup-patch-manager.html

“You must tag your IAM instance profile or IAM service role with the following key-value pair.
Key: QSConfigId-quick-setup-configuration-id, Value: quick-setup-configuration-id”

Hope This Helps,

imre Fitos