Container Security - 20 Questions for Securing Docker and Kubernetes

08/05/2023 :: Jeremy Pickett :: Become a Patron :: Buy Me a Coffee (small tip) :: @jeremy_pickett :: Discussion (FB)

Protecting containerized applications and infrastructure from attacks, protecting infrastructure from containers and microservices.

TLDR: The initial enthusiasm around containers led to major security gaps as adoption outpaced proper controls and best practices. Containers were often configured with excessive privileges, built from vulnerable base images, and lacked isolation between untrusted workloads. Immature orchestration platforms like Kubernetes also lacked hardened configurations. This left container environments open to exploits, lateral movement, supply chain attacks, and insider threats.

Securing containers requires using minimal read-only images, automating security scanning, managing vulnerabilities aggressively, restricting runtime privileges, and segmenting network traffic. Kubernetes has matured with role-based access control, network policies, and pod security standards to harden deployments. Monitoring runtime activity and implementing centralized security visibility across cloud environments is also critical. With proper controls, processes, and expertise, organizations can benefit from the agility of containers while ensuring risk is managed.

Introduction

The rise of Docker and Kubernetes has revolutionized application development and deployment, enabling portability and scalability through containerized microservices. However, this transformation happened so quickly that security was often an afterthought. The enthusiasm around containers led many organizations to rush adoption without implementing proper controls and practices to secure these environments.

This essay will examine the unique risks posed by containers, best practices for hardening Docker and Kubernetes deployments, and how to build security into DevOps workflows for container-based applications. We’ll also look at ways containers may enable organizations to ignore fundamentally unsafe practices if proper care is not taken.

What risks resulted from initial enthusiasm about containers?

In the early days of Docker and Kubernetes, security was not a major consideration for many adopters. The focus was on rapid application development, portability, and scalability. This resulted in several risks:

Vulnerable base images - Docker images were often built from stock operating system images containing many packages and vulnerabilities. Images were also infrequently rebuilt to apply patches.

Here are three of the most commonly exploited base images found in vulnerable container deployments:

- Ubuntu 18.04 - Ubuntu 18.04 LTS contains many installed packages and kernel versions with frequent vulnerabilities. Being LTS, it is widely used as a base image but often unpatched.

- Debian 9/10 - Like Ubuntu, Debian stable releases get outdated but continue being used in containers. Vulnerabilities in Debian 9 and 10 are frequent targets.

- CentOS 7 - CentOS 7 was supported until late 2020 but remains in widespread use. The end of life status makes it easy to exploit known issues that will not be patched.

Some specific vulnerabilities frequently targeted in these images include dirty COW privilege escalation in older Ubuntu kernels, vulnerabilities in OpenSSL affecting Debian 9/10, and log4j issues impacting apps running on top of outdated CentOS 7 deployments.

Organizations often pull these familiar distros without understanding the security implications of running outdated and vulnerable versions. Regularly rebuilding using latest patched versions of slim distros like Alpine or Distroless Linux can avoid many common exploits affecting these base images.

Overprivileged containers - Containers often run as root or privileged without restrictions on capabilities, resources, filesystem access etc. This results in a large attack surface.

Common COTS containers/images believed to be overprivileged include:

MySQL - Official MySQL images run as root by default. This provides unnecessary privileges, allowing a compromise of MySQL to gain full root access.

Elasticsearch - Elastic Docker images also run Elasticsearch as root. Again, this grants unnecessary privileges that could be abused by an attacker.

Nginx - Popular Nginx images commonly run as root as well, rather than creating a less privileged nginx user. This follows a pattern of containers carrying more privileges than required.

MongoDB - In addition to running as root, the official MongoDB image exposes MongoDB on all network interfaces by default, increasing exposure.

Jenkins - Jenkins containers built from official images run Jenkins with the privileged flag enabled, allowing access to host devices and more.

In general, many official vendor images grant more privileges than necessary to simplify deployment. But this leaves them open to privilege escalation attacks. Following the principle of least privilege by creating unprivileged users, limiting capabilities, and securing network exposure is important.

Lack of segmentation and isolation - Containers running on the same hosts were not properly isolated from each other using namespaces, AppArmor, SELinux, seccomp. Laterally movement between containers could occur.
Weak access controls - Kubernetes role-based access control (RBAC), network policies, and pod security policies were not implemented by default, allowing unnecessary access between containers and services.

Here are some best practices for implementing RBAC in Kubernetes for managing access to Docker containers:

Use Kubernetes namespaces to logically separate applications and teams. Assign containers to namespaces aligned with access control requirements.
Leverage Kubernetes RBAC to create roles that grant only the necessary permissions containers need to function. Avoid overly permissive roles.
For example, create a role that only allows containers to list, get, and update ConfigMaps needed for configuration.
Bind the roles to subjects like users, groups, or service accounts using role bindings scoped to namespaces. Avoid cluster-wide bindings.
Use Kubernetes NetworkPolicies to restrict network traffic between namespaces and pod segments based on identities and approved communication requirements.
For sensitive applications, create specific service accounts for pods to use instead of the default service account.
Configure the Kubernetes API server to only accept TLS certificates from approved CAs to authenticate human users and automation.
Use admission webhooks to dynamically validate Kubernetes objects meet security policies upon creation and updates.
Leverage pod security policies to enforce requirements like preventing privileged containers, limiting volumes, and restricting capabilities.

The key is granting least privilege via RBAC, namespaces, and network controls based on principals and approval workflows. Audit logs should be enabled to detect excessive access attempts.

Unpatched hosts - The underlying hosts running containers were not always hardened and kept up-to-date, providing a path to compromise containers.

CI/CD pipeline security - Automated build pipelines lacked basic practices like minimizing build tools, protecting secret management, isolating builds, scanning images.

This enthusiasm also resulted in a culture of ignoring security basics.

For example, secrets were often baked into container images instead of injected at runtime. Dangerous container capabilities like mounting host filesystems went unused.

Overall, the rate of container adoption outpaced the implementation of controls and best practices to secure these environments. This resulted in broad attack surfaces filled with vulnerabilities and misconfigurations for attackers to target.

How is orchestration hardened and isolation ensured?

Kubernetes has developed rapidly to provide better orchestration security:

- RBAC and network policies restrict communication between pods based on identity and namespaces. Pods can be isolated from each other.

- Security contexts and pod security policies allow enforcing configuration standards, like requiring read-only root filesystems.

- Secrets are stored securely outside pods and injected at runtime to prevent disclosure.

- TLS for Kubernetes API communication is enabled by default. API requests are authenticated.

- Kubernetes versions are tracked and nodes auto-update to stay current.

- Tools like Falco provide runtime behavioral monitoring to detect anomalous container activities.

At the container level, Docker and OCI standards now integrate security:

- Seccomp, AppArmor, and SELinux profiles can restrict container actions like syscalls.

- User namespaces provide additional isolation between containers and hosts.

- Read-only volumes, capabilities dropping, and reduced privileges improve runtime security.

- Container healthchecks monitor for failing containers needing restart.

Hardening Linux host machines running containers is also critical:

- Minimize unnecessary packages, running services, open ports to reduce attack surface.

- Harden the kernel with options like seccomp filtering.

- Use firewalls to limit network access to containers.

- Apply OS updates frequently and use immutable infrastructure where possible.

Do containers incentivize ignoring fundamentally unsafe practices?

The elasticity and portability of containers can unfortunately enable organizations to overlook insecure practices:

- Building from vulnerable base images without scanning or rebuilding frequently. Containers are seen as ephemeral.

- Granting excessive privileges to containers at runtime due to fast pace of development.

- Operating complex Kubernetes environments without controls like network segmentation between apps.

- Assuming compromised containers will fail fast and be replaced rapidly. Lack of runtime protections.

- Focusing too much on CI/CD pipeline speed over implementing pipeline security and quality checks.

These fundamentally unsafe practices stem from treating containers as disposable or short-lived. But modern applications still require hardening containers against attack and monitoring runtime activity.

Operations teams should carefully evaluate if containers are encouraging bad practices like lack of patching, weak runtime security, and poor secrets management. Processes and culture ultimately determine if containers improve resilience or introduce risks.

With this background on the risks posed by containers and areas for hardening, we will now dive into 20 key questions engineers should be asking when securing container environments.

Twenty questions to secure your containers

1. How are base container images being selected? Are minimal images like Distroless or Alpine being used where possible?

The base image used to build containers plays a critical role in their security posture. Images like Distroless and Alpine Linux provide a minimal attack surface by including only essential packages needed to run the application. However, many organizations still use images based on vulnerable Linux distributions without understanding the risks.

Popular base images like Ubuntu, Debian, and CentOS contain hundreds of unnecessary packages that increase the attack surface for containers. For example, Ubuntu 18.04 images include system utils, openssh, even an entire apt package manager that are not needed in most containers. These packages contain vulnerabilities that can be exploited to compromise containers. Images also tend to be based on outdated releases that are past end of life and no longer receive security patches.

Organizations should carefully evaluate base images to determine the minimum set of packages actually required for their containers to function. In most cases, stripped down images like Distroless, Alpine, or Red Hat Universal Base Images (UBI) will suffice and avoid many vulnerabilities present in general-purpose Linux distributions. Rebuilding container images frequently from the latest patched versions of selected base images is also recommended to incorporate security fixes. The best practice is to always use "skinny" minimal images that only include essential packages for the application.

2. What scanning tools are in place to check for vulnerabilities in images during the build process? Are scans incremental after each layer?

Scanning container images within CI/CD pipelines is essential to catch vulnerabilities early before vulnerable images reach production. Docker's build process of using a layered filesystem means it's important to scan after each step rather than just the final image.

Tools like Anchore Engine, Trivy, and Snyk can be integrated into CI/CD systems to automatically scan images after each layer is built in the Dockerfile. This incremental scanning catches issues like a vulnerable library being introduced so the build can fail fast before additional layers potentially obscure or magnify the problem. Scanning only the final image may miss vulnerabilities added earlier in the build process.

Organizations should research tools like Anchore, Trivy, and Snyk to find one that best fits their needs and integrates with their CI/CD ecosystem. Most importantly, the scanning solution should provide high fidelity vulnerability data and the ability to fail Docker builds when a vulnerability exceeds the risk threshold. Implementing incremental scanning after each layer is also strongly recommended as a best practice for early detection compared to scanning only completed images.

3. How frequently are container images scanned for newly discovered vulnerabilities after deployment? Is scanning automated?

Scanning container images pre-deployment is not sufficient by itself. Regular scanning is also needed post-deployment to check for newly discovered vulnerabilities in dependencies. Images should be scanned at least weekly after being deployed into production.

New vulnerabilities are disclosed constantly, meaning the security of container images degrades over time after deployment. Automating scheduled scanning of registries and production container hosts is critical to detecting these new threats. Relying on manual scanning processes is inconsistent compared to automated solutions.

Organizations should implement automated scanning tools like Trivy, Anchore, and Snyk that can run on registries or directly on hosts to regularly check images. Scan reports should integrate with ticket systems to alert on vulnerabilities based on severity levels so they can be quickly investigated and patched. Consistent automated scanning of production container images is key to minimizing the timeframe threats go undetected after emerging.

4. How are vulnerabilities prioritized for patching? Is there a remediation SLA based on severity?

When vulnerabilities are discovered in container images, organizations need a way to consistently prioritize which ones get fixed first based on severity and potential impact. Setting patching SLAs provides this structure. This may be for a remediation, patch, acceptance of the vulnerability, or explanation of why this is a false positive.

For example, critical severity vulnerabilities like remote code executions could be required to patch across all affected images within 48 hours of discovery. High severity issues like privilege escalations may have a 1 week SLA, while medium vulnerabilities get 30 days. Low severity problems with no exploit may have longer periods.

Having defined SLAs based on severity gets organizations in the mindset of taking action quickly on critical container vulnerabilities versus leaving issues open ended. However, SLAs should also consider threat intelligence on exploit likelihood and base images used. Overall though, severity-driven patch SLAs force accountability to fix the most dangerous issues within compressed timeframes.

5. What controls limit lateral movement between containers if one is compromised at runtime? Are namespaces, AppArmor profiles etc. applied?

Preventing lateral movement between containers running on the same hosts is critical to avoid widespread compromise if one container is breached. Linux namespaces, Mandatory Access Control (MAC) systems like AppArmor and SELinux, and seccomp profiles provide controls to limit lateral movement.

Namespaces isolate containers from each other by providing separate pid, net, ipc, and other namespaces. This prevents processes in one container from interacting with other containers. MAC systems like AppArmor confine container actions and prevent tampering with other containers. Seccomp enforces system call policies, restricting activities containers can perform if compromised.

Organizations should develop policies defining required namespace, MAC, and seccomp configurations to isolate containers at runtime based on risk levels. These access controls should be regularly audited to prevent misconfigurations enabling unwanted container interactions. The principle of least privilege should apply - containers should only be able to access resources and interfaces explicitly required for functioning.

6. How is least privilege being enforced for containers at runtime? Do they have reduced capabilities, read-only volumes etc.?

The principle of least privilege is critical to limit damage if a container is compromised at runtime. Containers should execute with the minimum set of Linux capabilities, mounts, and privileges needed to function.

Capabilities like SYS_ADMIN allow dangerous actions like mounting filesystems and should be dropped. Containers should run as non-root users where possible. Filesystems should be read-only wherever feasible to prevent malicious writes. Tools like SELinux and Seccomp add further restrictions on activities.

Organizations should have defined standards for container capabilities, mounts, user contexts, and restrictions based on risk levels and app requirements. These should be consistently enforced at runtime. Audit checks should validate running containers comply with least privilege policies. The more privileges granted, the more vulnerabilities become exploitable - so implementing true least functionality is ideal.

7. Are host protections like SELinux and seccomp being used to limit damage if a container escapes?

Hardening the hosts running containers is critical to limit damage if a container escapes isolation or a vulnerability is exploited to breakout. Linux security modules like SELinux and Seccomp enable policies that restrict what actions an escaped container can perform on the host.

SELinux provides Mandatory Access Controls that confine processes to only permitted actions, even a compromised root process. Seccomp enforces system call filtering to prevent dangerous syscalls like disabling SELinux. These provide defense-in-depth protections for hosts.

Organizations should develop SELinux and Seccomp policies tailored to container hosts that enforce strict rules on what a container can access if compromised. Policies will vary based on app needs but the goal is limiting damage through least privilege. Protections should be regularly audited and policies kept up to date as new attack methods emerge. Hardened hosts are the last line securing compromised containers.

8. How are secrets like passwords, API keys, certificates handled? Are they injected at runtime or built into images?

Secrets like passwords, API keys, and certificates should never be included in container images or source code. Instead, they should be stored securely in secrets management solutions like HashiCorp Vault and injected into containers at runtime.

Building secrets into images risks disclosure through leaked images, source code repos, or reverse engineering. Runtime injection ensures secrets stay protected in the vault and only load temporarily into ephemeral containers. Developers do not need direct access to view secrets.

Organizations should implement centralized secrets management vaults that programmatically inject secrets into containers rather than manual processes. Vaults and injection tools should use TLS encryption and support frequently rotating secrets. Following the practice of injecting secrets at runtime rather than during builds improves security and reduces risk.

9. Is container runtime activity being monitored for anomalous behavior that could indicate compromise?

Detecting anomalous behavior and activity that could indicate compromised containers is critical for response. Monitoring runtime traffic, mounts, privilege escalations, network connections, and actions enables identifying potentially malicious activities.

Tools like Sysdig Falco and Datadog provide Kubernetes and container monitoring to baseline “normal” behavior and detect outliers that could signify an attack, such as sudden crypto mining, unexpected outbound connections, disabled logging, or mounting non-standard volumes.

Organizations should deploy runtime monitoring tools across container hosts and integrate them with SIEMs for analysis and alerts. Monitored metrics include process monitoring, file integrity checking, network traffic, sysdig events, and user actions. Multi-layer runtime monitoring is key to early threat detection.

10. What network security controls are in place for traffic between containers and services? Segmentation? Firewalls? Traffic encryption?

Communications between containers and services should be restricted and encrypted to limit lateral movement if a container is compromised. Segmentation, firewalls, and network traffic encryption provide these controls.

Segment networks with Tools like Calico to enforce traffic policies between containers based on attributes like namespace, label, and IP address. This limits communication to only explicitly allowed flows. Deploy firewalls to restrict inbound traffic from containers to only required ports and protocols.

Require TLS encryption between containers and services so all traffic is encrypted rather than plain text. Terminate TLS at edge load balancers rather than individual containers. Certificate management solutions help automate TLS across containers.

Organizations should define network security policies tailored to container environments based on zero trust principles. Strict controls on traffic, restricted segmentation between applications, and encrypted flows limit exposure and prevent lateral movement.

11. How quickly can containers be rebuilt and redeployed in case of an incident requiring rollback or patching?

Having automated container build and deployment pipelines is critical to enable quickly rebuilding containers without vulnerabilities in the event of an incident requiring rollback or patching.

Reliance on manual processes results in slow, unreliable container rebuilding. Automated pipelines allow triggering rapid rebuilds of compromised containers from the baseline secure image. Integrating security scanning into pipelines also lets quickly iterate and validate rebuilt images are secure.

Organizations should invest in automated build and deployment tooling such as Jenkins, Spinnaker, and CircleCI that integrates with container registries and Kubernetes. Test automation also speeds validating patched containers. Prioritizing pipeline automation and security tool integration enables consistently rebuilding containers fast after incidents.

Having automated build pipelines allows quickly rebuilding containers without vulnerabilities if an incident occurs. Test automation helps verify patched containers.

12. Are Kubernetes pod security policies being used to enforce good configurations?

Kubernetes pod security policies allow enforcing standards and security settings on pods at time of creation. Policies can mandate things like required namespaces, user IDs, filesystem permissions, and restrict usage of host resources.

For example, require pods run with a non-root UID above 10000. Set read-only filesystems on pods to prevent malicious writes. Limit volume mount usage to approved paths. Disallow privilege escalation. Pods violating policies are not permitted to start.

Organizations should develop organization-wide pod security standards that pods must adhere to, tailored to app needs and risk levels. Enforcing these via pod security policies provides consistency and avoids insecure pod configurations. Policies should be regularly audited and updated as new threats emerge.

Pod security policies can require containers run with reduced capabilities, assigned security profiles, user and group IDs, and read only root filesystems.

13. What controls exist for managing and auditing changes to Kubernetes deployments over time? RBAC? GitOps?

Kubernetes enables rapid changes to container deployments, requiring controls for change management and auditing. Kubernetes RBAC, namespaces, and approaches like GitOps help provide governance.

RBAC assigns limited roles to users and systems interacting with the Kubernetes API to restrict changes to only authorized containers and resources. Namespaces also isolate teams and environments.

GitOps tools like Flux leverage Git to declare desired infrastructure states. Changes are reviewed as pull requests rather than directly applying. Infrastructure as code and policy as code provide oversight into changes.

Auditing logs should record all API activity and changes made over time. Organizations should leverage RBAC, namespaces, policy/code repositories, and audit logs to govern changes to Kubernetes deployments based on principles of least privilege and separation of duties.

14. How is CI/CD pipeline security configured? Are build tools/pipelines hardened and minimized?

Hardening and minimizing build tools and permissions is critical for CI/CD pipeline security. Pipelines should only have access to necessary resources for building containers to limit damage if compromised.

Build tools like Jenkins, CircleCI, and GitLab CI should run with least privileges needed rather than default privileged options. Only approved plugins that are kept updated should be installed. Infrastructure used during builds should be ephemeral if possible.

Secrets should be tightly scoped to pipelines and masked in logs. Webhooks for triggering builds require authentication. Network traffic between build tools should be encrypted and limited to required connections.

Organizations should perform an assessment of build pipeline permissions, data access, and network flows to determine the minimum set required. Security profiles specific to build tools that enforce least privileges help reduce risk. Hardened, ephemeral pipelines limit exposure.

15. Are static and dynamic application security testing conducted against containerized apps and services?

Verifying containerized application security requires combining static analysis (SAST), dynamic analysis (DAST), interactive analysis (IAST), and runtime protections (RASP).

SAST scans source code for vulnerabilities at rest. DAST tests the running app for issues. IAST instruments apps to detect issues in production. RASP provides runtime monitoring and blocking.

Tools like Snyk Code and Contrast provide SAST for container apps. DAST tools like OWASP ZAP scan containerized apps. Tools like Sysdig Falco add RASP protections. An integrated approach is ideal.

Organizations should implement SAST early in CI/CD pipelines to uncover code issues. DAST, IAST, and RASP should scan containerized apps pre-production, and provide runtime monitoring. Testing throughout the lifecycle improves identification and protection.

16. What types of container runtime attacks are tested for during security assessments?

Security assessments of container deployments should include testing exploitation of runtime vulnerabilities and misconfigurations that could allow container escapes, privilege escalation, denial of service, and supply chain attacks.

Testing often involves injecting malicious containers, exploiting unpatched hosts, targeting orchestration platforms, and attempting unauthorized access between containers. Assessments may also dynamically manipulate images and inject flaws into trusted pipelines.

Testing methods include fuzzing interfaces, reverse engineering images, orchestration weakness probing, and attacking from within compromised containers. The goal is demonstrating realistic methods attackers could use to identify and exploit container environments.

Organizations should ensure security assessments cover the spectrum of possible attacks targeting container infrastructure, applications, pipelines, and runtime environments. As assessments evolve to mimic real-world attacks, organizations can better understand and close gaps.

17. How are vulnerabilities managed across hybrid/multi-cloud environments running shared container images?

When container images are shared across multiple on-prem and cloud environments, centralized scanning and monitoring is essential to provide consistent vulnerability management.

Using a container registry security tool like Anchore or Trivy to scan images from a central hub provides visibility across environments. Cloud native tools like Twistlock also analyze running images across platforms.

Findings should feed into a shared ticketing system to track and prioritize vulnerabilities across hybrid/multi-cloud deployments so they are patched consistently. Patching should aim for "ship once, run anywhere" across environments.

Organizations should implement centralized container scanning tools and unified tracking systems usable across all environments. Shared images require shared visibility, reporting, and remediation workflows across on-prem, public cloud, and edge to standardize vulnerability management.

18. How are container hosts hardened? Minimal packages/processes? Firewalls? Kernel lockdown?

Hardening the underlying hosts running containers is critical to reduce the attack surface. This involves minimizing unnecessary packages, processes, open ports, and enabling security configurations.

Only essential packages should be installed on hosts. Unneeded bundled apps, utilities, languages, and tools should be removed. Processes should be limited to container runtimes and agents. Enable firewalls to restrict network access to required ports.

Enable kernel lockdown to restrict access to kernel APIs and parameters. Set secure sysctl options like disabling IPv6 if not needed. Utilize Linux security modules like SELinux for defense-in-depth. Restrict root access to administrative users only.

Organizations should develop standard hardened configuration profiles for container hosts based on app needs and security best practices. Profiles should be regularly reviewed and updated. Hardened hosts provide a critical foundation for running secure containers.

Running only required packages and processes, firewalling each host, enabling kernel lockdown, and regularly patching hosts reduces the container attack surface.

19. What controls exist for container image provenance? Signing? Immutability? Can images be tampered with?

Ensuring the integrity and provenance of container images is important to prevent tampering, hijacking, or injecting malicious images across their lifecycle. Signing images, immutable tags, and blockchain-based auditing provide controls.

Digital signing of images with GPG keys allows validating images came from a trusted source and have not been modified in transit. Immutable tags prevent images from being overwritten or altered after creation.

Blockchain-based auditing tracks image provenance across registries and environments to detect tampering and ensure validity of images deployed into production.

Organizations should implement signing of images in pipelines, immutable tags on repositories, and blockchain-based image auditing. Integrating these controls provides confidence in container image integrity through development, distribution, and production deployment.

20. Is there a container security incident response plan for handling compromises? How will forensic data be collected?

Organizations should develop and test incident response plans for container security compromises including how to collect forensic data. Plans help guide response to scenarios like poisoned images, host escapes, and orchestrator breaches.

Playbooks should outline steps for gathering forensic data like container runtime logs, Kubernetes audit logs, network flows, host processes, registries, CI/CD pipelines, and infrastructure as code repos. Response team roles and external reporting requirements should be defined.

Plans should be tested via tabletop exercises simulating different breach scenarios. Learnings should be incorporated into playbooks. The goal is having a tested response ready for investigating and recovering from real-world container attacks.

Organizations need to invest in incident response planning for container infrastructure much as they do for applications. Well-defined playbooks, data collection procedures, reporting flows, and breach simulations prepare teams to effectively respond.

Zero Day Vulnerability Discovery

Discovering New CVEs in Containers

Finding new vulnerabilities in containers that can potentially lead to CVE assignments involves techniques like fuzz testing, static analysis, and manual review. Fuzzing inputs to container APIs, exposed network services, volumes and bind mounts can reveal crashes indicating flaws. Fuzzing tools like Trivy Vulnerability Fuzzing, container-fuzz, and Kelp Fuzzer automate mutation testing of containers.

Static analysis examines container images themselves for misconfigurations and insecure coding practices that could introduce weaknesses. Tools like Anchore Engine disassemble images and scan configurations, security profiles, and dependencies for known bad practices that may constitute new vulnerabilities. Manual review is also needed to identify logical flaws in orchestration workflows, improperly configured controls, plain-text secrets, and other human errors that scanners cannot easily detect.

Organizations should utilize fuzz testing, static analysis, and manual reviews to thoroughly inspect container infrastructure. Finding CVE-worthy flaws provides valuable feedback to enhance security practices and reduce organizational risk.

There is no single shortcut to this process. Many tools used today are just a starting point for continuous security monitoring of containerized applications. It is assumed that for the time being the tasks for discovering new novel attacks will follow similar patterns as for the traditional methods of securing bare metal and virtual servers.

Fuzzing Container Network Services

Orchestrators like Kubernetes and automation tools employ network services and APIs to manage and configure containers. Fuzz testing these interfaces helps find flaws and poor input validation.

Tools like Kelp and container-fuzz enable fuzzing network services exposed by orchestrators via APIs, UIs, and intra-node communications. Mutating valid requests and introducing unexpected values exposes crashes, errors, and timeouts that may reflect vulnerabilities.

Fuzzing should cover application APIs, health check endpoints, metrics interfaces, intra-node coordination, webhook triggers, and any externally facing orchestrator components. Beyond crashes, analysis should check for output clipping, information leaks, strange responses, and unintended state changes. Crash analysis is critical to find new and novel vulnerabilities. Many papers have been published on this topic, by SANS, Team Cymru, Google Project Zero, and others.

Network service fuzzing also applies to CI/CD automation services, registry APIs, secrets management services, and administrative tools. Taken together, fuzz testing container management interfaces improves robustness and security.

Detecting 0-Days in Containers

Finding zero-day vulnerabilities in containerized applications requires dynamic and interactive analysis methods versus static scanning. Dynamic approaches like DAST probe running apps for common flaws. Interactive methods like IAST instrument apps and actively test for issues while running.

DAST tools like OWASP ZAP proxy traffic to apps in containers and inject payloads to test for SQL injection, XSS, command injection, authentication bypass, and other common issues. IAST solutions like Contrast Security plant sensors into containerized apps to detect attacks and vulnerabilities in production traffic flows. Tools such as Burp Suite may be used for bare metal, virtualized, and containerized application environments. It cannot be overstated that there are no shortcuts. If automated tools had the ability to detect every single vulnerability, there would be no need for security professionals.

Combining older flaw patterns with emerging techniques like business logic testing, behavioral detection, and synthesizing exploits allows discovering 0-days missed by static scanning. Testing pre-production mirror environments continuously versus just prod also improves fresh issue detection.

Auditing Real-World Containers

Auditing production container security configurations requires combining static inspection of image registries and runtime hosts with dynamic analysis of running containers.

Registry scanning tools like Anchore and Trivy validate production images for known vulnerabilities. Runtime tools like Falco and Sysdig monitor production container activities at scale. Compliance checks assess security controls like network rules, resource limits, and runtime privilege levels.

This is a critical process for container security, and should not be trivialized. As explored by previous questions and examples, this is a straightforward way of establishing baselines and detecting vulnerabilities.

Auditors also directly inspect production hosts for misconfigurations, monitor traffic between containers, and check volumes, secrets management, logging, and administrative access controls.

Taken together, static scanning, behavioral monitoring, configuration validation, and direct inspection of running production systems provides comprehensive auditing of real-world container deployments. The goal is ensuring security controls align with policies.

Conclusion

Most importantly, a strong security culture emphasizing least privilege, defense in depth, and patching is essential to avoid fundamental risky practices that containers may otherwise enable. With vigilance and discipline, organizations can harness the benefits of containers securely.

Containers and orchestrators like Docker and Kubernetes enable tremendous gains in agility, portability, and scalability for modern applications. But securing these dynamic environments requires new approaches compared to traditional long-lived servers.

This discussion covered risks resulting from initial enthusiastic adoption of containers without enough security consideration. We explored methods for hardening containers, integrating security across DevOps pipelines, and ensuring production deployments are locked down.

Key takeaways include leveraging minimal base images, automating security scanning, restricting unnecessary privileges, segmenting container networks, and implementing least privilege configs. Monitoring runtime activity, responding quickly to incidents, and centralizing security are also critical.

Overall, organizations must balance the benefits of containers and orchestration with the diligence required for defense-in-depth security. Containers incentivize automation, immutable delivery, and infrastructure as code, which can aid security when implemented conscientiously. With proper precautions and expertise, organizations can unlock the advantages of containers without undue risk.

The questions explored in this conversation highlight what security teams should be asking themselves when adopting containers. Readers are encouraged to use this as a starting point for driving conversations and improvements around container security within their own organizations.

References

1. Docker Documentation: [Docker security](https://docs.docker.com/engine/security/)

- This documentation explains the intrinsic security of the kernel, attack surface of the Docker daemon, container configuration profile loopholes, and kernel security features that interact with containers.

2. Docker Documentation: [Security best practices](https://docs.docker.com/develop/security-best-practices/)

- This documentation provides best practices for improving container security, including choosing the right base image, rebuilding images, checking images for vulnerabilities, and avoiding storing data in containers.

3. Google Cloud Architecture Center: [Best practices for building containers](https://cloud.google.com/architecture/best-practices-for-building-containers)

- This article covers a set of best practices for building containers, including file system security, running containers in read-only mode, and shortening build time.

4. OWASP Cheat Sheet Series: [Docker Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html)

- This cheat sheet provides common security mistakes and good practices for securing Docker containers, including container configuration, network security, and image security.

5. NIST Technical Series Publications: [Application Container Security Guide](https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf)

- This guide from the National Institute of Standards and Technology (NIST) provides recommendations for securing application containers, including container image security, runtime security, and orchestration platform security.

6. Kubernetes Hardening Guide: [Kubernetes Hardening Guide](https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF)

- This guide provides guidance on hardening Kubernetes clusters, including RBAC, network policies, and pod security standards.

These references provide detailed information and best practices for containerized security using Docker and Kubernetes, covering topics such as container isolation, vulnerability scanning, runtime monitoring, and secure deployment practices.

Citations:

[1] https://docs.docker.com/engine/security/

[2] https://docs.docker.com/develop/security-best-practices/

[3] https://cloud.google.com/architecture/best-practices-for-building-containers

[4] https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html

[5] https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf

[6] https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF

1. NIST Special Publication 800-190: [Application Container Security Guide](https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf)

- This NIST publication offers comprehensive guidance on securing application containers, covering topics such as container image security, runtime security, and orchestration platform security.

2. Red Hat OpenShift Documentation: [Container Security Guide](https://docs.openshift.com/container-platform/4.8/security/container_security/index.html)

- The Red Hat OpenShift documentation provides a detailed guide on container security, including securing container images, securing container runtimes, and securing container orchestration platforms.

3. Kubernetes Documentation: [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/)

- The Kubernetes documentation explains the concept of Pod Security Policies (PSPs), which allow administrators to define security policies for pods, such as enforcing read-only filesystems, restricting host access, and controlling privilege escalation.

4. Aqua Security: [Container Security Best Practices](https://www.aquasec.com/cloud-native-academy/container-security-best-practices/)

- This article by Aqua Security provides a list of best practices for container security, including securing the container image supply chain, implementing runtime security controls, and monitoring container behavior.

5. Sysdig: [Container Security and Compliance Guide](https://sysdig.com/blog/container-security-compliance-guide/)

- The Sysdig blog post offers insights into container security and compliance, covering topics such as vulnerability management, runtime security monitoring, and compliance considerations for containerized environments.

6. Snyk: [Container Security Best Practices](https://snyk.io/blog/10-container-security-best-practices/)

- This blog post by Snyk outlines ten best practices for container security, including minimizing the attack surface, regularly updating container images, and implementing strong access controls.

References and citations by Perplexity.ai

Hashtags

#containersecurity #dockersecurity #kubernetessecurity #cloudnativesecurity #DevSecOps #runtimesecurity #immutableinfrastructure #threatdetection #containerhardening #leastprivilege #iac #cicdsecurity #appsec #fuzztesting #zerotrust #runanywhere #secretsmanagement #vulnerabilitymanagement #incidentresponse #rbac #networksegmentation #serverless #kubernetesrbac #podsecuritypolicy #containerisolation #containermonitoring #containerforensics #containerdefense #microsegmentation #containerscan

Page updated

Google Sites

Report abuse