Docker 配置 TLS 认证

  上一篇文章中的警告在这里展开，警告内容如下：

WARNING: API is accessible on http://0.0.0.0:2376 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

  这样所有 ip 都能通过 docker -H <remote-dcoker-server_ip>:6379 [OPTION]命令与远程的 docker 守护进程通信，操作 docker 容器，生产上不提倡这种做法。

TLS

  传输层安全性协议（英语：Transport Layer Security，缩写作TLS），及其前身安全套接层（Secure Sockets Layer，缩写作SSL）是一种安全协议，目的是为互联网通信提供安全及数据完整性保障。

  CA是证书的签发机构。CA是负责签发证书、认证证书、管理已颁发证书的机关。它要制定政策和具体步骤来验证、识别用户身份，并对用户证书进行签名，以确保证书持有者的身份和公钥的拥有权。

  CA 也拥有一个证书（内含公钥）和私钥。网上的公众用户通过验证 CA 的签字从而信任 CA ，任何人都可以得到 CA 的证书（含公钥），用以验证它所签发的证书。

  证书包含以下信息：申请者公钥、申请者的组织信息和个人信息、签发机构 CA 的信息、有效时间、证书序列号等信息的明文，同时包含一个签名。

 签名的产生算法：首先，使用散列函数计算公开的明文信息的信息摘要，然后，采用 CA 的私钥对信息摘要进行加密，密文即签名。

  整个过程为：

• 1.服务器向CA机构获取证书，当浏览器首次请求服务器的时候，服务器返回证书给浏览器。（证书包含：公钥+申请者与颁发者的相关信息+签名）

• 2.浏览器得到证书后，开始验证证书的相关信息。

• 3.验证完证书后，如果证书有效，客户端是生成一个随机数，然后用证书中的公钥进行加密，加密后，发送给服务器，服务器用私钥进行解密，得到随机数。之后双方便开始用该随机数作为钥匙，对要传递的数据进行加密、解密。

  我们需要在远程 docker 服务器使用CA 认证来生成客户端和服务端证书、服务器密钥，然后自签名，再颁发证书给需要连接远程 docker ademon 的客户端。

以上内容部分摘自https://www.cnblogs.com/yunlongaimeng/p/9417276.html

TLS验证

  文档中指出：Docker over TLS should run on TCP port 2376.

服务端（CA机构、Docker Daemon在同一台服务器）

#1.生成CA私钥ca-key.pem（rsa加密）
root@admin-dsq:~/CA# openssl genrsa -aes256 -out ca-key.pem 4096 
Generating RSA private key, 4096 bit long modulus
.................................................................++
....................................................................
.................................................................................................++
e is 65537 (0x10001)
Enter pass phrase for ca-key.pem:
Verifying - Enter pass phrase for ca-key.pem:

#2.生成CA自签名证书ca.pem（其中包含证书信息、公钥信息）
root@admin-dsq:~/CA# openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem
Enter pass phrase for ca-key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:HB
Locality Name (eg, city) []:SJZ
Organization Name (eg, company) [Internet Widgits Pty Ltd]:HEUET
Organizational Unit Name (eg, section) []:NETLAB
Common Name (e.g. server FQDN or YOUR name) []:admin-dsq
Email Address []:

#3.生成服务器私钥server-key.pem
root@admin-dsq:~/local_p_key# openssl genrsa -out server-key.pem 4096
Generating RSA private key, 4096 bit long modulus
.....................................++
...........................................++
e is 65537 (0x10001)

#4. 生成服务器申请文件server.csr
root@admin-dsq:~/local_p_key# openssl req -subj "/CN=admin-dsq" -sha256 -new -key server-key.pem -out server.csr

#5.由于可以通过IP地址和DNS名称建立TLS连接，因此在创建证书时需要指定IP地址
#即允许通过admin-dsq、172.18.74.62/127.0.0.1建立TLS连接
root@admin-dsq:~/local_p_key# echo subjectAltName = DNS:admin-dsq,IP:172.18.74.62,IP:127.0.0.1 >> extfile.cnf

#6.将Docker守护程序密钥的扩展使用情况属性设置为仅用于服务器身份验证
root@admin-dsq:~/local_p_key# echo extendedKeyUsage = serverAuth >> extfile.cnf

#7.向服务器颁发CA证书server-cert.pem
root@admin-dsq:~/local_p_key# openssl x509 -req -days 365 -sha256 -in server.csr -CA /root/CA/ca.pem -CAkey \
/root/CA/ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
Signature ok
subject=/CN=admin-dsq
Getting CA Private Key
Enter pass phrase for ca-key.pem:

#8.现在，可以使Docker守护程序仅接受来自提供CA信任的证书的客户端的连接
root@admin-dsq:~/local_p_key# dockerd --tlsverify --tlscacert=/root/CA/ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem -H 0.0.0.0:2376
INFO[2019-04-03T10:37:04.136031407+08:00] parsed scheme: "unix"                         module=grpc
INFO[2019-04-03T10:37:04.136192758+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-04-03T10:37:04.136474710+08:00] parsed scheme: "unix"                         module=grpc
INFO[2019-04-03T10:37:04.136532152+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-04-03T10:37:04.137278333+08:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-04-03T10:37:04.140277683+08:00] [graphdriver] using prior storage driver: overlay2 
INFO[2019-04-03T10:37:04.140248053+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-04-03T10:37:04.140963667+08:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4205fc620, CONNECTING  module=grpc
INFO[2019-04-03T10:37:04.141777530+08:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4205fc620, READY  module=grpc
INFO[2019-04-03T10:37:04.142772412+08:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-04-03T10:37:04.142836434+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-04-03T10:37:04.142922653+08:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4205fc8e0, CONNECTING  module=grpc
INFO[2019-04-03T10:37:04.143886004+08:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4205fc8e0, READY  module=grpc
INFO[2019-04-03T10:37:04.179907069+08:00] Graph migration to content-addressability took 0.00 seconds 
WARN[2019-04-03T10:37:04.180499617+08:00] Your kernel does not support cgroup rt period 
WARN[2019-04-03T10:37:04.180563168+08:00] Your kernel does not support cgroup rt runtime 
INFO[2019-04-03T10:37:04.181663787+08:00] Loading containers: start.                   
INFO[2019-04-03T10:37:04.463375483+08:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2019-04-03T10:37:04.607165080+08:00] Loading containers: done.                    
INFO[2019-04-03T10:37:04.669799451+08:00] Docker daemon                                 commit=e8ff056 graphdriver(s)=overlay2 version=18.09.5
INFO[2019-04-03T10:37:04.669985943+08:00] Daemon has completed initialization          
INFO[2019-04-03T10:37:04.702975124+08:00] API listen on [::]:2376
#此时docker daemon已经启动监听2376端口

  使用scp将ca.pem，ca-key.pem发送到客户端服务器。其实应该先将服务器和客户端的申请文件发送至CA，由CA认证后向二者发放CA证书。

客户端

  对于客户端身份验证，应创建客户端密钥和证书签名请求

#1.生成客户端私钥client-key.pem
[root@docker-study client]# openssl genrsa -out client-key.pem 4096
Generating RSA private key, 4096 bit long modulus
......................................................++
.......................++
e is 65537 (0x10001)

#2.生成客户端申请文件client.csr
[root@docker-study client]# openssl req -subj '/CN=docker-study' -new -key client-key.pem -out client.csr

#3.使密钥适合客户端身份验证
[root@docker-study client]# echo extendedKeyUsage = clientAuth > extfile-client.cnf

#4.向客户端颁发CA证书client-cert.pem
[root@docker-study client]# openssl x509 -req -days 365 -sha256 -in client.csr -CA /root/ca.pem -CAkey \
/root/ca-key.pem -CAcreateserial -out client-cert.pem -extfile extfile-client.cnf
Signature ok
subject=/CN=docker-study
Getting CA Private Key
Enter pass phrase for ../ca-key.pem:

#5.向Docker Daemon连接，要携带CA自签名证书、客户端CA证书、客户端私钥
[root@docker-study client]# docker --tlsverify --tlscacert=/root/ca.pem --tlscert=client-cert.pem --tlskey=client-key.pem -H 172.18.74.62 info 
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.5
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-145-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 4.031GiB
Name: admin-dsq
ID: 26JC:YRWW:2HL7:W5AA:6FGM:UZEZ:EXMR:IR6A:GA2Z:IJBS:S5OA:QYKA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

#文章开头出的警告以消失

默认安全连接

  为简化操作，不必每次都对-H tcp://$host，–tls等参数进行调用，可以按下面步骤更改

  服务端：将安全验证添加到/lib/systemd/system/docker.service中

root@admin-dsq:~/.docker# ls # 此处为CA自签名证书、服务器证书、服务器密钥
ca.pem  cert.pem  key.pem
root@admin-dsq:~/.docker# cat /lib/systemd/system/docker.service |grep ExecStart
ExecStart=/usr/bin/dockerd --tlsverify --tlscacert=/root/.docker/ca.pem --tlscert=/root/.docker/cert.pem \
--tlskey=/root/.docker/key.pem -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock
root@admin-dsq:~/.docker# systemctl daemon-reload #重载守护进程
root@admin-dsq:~/.docker# systemctl restart docker
root@admin-dsq:~/.docker# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-04-04 09:03:25 CST; 5min ago
     Docs: https://docs.docker.com
 Main PID: 15180 (dockerd)
    Tasks: 10
   Memory: 30.5M
      CPU: 779ms
   CGroup: /system.slice/docker.service
           └─15180 /usr/bin/dockerd --tlsverify --tlscacert=/root/.docker/ca.pem --tlscert=/root/.docker/cert.pem --tlskey=/root/.docker/key.pem 
           ………………
root@admin-dsq:~/local_p_key# docker info | grep Name 
Name: admin-dsq
root@admin-dsq:~/.docker# docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:44:24 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false

  客户端：如果要在默认情况下保护Docker客户端连接，可以将文件移动到.docker目录中，并设置 DOCKER_HOST和DOCKER_TLS_VERIFY变量

[root@docker-study .docker]# ls #此处为CA自签名证书、客户端证书、客户端密钥
ca.pem  cert.pem  key.pem

# 添加环境变量，设置远程主机和开启TLS验证
[root@docker-study .docker]# export DOCKER_HOST=tcp://172.18.74.62:2376 DOCKER_TLS_VERIFY=1
[root@docker-study .docker]# docker version # 连接到了远程服务端
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false

  Docker现在默认安全连接

过程中出现的错误

• CA自签名证书默认路径为/root/.docker，服务器或客户端的CA证书和密钥也应该放在此目录下

[root@docker-study ~]# docker --tlsverify -H 172.18.74.62 info
could not read CA certificate "/root/.docker/ca.pem": open /root/.docker/ca.pem: no such file or directory

• info：x509: cannot validate certificate for 172.18.74.62 because it doesn’t contain any IP SANs

[root@docker-study /]# docker --tlsverify -H 172.18.74.62 info
error during connect: Get https://172.18.74.62:2376/v1.39/info: x509: cannot validate certificate for 172.18.74.62 because it doesn't contain any IP SANs

  在指定证书的过程中要指定IP地址，参照：https://stackoverflow.com/questions/42116783/x509-cannot-validate-certificate-because-it-doesnt-contain-any-ip-sans 或者可以按照Docker文档中所给出的步骤即上述中的过程。

• 在客户端连接Docker Daemon时未成功，在Docker Daemon端报出如下信息：

客户端：
[root@docker-study .docker]# docker --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem -H 172.18.74.62 info
The server probably has client authentication (--tlsverify) enabled. Please check your TLS client certification settings: \
Get https://172.18.74.62:2376/v1.39/info: remote error: tls: bad certificate

服务端显示：
2019-04-16 10:45:46.370039 I | http: TLS handshake error from 172.18.74.101:44292: tls: client didn't provide a certificate

  Docker Daemon 使用 dockerd –tlsverify –tlscacert=/root/CA/ca.pem –tlscert=server-cert.pem –tlskey=server-key.pem -H 0.0.0.0:2376 启动后仅接受来自提供CA信任的客户端的连接，所以客户端也需要进行CA认证，二者所使用的CA自签名证书是一样的。

  当ca.pem不同时有如下报错：

[root@docker-study .docker]# docker info
error during connect: Get https://172.18.74.62:2376/v1.39/info: x509: certificate signed by unknown authority

• 报错为私钥和公钥不匹配

[root@docker-study client]# docker --tlsverify --tlscacert=../ca.pem --tlscert=../server-cert.pem --tlskey=key.pem -H 172.18.74.62 info
Could not load X509 key pair: tls: private key does not match public key

• curl: (60) Peer’s Certificate issuer is not recognized.

[root@docker-study ~]# curl https://172.18.74.62:2376/info
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

  报错为无法识别证书颁发者，因为是自己制作的证书，系统对CA没有认证，所以无法识别。
  解决办法是将签发该证书的私有CA公钥cacert.pem文件内容，追加到/etc/pki/tls/certs/ca-bundle.crt下。

#ubuntu下
root@admin-dsq:~/.docker# cat ca.pem >> /etc/ssl/certs/ca-certificates.crt
#centos下
[root@docker-study .docker]# cat ca.pem >> /etc/pki/tls/certs/ca-bundle.crt

• curl: (58) NSS: client certificate not found (nickname not specified)

[root@docker-study .docker]# curl https://172.18.74.62:2376/version
curl: (58) NSS: client certificate not found (nickname not specified)

没有找到客户端认证，加–cert参数指定下路径就行了

• NSS error -8178 (SEC_ERROR_BAD_KEY)

[root@docker-study .docker]# curl --cert /root/.docker/cert.pem https://172.18.74.62:2376/version
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
[root@docker-study .docker]# curl -v --cert /root/.docker/cert.pem https://172.18.74.62:2376/version
* About to connect() to 172.18.74.62 port 2376 (#0)
*   Trying 172.18.74.62...
* Connected to 172.18.74.62 (172.18.74.62) port 2376 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
* NSS error -8178 (SEC_ERROR_BAD_KEY)
* Peer's public key is invalid.
* Closing connection 0
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

参考：https://stackoverflow.com/questions/22499425/ssl-certificate-generated-with-openssl-not-working-on-nss?answertab=active#tab-top

我尝试了无密码和另外的des3 加密算法都没有成功，仍然是这个错误，待更新。