CONTINUAL INTEGRATION

Problem scenario
When you use docker login from a client server and you are challenged for credentials.  But after you enter the password and press enter you get this error:  "Error response from daemon: Get https://: x509: certificate signed by unknown authority"

You do not login to the Docker registry.  How do you get around this problem so you can log into the Docker registry?

Solution
Use one solution or the other.  Do not use both solutions!

Possible solution #1(less secure method; good for when no one else has access to the Docker registry server and it is just for learning)
1.  On the client server back up /etc/default/docker (if it is an important server or if you are very concerned).  Then modify this file.

Add this line to the file (but replace "FQDNofDockerRegistryServer" with the FQDN of the Docker registry server):

DOCKER_OPTS="--insecure-registry FQDNofDockerRegistryServer:443"

2.  Restart the Docker services on this client:

sudo service docker restart

3.  Now try docker login again.

Possible solution #2(more secure method; preferable when you have access to the Docker registry server)
1.  On the client server back up /etc/ssl/certs/ca-certificates.crt

2.  Copy from the Docker registry server this file: /etc/ssl/certs/ca-certificates.crt

3.  On the client server, delete the /etc/ssl/certs/ca-certificates.crt file.  Place a copy from the Docker registry server and put it on the client server in the same location and name as you just deleted it.  In other words overwrite the original /etc/ssl/certs/ca-certificates.crt file on the client server with a copy from the Docker registry server.

4.  Restart the Docker services on this client: sudo service docker restart

5.  Now try docker login again.

Problem scenario
You do not want to rely on internet-available Docker registries.  You want to have your own private Docker registry in your own AWS network.  You want to share Docker images (to build containers) with your team.  You think that installing/deploying a Docker registry will help you.  What do you do to build and configure (or just set up) your own Docker registry with an Ubuntu 16.x server?

Solution
These directions were tested to work with Ubuntu 16.x in AWS and Azure.  (They were loosely based on this DigitalOcean tutorial.)

Prerequisites
You need one Linux server to be the Docker registry. You need a second server with Docker that will be a client of the registry; to install Docker, this posting can help you.

Procedures
1. On the server that will be the Docker registry install Docker (sudo apt-get -y update; sudo apt -y install docker.io).  If you need additional directions for installing Docker, click here to see alternatives with more detail.
2. Install Docker-Compose.  For directions, see this posting.
3. Install apache2-utils with this command:  sudo apt-get -y install apache2-utils
4. Run these two commands:  

    mkdir ~/docker-registry && cd $_
    mkdir data

5. In the ~/docker-registry directory, create a file called docker-compose.yml with the following content (taken from this DigitalOcean tutorial):

registry:
  image: registry:2
  ports:
    - 127.0.0.1:5000:5000
  environment:
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
  volumes:
    - ./data:/data

6. Run these three commands:

    cd ~/docker-registry
    sudo service docker start
    docker-compose up

# Do no be alarmed if you see a message like this 'warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple '.  That message is not important.

7.  After the screen's text stops changing wait 30 seconds.  If it is still after 30 seconds, press Ctrl-c to cancel the above.

8.  Create an "nginx" subdirectory with this command: mkdir ~/docker-registry/nginx

9.  Modify the docker-compose.yml file by placing these lines at the very top.  You should also create one blank line between the last line below and the top line of the original file's content. (The lines below were taken from this DigitalOcean tutorial.):

nginx:
  image: "nginx:1.13"
  ports:
    - 5043:443
  links:
    - registry:registry
  volumes:
    - ./nginx/:/etc/nginx/conf.d:ro

10. Create a file ~/docker-registry/nginx/registry.conf with the following content from "upstream docker-registry..." to the last "}" before the next step. (The lines below were taken from this DigitalOcean tutorial.)

upstream docker-registry {
  server registry:5000;
}

server {
  listen 443;
  server_name myregistrydomain.com;

  # SSL
  # ssl on;
  # ssl_certificate /etc/nginx/conf.d/domain.crt;
  # ssl_certificate_key /etc/nginx/conf.d/domain.key;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication to v2 use auth_basic setting plus add_header
    # auth_basic "registry.localhost";
    # auth_basic_user_file /etc/nginx/conf.d/registry.password;
    # add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

    proxy_pass                          http://docker-registry;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

11. Run these commands:

cd ~/docker-registry
docker-compose up

12. Create a duplicate terminal session to the Docker host (the server that will be the Docker registry by using Putty or some other means).  From this duplicate session run these commands:  

curl http://localhost:5000/v2/
curl http://localhost:5043/v2/

Observe the output on the original terminal window from both commands.  If you see lines that include "registry_" in the output in the original terminal, then your are making progress.  Leave this duplicate terminal session open.  Go to the next step.

13.  From the original terminal session, press Ctrl-c

14. Run these two commands (to start setting up web authentication to the registry):

    cd ~/docker-registry/nginx
    htpasswd -c registry.password contint # where "contint" is a username you want to add
    # enter a password that you can remember when you are prompted.    

(To add users subsequently, run this command:  htpasswd registry.password jdoe # where jdoe is the user name you want to add.)

15.i. Modify the registry file by running this command:

vi ~/docker-registry/nginx/registry.conf

15.ii.  These three lines should be next to each other before you edit the file.  So find them in registry.conf.  Remove the leading "#" symbol (uncomment) these three lines:

   #  auth_basic "registry.localhost";
   #  auth_basic_user_file /etc/nginx/conf.d/registry.password;
   #  add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

Save the registry.conf file.

16. Run these two commands:  

cd ..
docker-compose up

17. Go back to the duplicate session for the server.  Run this command:  curl http://localhost:5043/v2/

You should see a "401 Authorization Required" message.

18.  Now try this command:

curl http://contint:password@localhost:5043/v2/

# Substitute "contint" with your username.  Substitute "password" with its password.

# You should see {} as a result of the above command.

# On the first terminal you should see "registry_" and other output when the curl command runs.

19. On the first terminal, press ctrl-C.

20.i. Now set up SSL by modifying the registry file again (e.g., vi ~/docker-registry/nginx/registry.conf).
20.ii. Find this stanza:

 server_name myregistrydomain.com;

# Change the "myregistrydomain.com" to whatever you want (e.g., coolname.com).  But you may keep it the same for the purposes of these directions.  

20.iii.  Find the "# SSL" line in registry.conf. (Leave the "# SSL" alone.) Uncomment the three lines beneath it.  They will look like this before you remove the "#" marks:

  # ssl on;
  # ssl_certificate /etc/nginx/conf.d/domain.crt;
  # ssl_certificate_key /etc/nginx/conf.d/domain.key;

Save the registry.conf file once the above lines are uncommented.

21.  Run these commands and respond to the prompts of the final command as you see fit:

cd ~/docker-registry/nginx
sudo openssl genrsa -out devdockerCA.key 2048
sudo openssl req -x509 -new -nodes -key devdockerCA.key -days 10000 -out devdockerCA.crt

Here is an example of many valid options to the prompts you will see from the "openssl" command above:

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:ContinualIntegration.com
Organizational Unit Name (eg, section) []:Main Department
Common Name (e.g. server FQDN or YOUR name) []:<FQDN of server to be Docker registry>
Email Address []:dne@doesnotexist.com

# Remember to replace "<FQDN of server to be Docker registry>" with the FQDN of the sever you want to be the Docker registry.

22.  Run these two commands:

sudo openssl genrsa -out domain.key 2048
sudo openssl req -new -key domain.key -out dev-docker-registry.com.csr

# Answer the prompts as you see fit, but do not type anything for the "challenge password."
# Here is an example of how to respond:

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:ContinualIntegration.com
Organizational Unit Name (eg, section) []:Main department
Common Name (e.g. server FQDN or YOUR name) []:<FQDN of server to be Docker registry>
Email Address []:dne@doesnotexist.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:ContInt

# Remember to replace "<FQDN of server to be Docker registry>" with the FQDN of the sever you want to be the Docker registry.

23. Run these seven commands:

sudo openssl x509 -req -in dev-docker-registry.com.csr -CA devdockerCA.crt -CAkey devdockerCA.key -CAcreateserial -out domain.crt -days 10000

sudo mkdir /usr/local/share/ca-certificates/docker-dev-cert
sudo cp devdockerCA.crt /usr/local/share/ca-certificates/docker-dev-cert
sudo update-ca-certificates
sudo service docker restart
cd ~/docker-registry
docker-compose up

24. From a duplicate terminal run this command:

curl -k https://contint:password@localhost:5043/v2/

# Where "contint" is the username and "password" its password.
# You should get "{}" as the result (with no quotes).

25. From the same duplicate terminal, run this command:  docker login https://localhost:5043

# Respond with the username and password created earlier (e.g., "contint").

26. Modify this file ~/docker-registry/docker-compose.yml

Find the stanza "- 5043:443" and change the "5043" to "443".  It will look like this:  "- 443:443"

27.  Go back to the first terminal with the "registry_" output.  Press Ctrl-c.

28.  Run this command: docker-compose up

29.  From a duplicate session or terminal window, run this command:

curl -k https://contint:password@localhost:443/v2/

# Where "contint" is the username and "password" its password.
# You should get "{}" as the result (with no quotes).

30.  Go back to the first terminal with the "registry_" output.  Press Ctrl-c.

31.  Warning!  One of these commands below will remove all existing containers.  Run these five commands:

    sudo apt -y install docker-registry
    cd ~/docker-registry
    docker-compose rm  # respond with "y" to this one  
    sudo mv ~/docker-registry /docker-registry
    sudo chown -R root: /docker-registry

32.  Create this file /etc/init/dc.sh with the following content:

#!/bin/bash
cd /docker-registry
exec /usr/local/bin/docker-compose up

33.  Become root with this command: sudo su -

34.  Run this command:  export EDITOR=vi

35.  Run this command: crontab -e  # Place this single line somewhere in the crontab (with no "#" comment mark before it, possibly on the lowest line):

@reboot sudo /bin/bash /etc/init/dc.sh

36.  Save the crontab.  Exit out of the root user.

37.  Reboot the server.

38.  Run this command and make sure you see a Docker container with "registry" in the "IMAGE" column:  docker ps

39.i. Configure the client server by logging into a different Linux server.  If you are using Ubuntu, use this command "sudo apt -y install docker.io"  If you are using RedHat in Azure, see this posting.  If you are using RedHat in AWS, see this posting.
39.ii.  On the client server, back up the /etc/ssl/certs/ca-certificates.crt file (e.g., to the same location with almost the same name but with a ".bak" extension).
39.iii. Use scp (or some other method) to copy the /etc/ssl/certs/ca-certificates.crt file from the Docker registry server to the client server.  If you need directions on how to set up passwordless SSH between two Linux servers in different public clouds, see this posting.
39.iv.  Overwrite the /etc/ssl/certs/ca-certificates.crt file with the one from the Docker registry server.
39.v.  Restart the Docker service on the client (sudo service docker restart).

40.  The client server should have one of the following:

i.  A DNS server configured (resolv.conf) so the Docker registry's FQDN resolves to the correct IP address.
ii. An /etc/hosts entry so that the Docker registry's FQDN resolves to the correct IP address.

41.  You should be done setting up the registry.  To test the registry, do the following.  Create a Docker image, upload it to the registry, then download it again and log into it.  To do these four things, follow these directions.  On the client server, run these commands:

docker login https://<FQDN of Docker registry server>
docker run -t -i ubuntu /bin/bash  # create a Docker container and enter it
touch /SUCCESS
exit
docker commit $(docker ps -lq) test-image  

# If you choose a different name from test-image, replace "test-image" in the below commands with the name you choose.

docker login https://<FQDN of Docker registry server>:443/
docker tag test-image <FQDN of Docker registry server>:443/test-image
docker push <FQDN of Docker registry server>:443/test-image
docker login https://<FQDN of Docker registry server>:443/
docker pull <FQDN of Docker registry server>:443/test-image
docker run -t -i <FQDN of Docker registry server>:443/test-image /bin/bash
ls