Installation and Configuraton of Gitlab CE

The idea is to run GitLab inside of a docker host, rather than inside of k8s because we do not want the git server to be down because of some shenanigans in k8s. Also, if something in our CI/CD breaks k8s, we don’t want GitLab to be unaccessible because of that.

On the docker server, create a gitlab directory, and create a docker-compose.yml file. We will also copy in SSL certs we have created.

cd /srv/docker/apps
mkdir git
cd git
mkdir config logs data
mkdir config/ssl
cp git.weepytests.com.zip config/ssl/
cd config/ssl/
unzip git.weepytests.com.zip
rm git.weepytests.com.zip
cd ../../

We will set up gitlab to run with the docker repository enabled. You will need to create an SSL cert/key pair for use between gitlab and the gitlab registry. This can be self-signed. I have omitted the production key here, for obvious reasons.

version: '3'

services:

  web:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'wt-docker01.weepytests.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url "https://git.weepytests.com"
        nginx['redirect_http_to_https'] = true
        nginx['ssl_certificate'] = "/etc/gitlab/ssl/git.weepytests.com.crt"
        nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.weepytests.com.key"
        gitlab_rails['gitlab_https'] = true
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['registry_host'] = "registry.weepytests.com"
        gitlab_rails['registry_port'] = "443"
        gitlab_rails['registry_api_url'] = "http://registry:5000"
        gitlab_rails['registry_issuer'] = "gitlab-issuer"
        registry['internal_key'] = "<private key>"
    ports:
      - '8002:80'
      - '8003:443'
      - '2222:22'
    volumes:
      - '/srv/docker/apps/git/config:/etc/gitlab'
      - '/srv/docker/apps/git/logs:/var/log/gitlab'
      - '/srv/docker/apps/git/data:/var/opt/gitlab'
      - '/srv/docker/apps/git/certs:/certs'

  registry:
    image: registry
    restart: always
    expose:
      - "5000"
    ports:
      - "5000:5000"
    volumes:
      - '/srv/docker/apps/git/shared/registry:/registry'
      - '/srv/docker/apps/git/certs:/certs'
    environment:
      - REGISTRY_LOG_LEVEL=info
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://git.weepytests.com/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.weepytests.com.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true

Once this file has been saved, you can bring up the system. After starting it, tail the logfile for gitlab and wait for it to come up, it will take a while to start.

docker-compose up -d
docker ps
** look for the git sha **
docker logs -f <git sha>

Updating GitLab

GitLab will need to be kept current. They have a pretty aggressive release cycle, and if you lag behind, upgrading can be difficult. We will create a script and have cron run it every Sunday. This will bring down the service, backup all the files, SCP the backups to my main workstation, and then git pull the latest images, and bring everything back online.

Here is the update script (update_gitlab.sh):

#!/bin/bash
d=`date +%m-%d-%Y`
filename=wt-gitlab_backup_$d.tar.gz
cd /srv/docker/git
docker-compose down
tar -czvf $filename data/* config/* logs/* certs/* shared/*
scp $filename blair@192.168.76.33:/opt/storage/backups/
rm -f $filename
docker pull gitlab/gitlab-ce:latest
docker-compose up -d

We then add this to the crontab.

sudo crontab -e
30 1 * * 0 /bin/bash /srv/docker/git/update_gitlab.sh

nginx Reverse Proxy

It is also desirable to reverse-proxy these containers through nginx. Here are the required configs for the registry and the gitlab cluster.

git.conf

server {
        listen 80;
        server_name git.weepytests.com git;
        return 301 https://git.weepytests.com;
}

server {
        listen 443 ssl;
        ssl_certificate /etc/nginx/pki/git.weepytests.com.crt;
        ssl_certificate_key /etc/nginx/pki/git.weepytests.com.key;

        server_name git.weepytests.com;

        location / {
                proxy_pass https://192.168.77.100:8003;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}

registry.conf

server {
    root /dev/null;
    server_name registry.weepytests.com;
    charset UTF-8;
    access_log /var/log/nginx/registry.weepytests.com.access.log;
    error_log /var/log/nginx/registry.weepytests.com.error.log;

    # Set up SSL only connections:
    listen *:443 ssl http2;
    ssl_certificate             /etc/nginx/pki/registry.weepytests.com.crt;
    ssl_certificate_key         /etc/nginx/pki/registry.weepytests.com.key;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_session_timeout  5m;

    client_max_body_size        0;
    chunked_transfer_encoding   on;

    location / {
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_read_timeout                  900;
        proxy_pass        http://192.168.77.100:5000;
    }
}

server {
    listen *:80;
    server_name  registry.weepytests.com;
    server_tokens off; ## Don't show the nginx version number, a security best practice
    return 301 https://$http_host:$request_uri;
}

Installing ArgoCD into Kubernetes