Minio for local S3 storage using Podman

Introduction

When using Single Node OpenShift and other options for local clusters it may be helpful to also have a local s3 instance.

The very basics

An instance of minio without HTTPS can be created very easily.

Networking considerations

Because the default ports for HTTP and HTTPS are privileged you must make a decision how to run these commands.

  • Run the podman run commands as root
  • Create /etc/sysctl.conf.d/99-unprivileged.conf with the line net.ipv4.ip_unprivileged_port_start=80 and run sudo sysctl --system
  • Modify the first half of the port redirects and proxy configs from 80 and 443 to ports higher than 1024 (for example 8080 and 8443)

You can open the ports via firewall-cmd if you are running firewalld.

sudo firewall-cmd --add-port 80/tcp
sudo firewall-cmd --add-port 9001/tcp

To make these permanent run them again:

sudo firewall-cmd --permanent --add-port 80/tcp
sudo firewall-cmd --permanent --add-port 9001/tcp

Create a data directory

  • mkdir -p /srv/podman/minio/data

Launch Minio

podman run \
  --detach \
  --name=minio-server \
  -e MINIO_ROOT_USER=changeme \
  -e MINIO_ROOT_PASSWORD=changeme \
  -v /srv/podman/minio/data:/data:z \
  --network=private \
  --restart unless-stopped \
  -p 80:9000 \
  -p 9001:9001 \
  docker.io/minio/minio:latest gateway nas /data --console-address ":9001"

Create a bucket

  • Visit the UI at http://localhost or the http://$remote-hostname and login with the credentials from your podman command.
  • Click Buckets in the left menu
  • Click Create Bucket in the top right
  • Give the bucket a name like velero
  • Click save

You are now free to configure the bucket in MTC. Most likely you’ll need to use the hostname of your system to access it.

More advanced configuration with SSL

If you’d like to add SSL you can run minio and an nginx proxy in a pod together to accomplish this.

Configure an nginx reverse proxy

We need to create some more directories for nginx configs:

mkdir -p /srv/podman/proxy/etc/nginx/conf.d/
mkdir -p /srv/podman/proxy/etc/ssl/certs

Create /srv/podman/proxy/etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

http {
  include       /etc/nginx/mime.types;

  map $status $loggable {
    ~^[23]  0;
    default 1;
  }

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log main if=$loggable;

  sendfile        on;
  keepalive_timeout  65;
  autoindex on;
  ssl_certificate /etc/ssl/certs/server.crt;
  ssl_certificate_key /etc/ssl/certs/server.key;

  include /etc/nginx/conf.d/*.conf;
}

events {
  worker_connections  1024;
}

Next you’ll need to create the reverse proxy configuration. You will need to update the three server_name lines with the hostname of your system. Wildcards and multiple names are valid if you need to provide them. See http://nginx.org/en/docs/http/server_names.html for more information.

/srv/podman/proxy/etc/nginx/conf.d/minio.conf:

server {
  listen 80;
  server_name hostname;
  
  location / {
    return 301 https://$host$request_uri;
  }
}

server {
  listen      443 ssl;
  server_name hostname;
  ignore_invalid_headers off;
  client_max_body_size 0;
  client_body_buffer_size 10m; 
  proxy_buffering off;

  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;
	proxy_pass http://localhost:9000;
  }
}

server {
  listen 9002;
  server_name hostname;

  location / {
    return 301 https://$host:9443$request_uri;
  }
}

server {
  listen      9443 ssl;
  server_name hostname;
  ignore_invalid_headers off;
  client_max_body_size 0;
  proxy_buffering off;

  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;
    proxy_pass http://localhost:9001;
  }
}

Create or procure your SSL cert and place your server.crt and server.key in /srv/podman/proxy/etc/ssl/certs

This Red Hat Enable Sysadmin post contains a basic example of how to generate certs if you require one.

Networking considerations

Use firewall-cmd to open the SSL ports if you are running firewalld

sudo firewall-cmd --add-port 443/tcp
sudo firewall-cmd --add-port 9443/tcp

To make these changes permanent also run:

sudo firewall-cmd --permanent --add-port 443/tcp
sudo firewall-cmd --permanent --add-port 9443/tcp

Launch Minio

We once again run minio, but this time we do so by creating a pod in the process. The port redirects may look odd here. Ports need to be exposed when the pod is created regardless of which container they are for so we’re exposing all of the ports required for nginx. This includes ports to redirect HTTP requests to HTTPS.

podman run \
  --detach \
  --pod=new:minio \
  --name=minio-server \
  -e MINIO_ROOT_USER=changeme \
  -e MINIO_ROOT_PASSWORD=changeme \
  -v /srv/podman/minio/data:/data:z \
  --network=private \
  --restart unless-stopped \
  -p 80:80 \
  -p 443:443 \
  -p 9001:9002 \
  -p 9443:9443 \
  docker.io/minio/minio:latest gateway nas /data --console-address ":9001"

Now we start nginx in the pod:

podman run \
  --detach \
  --name=proxy \
  --pod=minio \
  -v /srv/podman/proxy/etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro,z \
  -v /srv/podman/proxy/etc/nginx/conf.d:/etc/nginx/conf.d:ro,z \
  -v /srv/podman/proxy/etc/ssl/certs:/etc/ssl/certs:ro,z \
  --restart unless-stopped \
  docker.io/nginx:latest

You should now be able to connect to the Minio service and console using SSL.

When dealing with a containers in a pod you manage them with podman pod commands. For example, podman pod rm -f minio.

Minio for local S3 storage using OpenShift

Introduction

If you would like to create a minio instance directly within your cluster that is also an option. The setup here is similar to what was done with podman above, but instead of leveraging nginx to provide SSL we make use of services and routes to expose the service with SSL.

Note that while there is a Minio Operator available in OpenShift’s OperatorHub that can complete more advanced setups it currently makes use of v1beta1 CSRs and does not deploy properly on OpenShift 4.9+.

Setup

Assuming the minio namespace is available the only thing you should need to change is the username and password for minio.

export MINIO_USERNAME=changeme
export MINIO_PASSWORD=changeme
cat << EOF | oc create -f -
---
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: minio
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio
  namespace: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: minio
  name: minio
  namespace: minio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  triggers:
    - type: ConfigChange
  template:
    metadata:
      labels:
        app: minio
    spec:
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      containers:
      - args:
        - gateway
        - nas
        - /data
        - --console-address
        - :9001
        env:
          - name: MINIO_ROOT_USER
            value: $MINIO_USERNAME
          - name: MINIO_ROOT_PASSWORD
            value: $MINIO_PASSWORD
        image: docker.io/minio/minio:latest
        name: minio
        ports:
        - containerPort: 9000
          protocol: TCP
        - containerPort: 9001
          protocol: TCP
        volumeMounts:
        - mountPath: /data
          name: minio
      volumes:
        - name: minio
          persistentVolumeClaim:
            claimName: minio
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
  labels:
    app: minio
    service: minio
spec:
  selector:
    app: minio
  ports:
    - protocol: TCP
      port: 9000
      targetPort: 9000
---
apiVersion: v1
kind: Service
metadata:
  name: minio-console
  namespace: minio
  labels:
    app: minio
    service: minio-console
spec:
  selector:
    app: minio
  ports:
    - protocol: TCP
      port: 9001
      targetPort: 9001
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: minio
  namespace: minio
  labels:
    app: minio
    service: minio
spec:
  to:
    kind: Service
    name: minio
  port:
    targetPort: 9000
  tls:
    termination:                   edge
    insecureEdgeTerminationPolicy: Redirect
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: minio-console
  namespace: minio
  labels:
    app: minio
    service: minio-console
spec:
  to:
    kind: Service
    name: minio-console
  port:
    targetPort: 9001
  tls:
    termination:                   edge
    insecureEdgeTerminationPolicy: Redirect
EOF