openshift origin multi-master manually deployment part-1

i have deployed an openshift origin muliti-master cluster successfully on centos and suse enterprise server.
Since the deployment is done manually, i will describe it in detail as possibly as i can.

first, take a whole view of architecture,
we have three master nodes, all of them are in active state. the number of nodes is according to your cluster size. we do not need a lb node, so no haproxy is needed. we setup a VIP floating in three master nodes. for this architecture, no loadbanlance muchinasm is provided, but it is okay for high availibilty. If you realy need a loadbanlance, it is easy to deploy one after the whole cluster is up.

next, for deployment environment.
three master nodes, based on centos minimal installation, with static ip address.
hostname: ip address
master1.openshift.qyos.com 192.168.2.206
master2.openshift.qyos.com 192.168.2.207
master3.openshift.qyos.com 192.168.2.208
node1.openshift.qyos.com 192.168.2.209
VIP 192.168.2.205

i will break this deployment into four parts.
part1: deploy named, keepalived, and docker
part2: deploy master service
part3: deploy etcd cluster with ssl enabled
part4: deploy node service
part5: deploy router and docker registry service

okay, let’s start.

1, disable default firewalld service on all nodes

systemctl stop firewalld
systemctl disable firewalld 

2, we need a dns service, i use named instead of dnsmasq, and install it on all three master node.

yum install -y bind

configure bind for all three master nodes.

vi /etc/named.conf, modify the following entries:

listen-on port 53 { any; };
allow-query     { 0.0.0.0/0; };
dnssec-enable no;
dnssec-validation no;

vi /etc/named.rfc1912.zones, add the following zones.

zone "openshift.qyos.com" IN {
    type master;
    file "named.openshift.qyos.com";
    allow-update { none; };
};

zone "cluster.local" IN {
    type forward;
    forward only;
    forwarders { 192.168.2.205 port 8053; };
};
zone "kubernetes.default" IN {
    type forward;
    forward only;
    forwarders { 192.168.2.205 port 8053; };
};
zone "openshift.local" IN {
    type forward;
    forward only;
    forwarders { 192.168.2.205 port 8053; };
};

NOTE1: for openshift internal serivce dnslookup, we just forword it to openshift skydns service.
NOTE2: 192.168.2.205 is mine, change it to yours.

create named.openshift.qyos.com file
vi /var/named/named.openshift.qyos.com

$TTL 1D
@       IN SOA openshift.qyos.com. rname.invalid. (
                                        0       ; serial
                                        1D      ; refresh
                                        1H      ; retry
                                        1W      ; expire
                                        3H )    ; minimum
@       IN NS @
        A       192.168.2.205
master1 A       192.168.2.206
master2 A       192.168.2.207
master3 A       192.168.2.208
node1   A       192.168.2.209
*.route A       192.168.2.205

enable and start named service

systemctl enable named
systemctl start named

3, install keepalived on all three master nodes.

yum install -y keepalived

backup old configuration

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.bak

create new configuration file with contents below.
vi /etc/keepalived/keepalived.conf

global_defs {
  router_id master1
}

vrrp_instance 60 {
  virtual_router_id 60
  advert_int 1
  priority 101
  state BACKUP
  interface enp0s3
  virtual_ipaddress {
    192.168.2.205 dev enp0s3
  }
  unicast_src_ip  192.168.2.206
  unicast_peer {
    192.168.2.207
    192.168.2.208
  }

}

NOTE:
router_id should be updated for each master node, master1 for master node 1,
master2 for master node 2 .etc.
enp0s3 should be updated to your eth0 interface name.
we configure keepalived to use unicast to communicate with each other, so, the
src ip and peer ip should be updated on different master node. for example, on master 2,
src ip is 192.168.2.207, and peer is 206,208.

enable and start keepalived.

systemctl enable keepalived
systemctl start keepalived

to verify it is okay, run “ip a” on every master node, you can see that vip 192.168.2.205 is
on one of your master node.

last, add 192.168.2.205 in /etc/resolv.conf on every node.
vi /etc/resolv.conf
nameserver 192.168.2.205

4, install and setup docker, add parameters to docker daemon, on all nodes

yum install -y docker

vim /etc/sysconfig/docker

OPTIONS=' --selinux-enabled --log-driver=json-file --log-opt max-size=50m'
DOCKER_CERT_PATH=/etc/docker

do not start docker service at this time.