Currently I’m working on my RHCE certification and I’m facing some issue in settings up a lab in order to properly test the Kerberized NFS share. Since this has been reported as one of the most complex topic to test I hope someone will find this post helpful for his/her RHCE preparation.
I have deployed my lab on 3 t2.micro virtual EC2 instances on Amazon while using my free annual subscription. That t2.micro configuration offers a 1core/1GB memory/10GB disk featured host that will serve you (almost) well while studying for your RHCE (and RHCSA) exam.
I will be working with the AMI RedHat 7.2 that is available directly from AWS images.
The 3 hosts are configured like below:
- ipa1a.example.com, the host where you will install the ipa server that will provide LDAP/Kerberos to the lab
- backend1a.example.com, the host where you will install and configure your NFS server
- frontend1a.example.com, the host where you will install and configure your NFS client
Please note: hostname configuration on AWS/EC2 requires user data and cloud-init tool (this is not part of a standard RH deployment nor RHCE objectives). Also, IPA setup isn’t part of the RHCE objectives, but since you will need it in order to make some practice with Kerberos and IDM it is worth to know something about how it works…
On AWS I’m using a single jump-box configured with a public dynamically assigned IP and accepting SSH connection only from my home network (check AWS Security Groups for more details). From that jumpbox I can reach my RH7 lab (from its /etc/hosts):
172.31.1.10 ipa1a.example.com ipa1a 172.31.1.20 backend1a.example.com backend1a 172.31.1.30 frontend1a.example.com frontend1a
1. Prepare the hosts.
The default template for the RedHat 7 hosts on AWS provides a very minimal set of features that won’t fit more likely your needs for the RHCE / RHCSA practice. You will need to make some change in order to have a properly configured host. A major issue will be about the memory. By default, your virtual host has no swap space configured, while installing IPA you will have issue because of the “Out of Memory” issue you will trigger with the ipa-server-install script, so I do recommend to add at least 1GB volume and configure it as SWAP partition.
Once your named will be installed on ipa1a.example.com it is also required that all your hosts in your lab will use ipa1a.example.com as DNS and prevent the DHCP client to update the nameserver entry in /etc/resolv.conf. They are both tasks you should be able to manage, since they are in the objectives of the RHCSA / RHCE (you will find more details later on this post).
If you have trouble in configuring the hostname please have a look at how cloud-init and user-data can work on AWS (as far as I can see normal hostname configuration procedure as it is documented by RH doesn’t work properly on AWS and won’t survive a reboot of the instance), you may want to add in your “User Data” something like:
#cloud-config hostname: frontend1a.example.com
in order to properly set the desired hostname on boot.
Please note: you need to stop your EC2 instance in order to update your User Data.
Also, I do recommend to install on every host the “Server with GUI” group, since it provides an almost complete environment for your RHCSA/RHCE practice
sudo yum groupinstall -y "Server with GUI"
2. Install the required packages for IPA server (ipa1a.example.com).
You will need to install the ipa server packages:
sudo yum install -y ipa-server bind nds-ldap ipa-server-dns
Please note: you will read somewhere that ipa-server-dns is not required and you can leverage on the /etc/hosts. I recommend to install it and configure your other hosts in order to use ipa.example.com as DNS, it will make your life a lot easier during the clients configuration.
3. Configure your IPA server (ipa1a.example.com).
I will use example.com domain and realm. First of all, verify your host is resolved on the external IP, not on 127.0.0.1, your /etc/hosts should look like below:
[root@ipa1a ~]# getent hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 localhost4.localdomain4 localhost4 127.0.0.1 localhost.localdomain localhost 127.0.0.1 localhost6.localdomain6 localhost6 172.31.1.10 ipa1a.example.com ipa1a
It is important that your hostname doesn’t resolve on 127.0.0.1 or ::1.
Then proceed with the configuration of the ipa server that for me will be ipa1a.example.com
sudo -i ipa-server-install --hostname=ipa1a.example.com -n example.com -r EXAMPLE.COM -p password -a password -U --no-ntp --setup-dns --forwarder=172.31.0.2
For me, “172.31.0.2” is the IP of the available resolver, you may have a different IP, bear in mind DNS will be configured as forwarder in order to resolve external host, so it has to be a valid DNS or once your hosts will be configured in order to work with ipa1a as primary DNS they will fail in finding external hostnames for update and other stuff.
It will take some time to complete the configuration and deployment of the ipaserver and all its components. Installation procedure will be complete when you will receive the below message:
============================================================================== Setup complete Next steps: 1. You must make sure these network ports are open: TCP Ports: * 80, 443: HTTP/HTTPS * 389, 636: LDAP/LDAPS * 88, 464: kerberos * 53: bind UDP Ports: * 88, 464: kerberos * 53: bind 2. You can now obtain a kerberos ticket using the command: 'kinit admin' This ticket will allow you to use the IPA tools (e.g., ipa user-add) and the web user interface. 3. Kerberos requires time synchronization between clients and servers for correct operation. You should consider enabling ntpd. Be sure to back up the CA certificates stored in /root/cacert.p12 These files are required to create replicas. The password for these files is the Directory Manager password
Next step will be to configure your firewall as suggested above:
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=http success [root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=https success [root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=ldap success [root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=ldaps success [root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=kerberos success [root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=dns success [root@ipa1a ~]# sudo firewall-cmd --reload success
Now you can verify ipa server has been installed properly by performing a quick check like obtaining a token from Kerberos in order to access the IPA tools (password is ‘password’)
[root@ipa1a ~]# sudo -i ; kinit admin Password for admin@EXAMPLE.COM:
Then look for the admin profile in the LDAP
[root@ipa1a ~]# ipa user-find admin -------------- 1 user matched -------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash UID: 702400000 GID: 702400000 Account disabled: False Password: True Kerberos keys available: True ---------------------------- Number of entries returned 1 ----------------------------
Command “kinit admin”creates a kerberos token you can use in order to access as “admin” you management tools (ipa commands). Password is the one we have specified with “-a password”, so it will be ‘password’ unless you have specified something else while running ipa-server-install command. Every time you open a shell and plan to access ipa tools don’t forget to get a token with “kinit admin”.
4. Configure DNS, hosts and services on IPA.
In order to properly work and interact with Kerberos it is required that every host and service is provisioned. So we’ll need to configure backend1a and frontend1a and enable the NFS service access for both.Since your ipa1a instance is configured in order to use dhcp (and on a cloud environment I do recommend to work like this) just verify you don’t let dhcp client update your /etc/resolv.conf with the nameserver the dhcp server provides (this is part of what a RHCSA is supposed to know, see “man -K PEERDNS” for details).
At first, update your DNS in order to resolve the involved hosts, add the DNS record for backend1a (ip 172.31.1.20), the command will prompt for the record type (A) and the IP Address (172.31.1.20 or whatever you have defined for your lab)
[root@ipa1a ~]# ipa dnsrecord-add example.com backend1a Please choose a type of DNS resource record to be added The most common types for this type of zone are: A, AAAA DNS resource record type: A A IP Address: 172.31.1.20 Record name: backend1a A record: 172.31.1.20
Then add the DNS record for frontend1a (ip 172.31.1.30), the command will prompt for the record type (A) and the IP Address (172.31.1.30).
[root@ipa1a ~]# ipa dnsrecord-add example.com frontend1a Please choose a type of DNS resource record to be added The most common types for this type of zone are: A, AAAA DNS resource record type: A A IP Address: 172.31.1.30 Record name: frontend1a A record: 172.31.1.30
Now create the profile for the hosts with ipa host-add <host>:
[root@ipa1a ~]# ipa host-add backend1a.example.com ---------------------------------- Added host "backend1a.example.com" ---------------------------------- Host name: backend1a.example.com Principal name: host/backend1a.example.com@EXAMPLE.COM Password: False Keytab: False Managed by: backend1a.example.com [root@ipa1a ~]# ipa host-add frontend1a.example.com ----------------------------------- Added host "frontend1a.example.com" ----------------------------------- Host name: frontend1a.example.com Principal name: host/frontend1a.example.com@EXAMPLE.COM Password: False Keytab: False Managed by: frontend1a.example.com
And finally create the NFS service for both with ipa service-add <service/host>
[root@ipa1a ~]# ipa service-add nfs/backend1a.example.com ----------------------------------------------------- Added service "nfs/backend1a.example.com@EXAMPLE.COM" ----------------------------------------------------- Principal: nfs/backend1a.example.com@EXAMPLE.COM Managed by: backend1a.example.com [root@ipa1a ~]# ipa service-add nfs/frontend1a.example.com ------------------------------------------------------ Added service "nfs/frontend1a.example.com@EXAMPLE.COM" ------------------------------------------------------ Principal: nfs/frontend1a.example.com@EXAMPLE.COM Managed by: frontend1a.example.com
4. Configure your hosts in order to work with IPA server.
Now that your server is configured it’s time to bind the other hosts to Kerberos and let them use its authentication methods for you lab and tests.
Add 1GB of swap to your host (if you haven’t done that already), then configure your host in order to use use ipa1a.example.com (for me 172.31.1.10) as dns resolver by updating /etc/sysconfig/network-scripts/ifcft-eth0 (or any other file) by disabling the PEERDNS option and adding DNS1 and DOMAIN entries, e.g.
DEVICE="eth0" BOOTPROTO="dhcp" ONBOOT="yes" TYPE="Ethernet" USERCTL="yes" PEERDNS="no" IPV6INIT="no" DOMAIN=example.com DNS1=172.31.1.10
Reboot your hosts and verify everything is working fine.
Perform the above steps on all the hosts you want to use for your RedHat 7 Lab (more likely frontend1a and backend1a).
Then verify name resolving is properly working (e.g. with dig ipa1a.example.com):
[root@backend1a ~]# cat /etc/resolv.conf # Generated by NetworkManager search example.com nameserver 172.31.1.10 [root@backend1a ~]# dig ipa1a.example.com ; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> ipa1a.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 779 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ipa1a.example.com. IN A ;; ANSWER SECTION: ipa1a.example.com. 1200 IN A 172.31.1.10 ;; AUTHORITY SECTION: example.com. 86400 IN NS ipa1a.example.com. ;; Query time: 1 msec ;; SERVER: 172.31.1.10#53(172.31.1.10) ;; WHEN: Sat Aug 20 11:06:25 EDT 2016 ;; MSG SIZE rcvd: 76
Now you can proceed by installing the ipa-client packages.
sudo yum install -y ipa-client ipa-admintools
Then proced with the configuration by executing
sudo -i ipa-client-install --hostname=<hostname> --enable-dns-update
replace <hostname> with frontend1a.example.com or backend1a.example.com. It will be requested a user authorized to enroll computers, specify “admin” and then when prompted for the password type “password”
[root@frontend1a ~]# ipa-client-install --hostname=frontend1a.example.com --enable-dns-update WARNING: ntpd time&date synchronization service will not be configured as conflicting service (chronyd) is enabled Use --force-ntpd option to disable it and force configuration of ntpd Discovery was successful! Client hostname: frontend1a.example.com Realm: EXAMPLE.COM DNS Domain: example.com IPA Server: ipa1a.example.com BaseDN: dc=example,dc=com Continue to configure the system with these values? [no]: yes Skipping synchronizing time with NTP server. User authorized to enroll computers: admin Password for admin@EXAMPLE.COM: Successfully retrieved CA cert Subject: CN=Certificate Authority,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Valid From: Sat Aug 20 14:15:41 2016 UTC Valid Until: Wed Aug 20 14:15:41 2036 UTC Enrolled in IPA realm EXAMPLE.COM Created /etc/ipa/default.conf New SSSD config will be created Configured sudoers in /etc/nsswitch.conf Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm EXAMPLE.COM trying https://ipa1a.example.com/ipa/json Forwarding 'ping' to json server 'https://ipa1a.example.com/ipa/json' Forwarding 'ca_is_enabled' to json server 'https://ipa1a.example.com/ipa/json' Systemwide CA database updated. Added CA certificates to the default NSS database. Missing reverse record(s) for address(es): 172.31.1.30. Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Forwarding 'host_mod' to json server 'https://ipa1a.example.com/ipa/json' SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config Configuring example.com as NIS domain. Client configuration complete.
Now you can perform some check and verify the communications between the hosts and the server, from the host you can use “kinit admin” and access the ipa tools in the same way you can do on the server, so you can lookup and check services, e.g.
[root@frontend1a ~]# kinit admin Password for admin@EXAMPLE.COM: [root@frontend1a ~]# ipa service-find nfs ------------------ 2 services matched ------------------ Principal: nfs/backend1a.example.com@EXAMPLE.COM Keytab: False Managed by: backend1a.example.com Principal: nfs/frontend1a.example.com@EXAMPLE.COM Keytab: False Managed by: frontend1a.example.com ---------------------------- Number of entries returned 2 ----------------------------
5. Retrieve the keytab you will use for your Kerberized NFS exercises.
In this lab you can leverage on ipa tool and update your local copy of the /etc/krb5.keytab with the required principals. After the configuration, your hosts will have the principals for the host, but they will miss the ones for the NFS service, you will need to update your local /etc/krb5.keytab by running on every host the command below, on backend1a:
[root@backend1a ~]# ipa-getkeytab -s ipa1a.example.com -p nfs/backend1a.example.com -k /etc/nfs.keytab Keytab successfully retrieved and stored in: /etc/krb5.keytab
while on frontend1a
[root@frontend1a ~]# ipa-getkeytab -s ipa1a.example.com -p nfs/frontend1a.example.com -k /etc/krb5.keytab Keytab successfully retrieved and stored in: /etc/krb5.keytab
(command will update the existing keytab)
It is possible to check the content of your /etc/krb5.keytab file with the command klist -k /etc/krb5.keytab that will provide the output below:
[root@frontend1a ~]# klist -k /etc/krb5.keytab Keytab name: FILE:/etc/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 1 host/frontend1a.example.com@EXAMPLE.COM 1 host/frontend1a.example.com@EXAMPLE.COM 1 host/frontend1a.example.com@EXAMPLE.COM 1 host/frontend1a.example.com@EXAMPLE.COM 1 nfs/frontend1a.example.com@EXAMPLE.COM 1 nfs/frontend1a.example.com@EXAMPLE.COM 1 nfs/frontend1a.example.com@EXAMPLE.COM 1 nfs/frontend1a.example.com@EXAMPLE.COM
6. Test your Kerberized NFS.
Now you can give NFS with Kerberos a try and verify if your LAB works.
On backend1a.example.com configure the NFS service in order to export a krb5p secured filesystem, e.g.
on backend1a.example.com:
sudo -i mkdir /krbnfs chown nfsnobody /krbnfs echo "/krbnfs frontend1a.example.com(sec=krb5p,rw)" > /etc/exports.d/krbnfs.exports systemctl restart nfs-server firewall-cmd --permanent --add-server=nfs firewall-cmd --reload
on frontend1a.example.com:
sudo -i systemctl restart nfs-secure mount -t nfs4 -o sec=krb5p backend1a.example.com:/krbnfs /mnt/ mount|grep krbnfs backend1a.example.com:/krbnfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.31.1.30,local_lock=none,addr=172.31.1.20)
Congratulations! Enjoy your kerberized RH7 Lab on AWS 🙂
PS: on older version of RH (7.0, not sure about 7.1) if was also required to enable and use nfs-secure-server on the NFS server while on latest builds (like the RH 7.2 we have on AWS) it is not required since GSSProxy is taking care of the “secure” jobs (see https://fedorahosted.org/gss-proxy/wiki/NFS for details).
PPS: this lab will work well also in order to test the LDAP based Authentication that is part of the RHCSA objectives
7. How to troubleshoot IPA/Kerberos.
If you have trouble and your Kerberized NFS exports don’t work, you may find helpful to check on ipa1a.example.com the /var/log/krb5kcd.log and see if odd requests / host / service are passing. It could help to spot misconfiguration / typos or other stuff. Good luck!
Yari 