Tag: Kerberos

RH7LAB – Kerberized NFS with IPA Server

RH7LAB – Kerberized NFS with IPA Server

Currently I’m working on my RHCE certification and I’m facing some issue in settings up a lab in order to properly test the Kerberized NFS share. Since this has been reported as one of the most complex topic to test I hope someone will find this post helpful for his/her RHCE preparation.

I have deployed my lab on 3 t2.micro virtual EC2 instances on Amazon while using my free annual subscription. That t2.micro configuration offers a 1core/1GB memory/10GB disk featured host that will serve you (almost) well while studying for your RHCE (and RHCSA) exam.

I will be working with the AMI RedHat 7.2 that is available directly from AWS images.


The 3 hosts are configured like below:

  • ipa1a.example.com, the host where you will install the ipa server that will provide LDAP/Kerberos to the lab
  • backend1a.example.com, the host where you will install and configure your NFS server
  • frontend1a.example.com, the host where you will install and configure your NFS client

cloudcraft - RH7LAB - Kerberized NFS (1)

Please note: hostname configuration on AWS/EC2 requires user data and cloud-init tool (this is not part of a standard RH deployment nor RHCE objectives). Also, IPA setup isn’t part of the RHCE objectives, but since you will need it in order to make some practice with Kerberos and IDM it is worth to know something about how it works…

On AWS I’m using a single jump-box configured with a public dynamically assigned IP and accepting SSH connection only from my home network (check AWS Security Groups for more details). From that jumpbox I can reach my RH7 lab (from its /etc/hosts): ipa1a.example.com ipa1a backend1a.example.com backend1a frontend1a.example.com frontend1a

1. Prepare the hosts.

The default template for the RedHat 7 hosts on AWS provides a very minimal set of features that won’t fit more likely your needs for the RHCE / RHCSA practice. You will need to make some change in order to have a properly configured host. A major issue will be about the memory. By default, your virtual host has no swap space configured, while installing IPA you will have issue because of the “Out of Memory” issue you will trigger with the ipa-server-install script, so I do recommend to add at least 1GB volume and configure it as SWAP partition.

Once your named will be installed on ipa1a.example.com it is also required that all your hosts in your lab will use ipa1a.example.com as DNS and prevent the DHCP client to update the nameserver entry in /etc/resolv.conf. They are both tasks you should be able to manage, since they are in the objectives of the RHCSA / RHCE (you will find more details later on this post).

If you have trouble in configuring the hostname please have a look at how cloud-init and user-data can work on AWS (as far as I can see normal hostname configuration procedure as it is documented by RH doesn’t work properly on AWS and won’t survive a reboot of the instance), you may want to add in your “User Data” something like:

 hostname: frontend1a.example.com

in order to properly set the desired hostname on boot.
Please note: you need to stop your EC2 instance in order to update your User Data.

Also, I do recommend to install on every host the “Server with GUI” group, since it provides an almost complete environment for your RHCSA/RHCE practice

sudo yum groupinstall -y "Server with GUI"

2. Install the required packages for IPA server (ipa1a.example.com).

You will need to install the ipa server packages:

sudo yum install -y ipa-server bind nds-ldap ipa-server-dns

Please note: you will read somewhere that ipa-server-dns is not required and you can leverage on the /etc/hosts. I recommend to install it and configure your other hosts in order to use ipa.example.com as DNS, it will make your life a lot easier during the clients configuration.

3. Configure your IPA server (ipa1a.example.com).

I will use example.com domain and realm. First of all, verify your host is resolved on the external IP, not on, your /etc/hosts should look like below:

[root@ipa1a ~]# getent hosts localhost.localdomain localhost localhost4.localdomain4 localhost4 localhost.localdomain localhost localhost6.localdomain6 localhost6 ipa1a.example.com ipa1a

It is important that your hostname doesn’t resolve on or ::1.

Then proceed with the configuration of the ipa server that for me will be ipa1a.example.com

sudo -i
ipa-server-install --hostname=ipa1a.example.com -n example.com -r EXAMPLE.COM -p password -a password -U --no-ntp --setup-dns --forwarder=

For me, “” is the IP of the available resolver, you may have a different IP, bear in mind DNS will be configured as forwarder in order to resolve external host, so it has to be a valid DNS or once your hosts will be configured in order to work with ipa1a as primary DNS they will fail in finding external hostnames for update and other stuff.

It will take some time to complete the configuration and deployment of the ipaserver and all its components. Installation procedure will be complete when you will receive the below message:

Setup complete

Next steps:
 1. You must make sure these network ports are open:
 TCP Ports:
 * 80, 443: HTTP/HTTPS
 * 389, 636: LDAP/LDAPS
 * 88, 464: kerberos
 * 53: bind
 UDP Ports:
 * 88, 464: kerberos
 * 53: bind
 2. You can now obtain a kerberos ticket using the command: 'kinit admin'
 This ticket will allow you to use the IPA tools (e.g., ipa user-add)
 and the web user interface.
 3. Kerberos requires time synchronization between clients
 and servers for correct operation. You should consider enabling ntpd.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

Next step will be to configure your firewall as suggested above:

[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=http
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=https
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=ldap
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=ldaps
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=kerberos
[root@ipa1a ~]# sudo firewall-cmd --permanent --add-service=dns
[root@ipa1a ~]# sudo firewall-cmd --reload

Now you can verify ipa server has been installed properly by performing a quick check like obtaining a token from Kerberos in order to access the IPA tools (password is ‘password’)

[root@ipa1a ~]# sudo -i ; kinit admin
Password for admin@EXAMPLE.COM:

Then look for the admin profile in the LDAP

[root@ipa1a ~]# ipa user-find admin
1 user matched
 User login: admin
 Last name: Administrator
 Home directory: /home/admin
 Login shell: /bin/bash
 UID: 702400000
 GID: 702400000
 Account disabled: False
 Password: True
 Kerberos keys available: True
Number of entries returned 1

Command “kinit admin”creates a kerberos token you can use in order to access as “admin” you management tools (ipa commands). Password is the one we have specified with “-a password”, so it will be ‘password’ unless you have specified something else while running ipa-server-install command. Every time you open a shell and plan to access ipa tools don’t forget to get a token with “kinit admin”.

4. Configure DNS, hosts and services on IPA.

In order to properly work and interact with Kerberos it is required that every host and service is provisioned. So we’ll need to configure backend1a and frontend1a and enable the NFS service access for both.Since your ipa1a instance is configured in order to use dhcp (and on a cloud environment I do recommend to work like this) just verify you don’t let dhcp client update your /etc/resolv.conf with the nameserver the dhcp server provides (this is part of what a RHCSA is supposed to know, see “man -K PEERDNS” for details).

At first, update your DNS in order to resolve the involved hosts, add the DNS record for backend1a (ip, the command will prompt for the record type (A) and the IP Address ( or whatever you have defined for your lab)

[root@ipa1a ~]# ipa dnsrecord-add example.com backend1a
Please choose a type of DNS resource record to be added
The most common types for this type of zone are: A, AAAA

DNS resource record type: A
A IP Address:
 Record name: backend1a
 A record:

Then add the DNS record for frontend1a (ip, the command will prompt for the record type (A) and the IP Address (

[root@ipa1a ~]# ipa dnsrecord-add example.com frontend1a
Please choose a type of DNS resource record to be added
The most common types for this type of zone are: A, AAAA

DNS resource record type: A
A IP Address:
 Record name: frontend1a
 A record:

Now create the profile for the hosts with ipa host-add <host>:

[root@ipa1a ~]# ipa host-add backend1a.example.com
Added host "backend1a.example.com"
 Host name: backend1a.example.com
 Principal name: host/backend1a.example.com@EXAMPLE.COM
 Password: False
 Keytab: False
 Managed by: backend1a.example.com
[root@ipa1a ~]# ipa host-add frontend1a.example.com
Added host "frontend1a.example.com"
 Host name: frontend1a.example.com
 Principal name: host/frontend1a.example.com@EXAMPLE.COM
 Password: False
 Keytab: False
 Managed by: frontend1a.example.com

And finally create the NFS service for both with ipa service-add <service/host>

[root@ipa1a ~]# ipa service-add nfs/backend1a.example.com
Added service "nfs/backend1a.example.com@EXAMPLE.COM"
 Principal: nfs/backend1a.example.com@EXAMPLE.COM
 Managed by: backend1a.example.com
[root@ipa1a ~]# ipa service-add nfs/frontend1a.example.com
Added service "nfs/frontend1a.example.com@EXAMPLE.COM"
 Principal: nfs/frontend1a.example.com@EXAMPLE.COM
 Managed by: frontend1a.example.com

4. Configure your hosts in order to work with IPA server.

Now that your server is configured it’s time to bind the other hosts to Kerberos and let them use its authentication methods for you lab and tests.

Add 1GB of swap to your host (if you haven’t done that already), then configure your host in order to use use ipa1a.example.com (for me as dns resolver by updating /etc/sysconfig/network-scripts/ifcft-eth0 (or any other file) by disabling the PEERDNS option and adding DNS1 and DOMAIN entries, e.g.


Reboot your hosts and verify everything is working fine.

Perform the above steps on all the hosts you want to use for your RedHat 7 Lab (more likely frontend1a and backend1a).

Then verify name resolving is properly working (e.g. with dig ipa1a.example.com):

[root@backend1a ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search example.com

[root@backend1a ~]# dig ipa1a.example.com

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> ipa1a.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 779
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 4096
;ipa1a.example.com. IN A

ipa1a.example.com. 1200 IN A

example.com. 86400 IN NS ipa1a.example.com.

;; Query time: 1 msec
;; WHEN: Sat Aug 20 11:06:25 EDT 2016
;; MSG SIZE rcvd: 76

Now you can proceed by installing the ipa-client packages.

sudo yum install -y ipa-client ipa-admintools

Then proced with the configuration by executing

sudo -i
ipa-client-install --hostname=<hostname> --enable-dns-update

replace <hostname> with frontend1a.example.com or backend1a.example.com. It will be requested a user authorized to enroll computers, specify “admin” and then when prompted for the password type “password”

[root@frontend1a ~]# ipa-client-install --hostname=frontend1a.example.com --enable-dns-update
WARNING: ntpd time&date synchronization service will not be configured as
conflicting service (chronyd) is enabled
Use --force-ntpd option to disable it and force configuration of ntpd

Discovery was successful!
Client hostname: frontend1a.example.com
DNS Domain: example.com
IPA Server: ipa1a.example.com
BaseDN: dc=example,dc=com

Continue to configure the system with these values? [no]: yes
Skipping synchronizing time with NTP server.
User authorized to enroll computers: admin
Password for admin@EXAMPLE.COM:
Successfully retrieved CA cert
 Subject: CN=Certificate Authority,O=EXAMPLE.COM
 Issuer: CN=Certificate Authority,O=EXAMPLE.COM
 Valid From: Sat Aug 20 14:15:41 2016 UTC
 Valid Until: Wed Aug 20 14:15:41 2036 UTC

Enrolled in IPA realm EXAMPLE.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm EXAMPLE.COM
trying https://ipa1a.example.com/ipa/json
Forwarding 'ping' to json server 'https://ipa1a.example.com/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://ipa1a.example.com/ipa/json'
Systemwide CA database updated.
Added CA certificates to the default NSS database.
Missing reverse record(s) for address(es):
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://ipa1a.example.com/ipa/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring example.com as NIS domain.
Client configuration complete.

Now you can perform some check and verify the communications between the hosts and the server, from the host you can use “kinit admin” and access the ipa tools in the same way you can do on the server, so you can lookup and check services, e.g.

[root@frontend1a ~]# kinit admin
Password for admin@EXAMPLE.COM:
[root@frontend1a ~]# ipa service-find nfs
2 services matched
 Principal: nfs/backend1a.example.com@EXAMPLE.COM
 Keytab: False
 Managed by: backend1a.example.com

Principal: nfs/frontend1a.example.com@EXAMPLE.COM
 Keytab: False
 Managed by: frontend1a.example.com
Number of entries returned 2

5. Retrieve the keytab you will use for your Kerberized NFS exercises.

In this lab you can leverage on ipa tool and update your local copy of the /etc/krb5.keytab with the required principals. After the configuration, your hosts will have the principals for the host, but they will miss the ones for the NFS service, you will need to update your local /etc/krb5.keytab by running on every host the command below, on backend1a:

[root@backend1a ~]# ipa-getkeytab -s ipa1a.example.com -p nfs/backend1a.example.com -k /etc/nfs.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

while on frontend1a

[root@frontend1a ~]# ipa-getkeytab -s ipa1a.example.com -p nfs/frontend1a.example.com -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

(command will update the existing keytab)

It is possible to check the content of your /etc/krb5.keytab file with the command klist -k /etc/krb5.keytab that will provide the output below:

[root@frontend1a ~]# klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
 1 host/frontend1a.example.com@EXAMPLE.COM
 1 host/frontend1a.example.com@EXAMPLE.COM
 1 host/frontend1a.example.com@EXAMPLE.COM
 1 host/frontend1a.example.com@EXAMPLE.COM
 1 nfs/frontend1a.example.com@EXAMPLE.COM
 1 nfs/frontend1a.example.com@EXAMPLE.COM
 1 nfs/frontend1a.example.com@EXAMPLE.COM
 1 nfs/frontend1a.example.com@EXAMPLE.COM

6. Test your Kerberized NFS.

Now you can give NFS with Kerberos a try and verify if your LAB works.

On backend1a.example.com configure the NFS service in order to export a krb5p secured filesystem, e.g.

on backend1a.example.com:

sudo -i
mkdir /krbnfs
chown nfsnobody /krbnfs
echo "/krbnfs frontend1a.example.com(sec=krb5p,rw)" > /etc/exports.d/krbnfs.exports
systemctl restart nfs-server
firewall-cmd --permanent --add-server=nfs
firewall-cmd --reload

on frontend1a.example.com:

sudo -i
systemctl restart nfs-secure
mount -t nfs4 -o sec=krb5p backend1a.example.com:/krbnfs /mnt/
mount|grep krbnfs
backend1a.example.com:/krbnfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=,local_lock=none,addr=

Congratulations! Enjoy your kerberized RH7 Lab on AWS 🙂

PS: on older version of RH (7.0, not sure about 7.1) if was also required to enable and use nfs-secure-server on the NFS server while on latest builds (like the RH 7.2 we have on AWS) it is not required since GSSProxy is taking care of the “secure” jobs (see https://fedorahosted.org/gss-proxy/wiki/NFS for details).

PPS: this lab will work well also in order to test the LDAP based Authentication that is part of the RHCSA objectives

7. How to troubleshoot IPA/Kerberos.

If you have trouble and your Kerberized NFS exports don’t work, you may find helpful to check on ipa1a.example.com the /var/log/krb5kcd.log and see if odd requests / host / service are passing. It could help to spot misconfiguration / typos or other stuff. Good luck!