FreeIPA Architecture

& Troubleshooting

Martin Kosek <mkosek@redhat.com>

Supervisor, Identity Management Engineering
June 15, 2015
The Challenge
FreeIPA is not a monolithic application
Instead, many FOSS projects integrated together
Makes troubleshooting difficult, logging is not unified
Troubleshooting is not straightforward

Martin Kosek | Identity Management Team

FreeIPA Server
Core KRA Cert
BIND NTP
Dogtag SSSD mon
ger

Host User Group

FreeIPA Client
DNS RBAC DNS
KDC
Topo Cert HBAC
Apache Cert
SSSD mon
ger

LDAP FreeIPA Client

FreeIPA Server
Mandatory

Optional
Discussion
How are the parts integrated, how do they communicate?
Where all the components logs, how do we control them?

Martin Kosek | Identity Management Team

In detail - Apache
Host User Group
JSON-RPC /ipa/json
DNS RBAC DNS
XML-RPC /ipa/xml Browser
Topo Cert HBAC

ipaldap plugins Web UI /ipa/ui

AJP Proxy /ca/...

Apache

LDAP PKI

Martin Kosek | Identity Management Team

In detail - Directory Server
IPA Winsync IPA Pwd Extop ldappasswd

IPA Lockout LDAP BIND

IPA CLDAP

IPA enrollment ipa-join

IPA Sidgen

IPA Extdom SSSD

IPA CLDAP Active Directory

DB

Martin Kosek | Identity Management Team

Troubleshooting - the basics
Parts log in different places, in different format
Most useful logs:
Apache: /var/log/httpd/error_log, /var/log/httpd/access_log
Directory Server: /var/log/dirsrv/slapd-*/errors, /var/log/dirsrv/slapd-
*/access
Kerberos KDC: /var/log/krb5kdc.log
PKI: /var/log/pki/pki-tomcat/ca/system, /var/log/pki/pki-
tomcat/catalina.log, /var/log/pki/pki-tomcat/ca/debug
These can tell us the root cause in most cases

Martin Kosek | Identity Management Team

Troubleshooting - advanced
DNS: dig, host, tcpdump
Crashes, hangs
Mostly for C-based parts - Directory Server, BIND, SSSD
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debug_crash
es
Tools: gdb, strace, valgrind
Firewalls: nc
Specialized debugging for the components
http://www.freeipa.org/page/Troubleshooting
http://fedorahosted.org/sssd/wiki/Troubleshooting

Martin Kosek | Identity Management Team

Troubleshooting - DNS
DNS: Use dig to check response from the DNS server.
http://www.freeipa.org/page/Troubleshooting#DNS_Issues
http://www.freeipa.org/page/Troubleshooting#Failed_to_update_DNS_re
cords
Check if records in LDAP looks okay:
$ ipa dnszone-show <zone name>
$ ipa dnsrecord-show <zone name> <record name in the zone>
Restart BIND to force resynchronization with LDAP:
$ service named restart
In RHEL 7.2+ the service name is “named-pkcs11”
Check BIND logs if restart did not help:
Martin Kosek | Identity Management Team
Scenario: ipa cert-show 1
1
Dogtag DNS
5
6

3
7 Apache 2 $ ipa cert-show 1
Troubleshooting
KDC 1) host, ping, iptables
2) LDAP access log, BIND log
4 3) KRB5_TRACE=/dev/stdout, KDC log
LDAP
4) LDAP access log
5) Apache error log, ipa -vvv, strace
FreeIPA Server 6) Apache access log, PKI logs
7) LDAP access log

Martin Kosek | Identity Management Team

Scenario: ipa-server-install
3 1 Extra troubleshooting
Dogtag NTP 1) systemctl status ntpd.service
2) dirsrv errors log, systemctl status dirsrv-
6 7 REALM.service
3) /var/log/pki/pki-ca-spawn*,
Apache DNS /var/log/pki/pki-tomcat/catalina.2015-06-
12.log
5 4) dirsrv errors log
KDC 5) kdc errors log
2
6) Apache access log
7) systemctl status named.service
4 8 8) /var/log/ipaserver-install.log
LDAP Client 9) /var/log/ipaclient-install.log

FreeIPA Server

Martin Kosek | Identity Management Team

Scenario: ipa-replica-install
Most frequent problems:
DNS: make sure forward, reverse records are OK
Firewall: ipa-replica-conncheck was built for a reason
PKI: runs own API calls to remote CA, routed through proxy -
remember?
When installation crashes on PKI, do not look just at replica PKI
logs, check Apache errors/access log on remote master, PKI
status
And as always, keep the time in sync
Debug log in /var/log/ipareplica-install.log

Martin Kosek | Identity Management Team

Scenario: Trust setup
Rule #1:
Always consult
https://www.freeipa.org/page/Active_Directory_trust_setup
It is awesome!
Most frequent problems:
DNS: AD&IPA cannot see each other DNS SRV records
Firewall - shut it down for testing

Martin Kosek | Identity Management Team

Scenario: AD user resolution
id aduser@ad.domain.test does not show secondary groups,
why?
This is a complex setup, many parts in play:
Client where “id” is run
Remote FreeIPA server with DS extended call support
Remote FreeIPA server SSSD - does the real heavy-lifting

Martin Kosek | Identity Management Team

References
Troubleshooting information
https://www.freeipa.org/page/Troubleshooting
https://fedorahosted.org/sssd/wiki/Troubleshooting
Architecture, designs
http://www.freeipa.org/page/V4_Designs
https://fedorahosted.org/sssd/wiki/DesignDocs
User docs
http://www.freeipa.org/page/Documentation#User_Guides
Blogs
http://planet.freeipa.org/

Martin Kosek | Identity Management Team

FreeIPA Client
SSSD
 Anatomy of SSSD
nss_sss
NSS
Client Responder
Identity Identity
“id”
Provider Server

Network Boundary
Backend
Cache
Process
nss_sss
Client
“login” PAM Auth Authentication
Responder Provider Server

SSSD

Martin Kosek | Identity Management Team

The Challenge
Collection of parallel processes, communicating with each
other
Finding a bug may be a challenge, which part to debug and
look at?

Martin Kosek | Identity Management Team

 Scenario: User lookup 1/2
1
nss_sss
NSS 3
Client 7 Responder 4
Identity Identity
“id” 6 Provider Server
2

Network Boundary
5
Backend
Cache
Process
nss_sss
Client
“login” PAM Auth Authentication
Responder Provider Server

SSSD

Martin Kosek | Identity Management Team

Scenario: User lookup 2/2
As you see, many processes are in play
Troubleshoot tips
Go with the data flow, enable debugging in sssd.conf
In this case, we would start with adding “debug_level = 7” to [nss]
If this does not help, add it to respective [domain/*]

Martin Kosek | Identity Management Team

Discussion
Do you know the materials that can be used for learning SSSD
anatomy and troubleshooting?
The scenario:
https://jhrozek.wordpress.com/2015/03/11/anatomy-of-sssd-user-looku
p/
SSSD troubleshooting:
https://fedorahosted.org/sssd/wiki/Troubleshooting

Martin Kosek | Identity Management Team

 Scenario: User Authentication 1/2
nss_sss
NSS 3
Client Responder 4
Identity Identity
“id”
Provider Server
2

Network Boundary
5
Backend
1 Cache
Process
nss_sss

pam_sss
PAM Auth Authentication
Client Responder Provider Server
6 7 8
“login”
SSSD

Martin Kosek | Identity Management Team

Scenario: User Authentication 2/2
User must be resolvable with id or getent passwd in order for
authentication to succeed at all
Start with /var/log/secure to see what pam_sss returned
Then continue with the data flow
First enable the debugging in the PAM responder
Then in the domain section and the authentication helpers (krb5_child)
It may be useful to perform similar authentication request manually (kinit)
SSSD only handles PAM-based authentication. Some
authentication methods (GSSAPI, public keys) bypass SSSD

Martin Kosek | Identity Management Team

Advanced Troubleshooting
debug_level can be changed on the fly with sss_debuglevel
Online/offline status can be changed as well
kill -USR1/-USR2 $(pidof sssd)
Useful if sssd switches to offline mode as a fallback
See troubleshooting document to see how to run SSSD
together with valgrind or strace
Invalidate cache with sss_cache to make sure sssd reaches
the back end and try to test while SSSD is not (yet) offline
Make sure to record times of lookups/authentication when
performing tests
Very useful to cross-reference logs from different machines

Martin Kosek | Identity Management Team