PAM Administration

Troubleshooting

© 2023 CyberArk Software Ltd. All rights reserved

 Agenda
By the end of this session, you will be able to:
1. Describe the basic flow for troubleshooting
issues in the CyberArk environment

2. Describe, locate, and manage the log files

generated by the Vault and various
components

3. Describe, configure and use the xRay agent

© 2023 CyberArk Software Ltd. All rights reserved

 Troubleshooting Flow

© 2023 CyberArk Software Ltd. All rights reserved

 Overview
The basic troubleshooting methodology for the PAM solution
requires a thorough understanding of:
1. Your system implementation
2. How components communicate with each other
in your environment
3. What is the current behavior compared to the
expected behavior?
This methodology is designed to provide guidance
and might not apply to every scenario

It is important to write down any information gathered

during this process and any tests performed, as all of this
information will be required when opening a case with
CyberArk support

© 2023 CyberArk Software Ltd. All rights reserved

 1. Knowledge of the environment layout

2. Access to the different servers

3. Access to CyberArk Knowledgebase

Prerequisites (Customer Community)

4. Access to CyberArk documentation

(publicly available online)

The latest version of the documentation will

contain the most recent enhancements and notes.

© 2023 CyberArk Software Ltd. All rights reserved

 Troubleshooting Flow
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
Here is an overview of the basic steps and
of the troubleshooting flow knowledgebase

During this presentation we will review each block

on the flow and apply it to a basic scenario Contact
Support

© 2023 CyberArk Software Ltd. All rights reserved

 Understanding the Environment
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
• Which components are installed and where? knowledgebase

• What is the version of the relevant components?

• Is a Load Balancer being used? Contact

Support
• Are DR or HA solutions implemented?

© 2023 CyberArk Software Ltd. All rights reserved

 Initial Questions
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

• User experience? Check

documentation
• Affected users? and
• Error message displayed? knowledgebase

• New implementation or worked and broken?

• Something changed when this issue started?
Contact
• Was there a process crash? Support
• How does it impact production?
• Reproducible?
© 2023 CyberArk Software Ltd. All rights reserved
 Isolation and Reproduction
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Reproducible Check
documentation
• Modify a variable and try to reproduce again. and
• Repeat in different scenarios knowledgebase
• Write down each scenario and the outcome of the test
• Review the logs of reproduced scenarios (working and
not working) Contact
Not reproducible Support

• Review the logs relevant for the reported flow

© 2023 CyberArk Software Ltd. All rights reserved

 Checking the Logs
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
• Log location knowledgebase
• Log types
• Log correlation
Contact
Support

© 2023 CyberArk Software Ltd. All rights reserved

 Follow-Up Questions
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase
• Review and refine questions

Contact
Support

© 2023 CyberArk Software Ltd. All rights reserved

 Documentations and Knowledge Base
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
• Colleagues and end users documentation
and
• Knowledge base knowledgebase
• Messages and Responses
document; Installation and
implementation Contact
documents Support
• Re-run scenarios

© 2023 CyberArk Software Ltd. All rights reserved

 Contacting CyberArk Support
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

• Environment details? Check

• User experience? documentation
and
• Did it work in the past? knowledgebase
• Are there any error messages?
• Flow, current and expected behavior?
• Troubleshooting steps? Contact
Support
• Steps to reproduce this issue?
• All relevant logs, screenshots and configuration files

© 2023 CyberArk Software Ltd. All rights reserved

 Troubleshooting Flow:
Example

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
A user is unable to login to the PrivateArk client using the administrator user.

They see the following

message.

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Understand the Environment
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase
• 1 Vault Prod
• 1 DR vault
• 1 PVWA, CPM, PSM Contact
• All running with Version 12.6 on Windows 2019 servers Support

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Initial Questions
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
• Is this issue experienced by all users? One user knowledgebase

• Did it work before? Yes

• Was something changed? No Contact

Support
• Is there any error message? Yes

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Isolation and Reproduction
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase
• Same issue via PVWA? Yes

• Reproducible? Yes
Contact
Support

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Checking the Logs
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase

Error origin
ITA Origin is vault
ITATS004E
Contact
Support
Vault logs:
• ITAlog.log
© 2023 CyberArk Software Ltd. All rights reserved
• Trace.d0
 Check Messages and Responses
Try to identify the problem by searching in the Messages and Responses page in on the online
documentation

© 2023 CyberArk Software Ltd. All rights reserved

 Check Messages and Responses
Messages displayed to end users are intentionally generic, listing many possible causes.

© 2023 CyberArk Software Ltd. All rights reserved

 Check Messages
and Responses
Because the error message starts with
ITA, we know that the Vault server
originated this error.
• At this point we will go to the Vault
server and inspect the ITA log.
• There may be multiple log entries for
the same problem.
• Try to find the first entry related to
this problem
• When looking at the ITA log, we see
an error message ITATS528E with a
code of 66
• When we search for that error, we
see the exact cause of the problem
and the solution.

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Solution
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase
Does resetting the user
password solve the problem? Yes (solved)

Contact
Support
No

© 2023 CyberArk Software Ltd. All rights reserved

 User Unable to Login
Problem Not Resolved
Isolate the issue
Understand the Initial questions,
to specific Check relevant Follow-up
environment’s focus on user
scenario by trying logs questions
topology experience
to reproduce

Check
documentation
and
knowledgebase
In the event of another login failure:
• Check the relevant logs again – same error or a new one?
• Repeat the troubleshooting flow Contact
• Contact support when no more logical steps are found Support

© 2023 CyberArk Software Ltd. All rights reserved

 Logs

In this section we will discuss the logs

generated by the various system
components, how to set the debug
mode, and the logs location

© 2023 CyberArk Software Ltd. All rights reserved

 Overview

© 2023 CyberArk Software Ltd. All rights reserved

 Types of Logs
Log files are divided into several types:

Console Provides component-level entries Trace Provides detailed entries of workflows

Log such as service up or down Log related to that component

Error Exists in some components, Debug Those logs may come in different types,
Log and will include only error entries Log sometimes they will be the trace files, with
additional information and sometimes
they will come at a form of separate files
depending on the component.
For the full list of log locations, please
see the implementation guide

© 2023 CyberArk Software Ltd. All rights reserved

 Understanding CyberArk Logs
The log message code is built from four segments
for example:
ITA – The source component of the message is the Vault server
ITA FW 001 I FW – The module with the message is the Vault FW
Firewall is open for client communication 001 – Message number
I – The message category

Log messages are separated into four major categories:

Informational: Error:
ITAFW001I Firewall is open for client communication ITATS691E LDAP synchronization error

Warning: System:
ITATS319W Firewall contains external rules ITADB367S Server unable to communicate with firewall

See CyberArk Messages and

Responses for additional information
© 2023 CyberArk Software Ltd. All rights reserved
 Reviewing the Logs
Once we get to a point where we need to go over log files, there
are a number of questions to ask:

• Which log file do we need to review?

• What do we search for?
⎼ Keywords (Error, Failed, Failure…)
⎼ Timestamps
⎼ User name
⎼ Object name (Account name, safe name)
• Are there correlated entries in other logs?
⎼ Log events and time of the issue
⎼ Different components
⎼ CyberArk logs and OS logs

© 2023 CyberArk Software Ltd. All rights reserved

 Debug Mode and Log Location

© 2023 CyberArk Software Ltd. All rights reserved

 Set the Debug Mode
for the Vault ITAlog
• The Vault debug levels can be
changed in the dbparmi.ini file
(requires a restart)
• The Vault debug levels can be
changed without a restart using the
PARclient or Central Administration
Station

© 2023 CyberArk Software Ltd. All rights reserved

 Set the Debug Mode for the Components
Debug mode for components can be set in the configuration files stored on the Vault or
via the PVWA Web UI

Set the debug

level for CPM
Set the debug
level for PSM

© 2023 CyberArk Software Ltd. All rights reserved

 Log Locations and Configuring the Debug Levels
Detailed information about setting debug level for different components and location of the log files
can be found in the online documentation

Setting Vault log levels to

Debug should only be
done under the guidance
of CyberArk Support

© 2023 CyberArk Software Ltd. All rights reserved

Cheat Sheet – Vault and Related Components
Vault Changes Require a Vault Restart Ene Event Notification Engine
Configuration Configuration \Program Files\PrivateArk\Server\Event Notification
DBParm.ini File Engine\ENEConf.ini
File
…\Database\my.ini. - Database Configuration File Vault ➔Safe:”Notification Engine”➔root\EventNotificationEngine.ini

Debug EventNotificationEngine.ini
Debug DebugLevel=PE(1),PERF(1) - Detailed Vault services debug
[Debug]
• ControllerDebugLevel=1,2,3,4
LDAP(14,15) - Detailed LDAP debug • CollectorDebugLevel=1,2
• ParserDebugLevel=1,2
Logs Italog.log • SMTPSenderDebugLevel=1,2
Disaster Recovery • ConfigurationManagerDebugLevel=1,2
Trace.dX (X is a number from 0 to 4)
Configuration File PADR.ini Logs ProgramFiles\PrivateArk\Server\EventNotification
Engine\Logs\ENEConsole.log
…\Database\VaultDB.log - Database log
Debug EnableTrace=yes ProgramFiles\PrivateArk\Server\EventNotification
Engine\Logs\ENETrace.log
Logs PADR.log
Logic Container

File Name LogicContainer.Log Client Run –PAInfo.exe

C:\ProgramFiles Debug
Logs In the Client:
(x86)\PrivateArk\Server|LogicContainer\LogicContainer.log
Tools ➔ Options ➔ Advanced ➔ Log Configuration

PAReplicate Backup and Restore Logs (Win XP \Documents and Settings\<user>\Application Data\CyberArk\PrivateArk\PALog.txt
Debug In the PAReplicate.exe command executed, add the and Win 2003
following flag: /EnableTrace
Logs (Win7 and \Users\<user>\AppData\Roaming\CyberArk\PrivateArk
Win 2008
Logs PAReplicate.log

© 2023 CyberArk Software Ltd. All rights reserved CLICK “NEXT” TO CONTINUE
Cheat Sheet – Components PSM Privileged Session Manager
Configuration \Program Files\CyberArk\PSM\Basic_psm.ini
File
PVWA ➔ Administration Tab ➔ Options ➔ Privileged Session
Management
Debug PVWA ➔ System tab ➔ Options ➔ Privileged Session Management ➔
CPM Central Password Manager General Settings

Configuration Vault ➔ Safe “Password Manager”➔ root\policies\<policy>.ini Server Settings ➔ TraceLevels=1,2,3,4,5,6,7

File Recorder settings ➔ TraceLevels=1,2
Debug PVWA ➔ Administration Tab ➔ CPM settings Connection Client Settings ➔ TraceLevels=1,2

Logs <installation folder>\Logs (and subfolders) or according to parameter

CPMDebugLevels=2 (default) “LogsFolder” (located in Basic_psm.ini file)
0 – No messages will be written to the trace log.
1 – CPM exceptions will be written to the trace log (Default Level)
2 – CPM trace messages will be written to the trace log. PVWA Password Vault Web Access
3 – CPM CASOS activities will be written to the trace log. Configuration
4 – CPM CASOS debug activities will be written to the trace log. \wwwroot\PasswordVault\web.config
File
5 – CPM CASOS errors will be written to the trace log.
6 – All CPM CASOS activities and errors will be written to the trace log. Vault ➔ Safe “PVWAConfig” ➔ root\PVConfiguration.xml

Vault ➔ Safe “PVWAConfig” ➔ root\Policies.xml

Debug PVWA ➔ Administration Tab ➔Options ➔ Logging

Logs – CPM \Program Files\CyberArk\PasswordManager\Logs\pm.log
\Program Files\CyberArk\PasswordManager\Logs\pm-error.log\Program DebugLevel=High (options are None/High/Low/Profiling)
Files\CyberArk\PasswordManager\Logs\PMConsole.log\Program
InformationLevel=High (options are None/High/Low/Profiling)
Files\CyberArk\PasswordManager\Logs\PMTrace.log
Logs %windir%\temp\

Logs –Plug-ins \Program Files\CyberArk|passwordManager\Logs\ThirdParty\*.log CyberArk.Webapplication.log

CyberArk.WebConsole.log

CyberArk.WebSession.<Sessionid>.log

© 2023 CyberArk Software Ltd. All rights reserved CLICK “NEXT” TO CONTINUE
 In this section we will discuss the
CyberArk xRay utility, which can be
xRay Agent used to collect log and configuration
files from the CyberArk components
and share them with CyberArk or
partner support

© 2023 CyberArk Software Ltd. All rights reserved

 Overview
CyberArk xRay collects logs and configuration
files from PAM components in a simple, single-
step process
• The utility can be run from a remote machine or
on any of the CyberArk servers
• All data files are encrypted during collection,
regardless of whether they are collected locally or
remotely, and then transferred back to the xRay
machine
• You can share the collected data with your partner
or CyberArk, knowing that it is safely encrypted
during transfer The utility can be downloaded
• When sharing with CyberArk, shared data is from the CyberArk Marketplace
linked to a case to allow Enterprise Support easy
and secure access to the collected data

© 2023 CyberArk Software Ltd. All rights reserved

 Agent Setup

• Select the component

• Select time frame for the collection and
collection level.
• Select Collection scope
⎼ Logs from OS and the application
⎼ Logs from application only
• Optionally, enable and provide the Active
Vault IP address and Administrative user
credentials for configuration files collection
• Agree to the Terms of Use and click Start
Collection

© 2023 CyberArk Software Ltd. All rights reserved

 Monitor Collection
Process
You can monitor the collection
process as it collects the component
files

© 2023 CyberArk Software Ltd. All rights reserved

 Share the
Collected Data
• Once the process is complete, you
can select whether to:
⎼ Share the collected data with your
Partner
⎼ Share the collected data with
CyberArk
• You can also preview the data before
sending
• When sharing information with
CyberArk, make sure you have:
⎼ A Technical Community account
⎼ Case number

© 2023 CyberArk Software Ltd. All rights reserved

 Documentation
Additional information can be found in the CyberArk documentation

© 2023 CyberArk Software Ltd. All rights reserved

 Summary

© 2023 CyberArk Software Ltd. All rights reserved

 In this session we covered:

• The basic flow for troubleshooting

issues in the CyberArk environment
Summary • How locate and manage the log files
generated by the Vault and various
components

• How to configure and use

the xRay agent

© 2023 CyberArk Software Ltd. All rights reserved

 Utilities
xRay (login required)

Additional Community Resources

Resources CyberArk Customer Community (login required)

CyberArk Subreddit Note: The CyberArk subreddit is not

hosted or moderated by CyberArk.

Online Training
Working with CyberArk Support

© 2023 CyberArk Software Ltd. All rights reserved