sys_diag: Configuration and performance snapshot, analysis, and reporting utility

Description: sys_diag is a Solaris utility (ksh script) that can perform several
functions, most notably, system configuration 'snapshot' and reporting
(detailed or high-level) plus workload characterization via performance
data gathering (over some specified duration or time in point 'snapshot'),
high-level analysis, and reporting of findings/exceptions (based upon
perf thresholds that can be easily changed within the script header).

The output is provided in a single .tar.Z of output and corresponding
data files, and a local sub-directory where report/data files are stored.
The report format is provided in .html, .txt, and .ps as a single file
for easy review (without requiring trudging through several subdirectories
of separate files to manually correlate and review).

sys_diag runs on any Solaris 2.6 (or above) Sun platform, and even
includes reporting of new Solaris 10 capabilities (zone/containers, SVM,
zfspools, fmd, ipfilter/ipnat, link aggr, Dtrace probing, etc...).

Beyond the Sun configuration reporting commands [System/storage HW config,
OS config, kernel tunables, network/IPMP/Trunking/LLT config, FS/VM/NFS,
users/groups, security, NameSvcs, pkgs, patches, errors/warnings, and
system/network performance metrics...], sys_diag also captures relevant
application configuration details, such as Sun N1, Sun Cluster 2.x/3.x,
Veritas VCS/VM/vxfs.., Oracle .ora/listener files, etc.. detailed
configuration capture of key files (and tracking of changes via -t), etc ...

Of all the capabilities, the greatest benefits are found by being able
to run this single ksh script on a system and do the analysis from one single report/
file... offline/elsewhere (in addition to being capable of historically
archiving system configurations, for disaster recovery.. or to allow for
tracking system chgs over time.. after things are built/tested/certified).
One nice feature for performance analysis is that the vmstat and netstat
data is exported in a text format friendly to import and created graphs
from in StarOffice or Excell.. as well as creating IO and NET device
Averages from IOSTAT / Netstat data (# IO's per device, AVG R/W K, etc..)
along with peak exceptions for CPU / MEM / IO / NET ..

Although I'm a Sun employee, this has been personally developed over many
years, in my spare time in order to make my life a lot easier and
more efficient. Hopefully others will find this utility capable of
doing the same for them, also making use of it's legwork.. to streamline
the admin/analysis activities required of them. This has been an invaluable
tool used to diagnose / analyze hundreds of performance and/or configs issues

Regarding the system overhead, sys_diag runs all commands in a serial
fashion (waiting for each command to complete before running the next)
impacting system performance the same as if an admin were typing these
commands one at a time on a console.. with the exception of the background
vmstat/mpstat/iostat/netstat that's done when (-g) gathering performance
data over some interval for report/analysis (which generally has minimal
impact on a system, especially if the sample interval [-I] is not every
second).

sys_diag is generally run from /var/tmp as "sys_diag -l" for creating
a detailed long report, or via "sys_diag -g -l " for gathering
performance data and generating a long/detailed config/analysis report),
however offers many command line parameters documented within the header,
or via "sysdiag -?".

Contact: Todd Jobson - todd.jobson [ at ] sun.com

Submitter: Todd Jobson

Collections/Keywords:
performance, networking, configuration, solarisx86, solaris, solaris10, solaris, configuration, diagnosis, performance, data, script.dtrace