CoolThreads - CMT Tuning and Resources

LDAP Performance Tuning

Sun Directory server / LDAP Performance Tuning

What’s Directory server / LDAP

• A high-speed, network-based, hierarchical data store, optimized for fast retrieval of data.
• Hierarchical data store in tree structure, unlike table based SQL database.
• Allow multiple values for an attribute. For example, a person can have many phone numbers, without a fix limit of phone numbers.
• Access control can be down to attribute level. For example, attribute phone number can be modified by yourself but not by others.

Benefit of Using Directory Server/LDAP

• Provides one central location to store user information.
• No need to update every application when user info changes.
• Users can maintain a single identity used by multiple applications.
• Enables changes to be distributed and handled by appropriate personnel.
• Permits users to change their own information, reducing administration workload.

Common Usage of LDAP

• Authentication
• Web service access control
• Mail server
• user login & routing information
• Web service Personalization
• MyAOL
• MyYahoo
• And many more

Tuning - Cache

• DB Cache
• The cache for internal embedded database, Berkeley-DB.
• The Database cache affects search and update performance. In most cases, increasing the size of the Database Cache will have a noticeably positive effect on update performance.
• This change requires a server restart.
• nsslapd-dbcachesize in "cn=config,cn=ldbm database,cn=plugins,cn=config"
• Entry Cache
  
  • The Entry Cache can have a very positive effect on search performance and should be tuned as large as possible.
  • Tune the nsslapd-cachememsize in "cn=<backend>,cn=ldbm database,cn=plugins,cn=config".
  • After search, read monitor entry "cn=monitor,cn=<backend>,cn=ldbm database,cn=plugins,cn=config".
  • The "currententrycachesize" shall be less than 95% of "maxentrycachesize" and high "entrycachehitratio".

Tuning – Files and Disks

• nsslapd-db-home-directory
• The location of the dB memory mapped cache files.
• Put this in /tmp as it is only a cache and won't affect the integrity of the database.
• Distribute Disk Activity
• Directory Server can be extremely disk I/O intensive, especially in systems supporting a large number of updates.
• Consider putting each database, the transaction logs, and access log on their own disks with separate controllers.
• Use NVRAM write cache storage if available.

Tuning – Operation Threads

• nsslpad-threadnumber
• The number of operation threads that the Directory Server will create during startup.
• The nsslpad-threadnumber value should be increased if you have many directory clients performing time-consuming operations such as add or modify. This ensures that there are other threads available for servicing short-lived operations such as simple searches.
• Should be increased if I/O generating long ops (read very large db) and there is a queue and many cpus are idle.
• Use "request-que-backlog" in cn=monitor to determine if threads should be increased. The request-que-backlog should be close to zero.
  

Tuning – Search Behavior

• ALL Ids threshold
• A threshold of the number of “index entries” for the server to switch to linear search. For example, more than 4000 entries matches for surname=chen, server switch to linear search.
• Setting the ALL IDS Threshold too high or too low can have negative effect on performance. The default is 4000. If an index entry is over this threshold, go to linear search code path.
• Need to re-index database after tune.
• nsslapd-allidsthreshold in "cn=config,cn=ldbm database,cn=plugins,cn=config".
• Disable non-indexed Searches
  
  • non-indexed searches can potentially consume large amounts of server resources and have a negative affect on overall performance. Consider enabling the nsslapd-require-index attribute to make the server reject non-indexed searches all together.
  • If an non-indexed search is required by some application, add an index to resolve this problem.
• Avoid subtree searches that match a large number of entries.

Tuning – Checkpoint and Update

• Longer checkpoint period, better performance for update performance.
• The checkpoint is for database recovery, if you do just read, make it longer will perform better.
• Set nsslapd-db-checkpoint-interval in "cn=config,cn=ldbm database,cn=plugins,cn=config" to 180 or larger.
• Plugins and Indexing for update performance.
• Many customers use the Referential Integrity and UID Uniqueness plug-ins. These plug-ins conduct equality searches on several attributes across the entire directory server. Make sure the proper equality indexing is set when these plug-ins are enabled.

Tuning – Customize Indexing

• Add Additional Indexes

• It may become necessary to add additional indexes to Directory Server over time as new applications begin to use the infrastructure.
• While indexes can be added, removed or modified while the server is still running, adding a new index will not index existing data. Only new data added to the directory will be indexed from that point forward.
• To realize the full benefits of the new index, rebuilding the index file will be necessary. This can be done by re-importing the database from LDIF or by using the db2index.pl program included with the server.
• it is sometimes faster to re-import the entire database than to re-build a single index.

Tuning – Limit the Expensive Operations

• Set Operational Limits
• Set the operational limits such as nsslapd-lookthroughlimit, nsslapd-maxsize, etc. to keep the server from spending too much time on certain operations.
• If certain clients or applications need increased capabilities, assign a unique binding DN and set the resource limits for these DNs to their required limits.
• nsslapd-sizelimit
• Specifies the maximum number of entries to return from a search operation.
• As soon as this limit is reached, the server returns an error message: Exceeded size limit error. The default value is 2000. A value of -1 means no limit, which is not recommend.
• nsslapd-lookthroughlimit
  
  • Specifies the maximum number of entries that Directory Server checks in response to a search request. If you bind as the Directory Manager, unlimited is the default and overrides other settings. However, if you set a a value for the operational attribute nsLookThroughlimit in the entry for binding, that value overrides the default. A value of -1 means no limit, which we do not recommend.
• nsslapd-timelimit
  • Specifies the search time limit for the database link in seconds. The default is 3600. A value of -1 means no limit, which we do not recommend.

Access Log

• Logging Levels  -- nsslapd-accesslog-level
• 0=No access logging
• 4=Logging for internal access operations
• 256=Logging for access to an entry (default)
• 512=Logging for access to an entry and referrals
• 131072=Precise timing of operation duration. This gives microsecond resolution for the Elapsed Time item in the access log.
• The higher the log level the more impact on performance, takes more time to produce log information

• Example

• Method Type
  bind method was used by the client. There are three possible bind method values:
• 0 = no authentication
• 128 = simple bind with user password
• sasl= SASL bind using external authentication mechanism
• Tag Number: indicate the type of result returned, which is almost always a reflection of the type of operation performed.
  tag=97 for a result from a client bind operation
  tag=100 indicates the actual entry for which you were searching
  tag=101 for a result from a search operation
  tag=103 for a result from a modify operation
  tag=105 for a result from an add operation
  tag=107 for a result from a delete operation
  tag=109 for a result from a moddn operation
  tag=111 for a result from a compare operation
  tag=115 indicates a search reference when the entry you perform your search on holds a referral to the entry you require. Search references are expressed in terms of a referral.
  tag=120 for a result from an extended operation
• Elapsed Time
  
  • Indicates the amount of time (in seconds) that it took the directory server to perform the LDAP operation. An etime value of 0 means that the operation actually took milliseconds to perform.
  • If many etime > 0, indicates bad performance.
  • If you want to have microsecond resolution for this item in the access log, enter a value of 131328 (256+131072) in the nsslapd-accesslog-level configuration attribute.
• Un-indexed Search Indicator
  
  • The un-indexed search indicator, notes=U, indicates that the search performed was un-indexed, which means that the database itself had to be directly searched instead of the index file.
  • Un-indexed searches occur either
  • when the All Ids Threshold was reached within the index file used for the search
  • when no index file existed or when the index file was not configured in the way required by the search.
  • If there are un-indexed searches in the log consider
  • Adjust All Ids Threshold
  • Add new index

LDAP Benchmark Tools

• LoadRunner
• Directory server resource kits
• rsearch
• serachrate
• DirectoryMark
• http://www.mindcraft.com/directorymark/
• SLAMD
• http://www.slamd.com/

Additional Tips for Tuning

• Run idsktune for tuning recommendation.
• Avoid subtree search if using base search is good enough
• Open connection is very expensive. Try to reuse opened connection as much as possible
• Disable DHCP on server
• Stress test may run out of anonymous ports, reduce tcp_time_wait_interval according to your network speed.
• The socket listen size was hard-coded to 50, may fail at high rate of opening new connections. Fixed in the next Directory server release.
• ACI impacts performance also.

NFS

NFS Configuration Notes

For throughput oriented applications where the objective is to maximize the data flowing out or into the Sun Fire CoolThreads server, there are 2 subsystems to consider:

• The disk subsystem:
  • Make sure that a local file system can support the bandwidth rates that you would like to see. Add more disks/LUNs/HBAs until the bandwidth requirements are met.
• The NFS subsystem:
  • The number of NFS threads is controlled by the NFSD_SERVERS parameter in /etc/default/nfs. You need to increase this parameter while monitoring the parallelism in NFS using

    prstat -c -p `pgrep nfsd`

    If the number of lwps reported is close to the limit of what is specified in /etc/default/nfs you need to increase this limit. Cranking this up to a high value is not very harmful since it is only an upper bound. Only the minimum number of threads needed to handle the incoming requests will be created by the NFS subsystem.

Symantec NetBack-Up

Edit the "/etc/system" file with the following statement:
set ip:ip_soft_rings_cnt=8

If you have a 4 core or 6 core system, set the statement to represent the number of cores available, ie:
set ip:ip_soft_rings_cnt=4
set ip:ip_soft_rings_cnt=6