Applies to:

Oracle Database - Enterprise Edition - Version 8.1.5.0 to 11.2.0.3 [Release 8.1.5 to 11.2]
Information  in this document applies to any platform.

Purpose

The purpose of the note is to help customers troubleshoot Virtual Circuit Waits.

Troubleshooting Steps

 

What is a virtual circuit ?

Virtual circuit is a piece of shared memory used by the dispatcher for client database connection requests and replies.

The dispatcher places a virtual circuit on a common queue when a request arrives. An idle shared server picks up the virtual circuit from the common queue, services the request, and relinquishes the virtual circuit before attempting to retrieve another virtual circuit from the common queue.

In the shared server architecture, clients connect to a dispatcher process, which creates a virtual circuit for each connection. When the client sends data to the server, the dispatcher receives the data into the virtual circuit for the relevant connection and places the now active circuit on the common queue to be picked up by an idle shared server. The shared server then reads the data from the virtual circuit and performs the database work necessary to complete the request. When the shared server must send data to the client, the server writes the data back into the virtual circuit and the dispatcher sends the data to the client. After the shared server completes the client request, the server releases the virtual circuit back to the dispatcher and is free to handle other clients.

What are Virtual Circuit Waits?

Virtual Circuit status wait represents the time session waits for a virtual circuit operation to complete.

There are 2 parameters:

• circuit#  - Indicates the virtual circuit# being waited on
• type - Indicates the type of operation the session is waiting for

Before 11.1.0.7 and 11.2.0.1, this wait was included in the wait event: 'virtual circuit status'.
This wait event was used for the shared server idle wait as well as  for waits when working on a circuit. This made it difficult to determine if there was indeed a problem waiting for a virtual circuit or not as the idle time dominated the total time for the event obscuring any true issues. The 'virtual circuit status' wait event has therefore been removed and replaced by two new wait events:

• "Virtual circuit wait" indicating when the shared server is blocked waiting on a specific circuit or message.

• "Shared server idle wait" indicating when when the shared server is idle waiting for something to do.

 

Causes for Virtual Circuit Waits

Virtual Circuit Waits may be a result of :

• Configuration
  We need to determine whether there are enough dispatchers for traffic and then whether there are enough shared servers for the number of dispatchers.
  For more information on the architecture of MTS and diagnostics on configuration see:
  
  
  Document:1005259.6 Shared Server (MTS) Diagnostics
• Performance
  With shared servers, the time a single process spends in a single call in a server can make a big difference to the end-users. It can end up in a state where every server is executing a long running call (for example , an extremely large sort), and no new calls for other users can be serviced.

Diagnostics for Virtual Circuit Wait

Obtain half an hour to an hour AWR report at the time of wait:

The above example shows 80.1% of the database time waiting for 'virtual circuit wait'.

In 11.2, the AWR report includes sections relating to Shared Server configuration and usage that may assist in diagnosing a bottleneck in the way Shared Servers have been configured:







Above diagrams show some of the information as seen in the AWR Report in 11.2.
In the above snippets, one can determine how busy the servers and dispatchers are and whether you would benefit by adding more resources here. The above example shows Shared servers as being 97.09% idle and Dispatchers being 99.78% idle so that it does not look like there is a problem with the amount of resources allocated for the Shared Servers.

More Background Information to collect

Having eliminated any problems related to configuration of the Shared Servers, it is useful to obtain the following information in order to troubleshoot the reason for the waits.

•  How many connections are there?
•  What is the status of the connections (idle or active)?
•   Is DCD (Dead Connection Detection) enabled?
    DCD cleans up OS and database processes  that have disconnected and terminated abnormally
  
  Document: 151972.1  Dead Connection Detection (DCD) Explained

Potential Problems and Solutions

Long running shared server connections

Both idle and active connections can cause high "virtual circuit wait".

• Idle connections
  
  
  1. Even if DCD (dead connection detection) is enabled, IDLE connections may not be cleaned up.  'IDLE / ABANDONED / INACTIVE sessions for OS processes will not be cleaned up even if DCD and Database Resource Limits + user Profiles are used in combination. These must be cleaned up manually.'  as explained in:
     
     
     Document:601605.1 A discussion of Dead Connection Detection, Resource Limits
  2. WORKAREA_SIZE_POLICY set to automatic
     Shared Servers may not be released when WORKAREA_SIZE_POLICY set to automatic.
     
     As of 10g, certain operations (like SORT) can intentionally spill into the PGA of the shared server rather than taking up space in shared memory. 9i behaviour can be restored by setting WORKAREA_SIZE_POLICY=MANUAL.
     
     When data spills to the PGA the session has to stay tied to that specific shared server. Sessions performing large sort operations are not good candidates for shared server connections and are better suited to dedicated connections.
     
     
     Bug 5689608: INACTIVE SESSION IS NOT RELEASING SHARED SERVER PROCESS (closed as not bug)

 

• Active Connections
  
  Check what the ACTIVE sessions are doing. Are they waiting for a resource held by other connections or are they consuming high resources.
  
  Use the AWR report to diagnose whether there are other performance issues contributing to poor performance that may result in the shared server processes not releasing the shared server and hence resulting in many sessions waiting for a virtual circuit to become available.
  
  See
  
  Document: 1359094.1  FAQ: How to Use AWR reports to Diagnose Database Performance Issues

 

Known Issues

 

Recommended Diagnostics  when opening a Service Request

• It may be useful to get a Systemstate Dump if possible:
  
  
  Document Doc ID 125568.1 SYSTEMSTATE DUMPs: An example of how to take and upload them

• If this is not an option , 10046 traces of waiting sessions may also help in diagnosing the issue
  
  
  Document 1274511.1 General SQL_TRACE / 10046 trace Gathering Examples

• AWR report covering a period of between 30-60 minutes
  
  
• Shared Server Diagnostics
  
  
  Document 981615.1 Diagnostics Available for Shared Server (Formally known as MTS)

• Shared Server Configuration script
  
  Download the following script to gather more diagnostics for shared server configuration and performance:
  
  shared_servers_check.sql

Troubleshooting Other Issues

For guidance troubleshooting other performance issues see: