How to Write Application Performance Agents in TeamQuest Performance Software 7.1

Note

This paper has been revised for TeamQuest Performance Software 7.2 and 8. If you are using these newer versions, please see the revised version of this paper.

TeamQuest Performance Software is a family of cooperative products that enable enterprise-wide management of computing resources. The foundation of the TeamQuest Performance Software family is TeamQuest Performance Framework (Framework). Framework is an integrated set of agents that collect, store, and distribute information to the actual TeamQuest Performance Software products. Framework is not sold separately, and is bundled with the products in the TeamQuest Performance Software family.

Framework automatically collects system level performance data such as CPU utilization, network packet rates, memory consumption, etc. Sometimes it is essential to have additional performance data from applications that are using these system resources. In many products this is not possible, or if it is possible it must be done in a very intrusive way that necessitates recompiling the application with new instrumentation points.

User Probe or Table Probe?

TeamQuest Performance Framework provides a very convenient mechanism for incorporating application level performance data into the TeamQuest performance database. TeamQuest calls this mechanism a User Data Probe. There are two types of Data Probes: Table Probes and User Probes. The difference between these two probe types can be summarized in the following way.

• User Probe: Used when application data is like system-level data.
• Table Probe: Used when application data is like process-level data.

In other words, you need to reflect first on the type of data you intend to collect and then write your probe accordingly. Here is a quick checklist to help you decide which probe type to use:

In most cases, you will want to write a User Probe to collect application performance data. Let's quickly review what a typical simple User Probe looks like in TeamQuest Performance Framework 7.1 and earlier.

Sample User Probe

The following commented script provides an example of a simple User Probe that wakes up every 30 minutes and sends the number of logins to a performance database.

From this example, we can identify the following actions in the User Probe:

1. Synchronize the probe's data collection to the system clock
2. Declare the statistics headers once for the performance database
3. Start the timing loop
4. Format relevant data from the who command
5. Pipe the formatted data to the performance database
6. Sleep for the remainder of the sampling interval

Although this User Probe is very simple in that it really only collects login data, there are at least 6 identifiable pieces that must be included. In a subsequent section of this note, we'll consider see how to modify this shell script to incorporate application performance data.

Next, we take a closer look at items 4 (data formatting) and 5 (data transport) in this example.

Data Abstractions

Here we present an example of how User Probes are set up to capture and incorporate application data into the TeamQuest performance database. It is important to understand two abstractions about the sampled data. These abstractions are:

1. The Data Format model
2. The Data Transport model

Understanding these data models clearly makes it much easier and faster to set up a successful User Probe. A general introduction can be found in the TeamQuest document entitled TeamQuest Performance Framework User Guide and we assume here that you are familiar with that documentation.

First, we consider the data formatting model that is assumed for a User Probe.

1. User Probe Data Format Model

Here is a typical window that you might see in TeamQuest View. It shows the number of TCP connections in a histogram or Cluster Bar format.

There are many TeamQuest View formats to choose from but they all presuppose that the performance data is stored as X-Y pairs. These X-Y pairs correspond to the "tops" of the Cluster Bars. The window above actually shows "family" of X-Y pairs corresponding to ftp, telnet, rlogins, etc. and depicted in different colors.

In the same way, all User Probe data must be delivered to the performance database as X-Y pairs. To clarify, think of the X-Y pairs as a set of points joined by an imaginary dotted line to indicate that the X-Y pairs belong to the same family of points. This example shows the number of logins measured every 30 minutes.

More accurately, the X-Y pairs are actually time-Y pairs (or t-Ypairs). The X-axis always represents time in TeamQuest View. The Y-axis represents whatever data has been measured. If we measured CPU utilization as %system and %user every 30 minutes we would then expect to see a TeamQuest View chart with two curves (Line mode).

We could also represent the CPU utilization data in a Cluster Bar format and then it would look like the previous TeamQuest View for TCP connections.

We can summarize this important principle about User Probe data formatting as:

This principle is very important to keep in mind when constructing a User Probe because it will permit you to examine the data in TeamQuest View using precisely the same tools you use to examine default system level data.

2. User Probe Data Transport Model

Recall item 5 in the example of a simple User Probe. The UNIX pipe is needed because the User Probe is not only the source of formatted data, but it is also responsible for getting that formatted data into the performance database. This is called a "push" model because, unlike system data, your probe has to push the data into the database. System data, on the other hand, is "pulled" from the source (the UNIX or NT kernel) and automatically inserted into the performance database.

We can summarize this point about User Probe data transport as:

Example Application Probe

As an example of how to implement an unintrusive application probe, we suppose you can write a benchmark to run your application in the production environment to measure some aspect of the application's performance e.g., end-to-end response times. This is a very common practice in performance testing and analysis. In addition, assume that the benchmark writes the measured performance data to a log file.

In this scenario, there are four major components:

• the application benchmark program
• the benchmark log file of performance data
• the TeamQuest application probe
• the TeamQuest performance database

The Application Probe can be set up to perform the following sequence:

1. Periodically run the benchmark program
2. Read the benchmark log file
3. Format the benchmark data as time-Y database pairs
4. Push the formatted performance data to a performance database

The following pseudocode called "applprobe.sh" shows how to incorporate these steps into a shell script.

The relationship between these components is captured in the following animation.

Summary Points

• Probes are an unintrusive way to collect and analyze application performance data.
• Choose your "weapon" carefully (either a User probe or a Table probe).
• For most applications you'll want to write a User Probe.
• Remember the underlying data format and data transport models.

Following these steps should make it easy to construct a successful application probe.