 |
 |
 |
 |
OpenSTA (Open System Testing Architecture) is a distributed software architecture for developing, executing and analyzing the results of Tests. A Test includes Scripts that are executed by Test Executers running on one or more Hosts and which may represent one or more Virtual Users.
In addition, a Test may include Profiles. These define sets of SNMP or NT Performance data to be collected during a Test-run. They can provide useful information about system activity and the Results can be analyzed alongside those from other OpenSTA Packages.
The OpenSTA Architecture provides generic facilities that may be used by other OpenSTA Packages. This chapter describes this architecture and its components.
OpenSTA Packages
The OpenSTA Architecture Package is the base OpenSTA Package and must be installed before all others. It is installed, as are all Packages, using Microsoft Installer. The graphical Commander utility is used to develop and run Tests. It is also used to display the results of Test-runs. Each Package, provides its own Plug-ins to provide Package-specific Configuration, Test-run Monitoring and Results display facilities. They are invoked by Commander.
An OpenSTA Test
A Test is represented in Commander as a table. This table may contain any number of rows, each defining one of two types of Task Group to be executed - a Script-based Task Group and a Profile-based Task Group.
A Script-based Task Group represents a sequence of Tasks to be performed by one or more Virtual Users. For example, within the HTTP Package, a Task Group represents a web browser session - to be executed by one or more Virtual Users. Each Task is represented by a Script - written in the English-like scripting language SCL (Script Control Language), developed by CYRANO. When a Test is executed the SCL compiler is invoked to compile these Scripts into portable, format object files for execution by Test Executers.
Each OpenSTA Package provides its own facilities for creating and maintaining Package-specific Scripts. For example, the HTTP Package provides the Script Modeler utility for producing Scripts from web browser session recordings. The file extension for SCL Script source files is Package-specific - for HTTP scripts the extension is .HTP. The object file extension is .TOF. A .SCD file is also created by the SCL compiler; this contains a list of script dependencies and is used to identify items required to compile and run a Script.
A Profile-based Task Group, defines a set of data to be collected from one or more Hosts at user-specified intervals during a Test-run. This data can be viewed alongside other Results to provide comprehensive information about a Test-run. Each such Task Group consists of a single Task - known as a Profile - defining the data to be collected. Profiles can be defined for collecting performance data from Hosts running Windows NT or Windows 2000 and for collecting SNMP data from Host computers or other devices running an SNMP agent or proxy SNMP agent. Data collector Test Executers collect data throughout a Test-run. Each Profile is stored as a portable format object file. NT Performance Profile object files have the extension .NTP. SNMP Profile object files have the extension .SMP.
The definition of a Test - as represented in the table displayed by Commander - is stored in a Test Definition file. This is held within the Test's subfolder. For example, if OpenSTA is installed in the default location, the Test Definition file for the Test MYTEST would be:
The Test Definition file is used to generate SCL scripts to control the execution of a Test. These scripts are automatically generated by Commander and are compiled by the SCL compiler. They have file extensions .TSS and .SCL.
One Test Script (.TSS file) is created for a Test and one for each of its Task Groups. A .SCL file is also created for each Task Group. The object files created for Test Scripts (.TSS files) have the extension .TSO. A .TSD file is also created by the SCL compiler; this contains a list of Test Script dependencies and is used to identify items required to compile and run a Test Script.
When a Test is initiated, all the Task Groups - identified by rows in a test table - are started together.
Test Managers and Test Executers
When a Test is executed, a Test Manager process and one or more Test Executer processes are created to execute the Test and its constituent Scripts and Profiles.
When a Test is initiated - by clicking the Start Test button - a single Test Manager process is created on the Repository Host to execute the TestManager.exe image. This is responsible for executing the Test Script (.TSO file) for the Test. This will create one or more Test Executer processes to execute the Scripts and Profiles in the Task Groups that make up the Test. The Test Executer process for a Task Group is created on the Host on which it is to be executed; i.e. the Host specified in the Task Group definition.
Task Groups containing Scripts are executed by Package-specific Test Executers. For example, a Task Group containing Scripts from the HTTP Package will be executed by the HTTP Test Executer (TExecuter_web.exe), which can be configured using the initialization file TestExecuter_web.ini. For more information see Appendix: Test Executer Parameter File. One Test Executer process is created for each Task Group to be executed on a Host.
Profile-based Task Groups are also executed by Package-sepcific Test Executers. One Test Executer process is created for each Task Group to be executed on a Host. The Host on which the Test Executer runs is specified in the Task Group settings. This is not the Host from which data will be collected during a Test-run. The target Hosts for data collection are specified in the Profile and defined in the queries it contains.
NT Performance Profile Task Groups are executed by the NT Performance Test Executer (TExecuter_ntp.exe). These Test Executers run for the duration of a Test-run.
SNMP Profile Task Groups are executed by the SNMP Test Executer (TExecuter_smp.exe). These Test Executers also run for the duration of a Test-run.
Script-based Test Executers close down when Task Group execution is complete. When all Script-based Test Execution is complete the Profile-based Test Executers and the Test Manager close down.
A Distributed Architecture
OpenSTA allows Task Groups to be executed on remote Hosts. In order to do this, OpenSTA must have been installed on each remote Host and the OpenSTA Name Server on each configured to specify the Repository Host for the Test.
The Test Manager process created for a Test, always runs on the Repository Host, from which the Test must be initiated. Test Executer processes are created by the Test Manager on the Hosts on which the Task Groups are to be executed.
OpenSTA's distributed architecture is based on the Common Object Request Broker Architecture (CORBA) developed by the Object Management Group (OMG), and uses the omniORB Object Request Broker (ORB) and Naming Service from AT&T Laboratories Cambridge.
The OpenSTA Repository
All the files that define a Test and the result files produced by a Test-run are stored in a flat-file structure on disk; this serves as the OpenSTA Repository. The default Repository folder, if OpenSTA is installed in the default location, is:
An empty OpenSTA Repository is created by Commander when it is invoked, if the local Host is the Repository Host and the Repository does not exist. An empty Repository contains the following files:
An alternative Repository folder can be specified within Commander using the Repository Path option on the Tools menu.
The Tests folder may contain subfolders containing further Test files. This allows Test files to be grouped for the convenience of users.
Below is an example listing of the contents of an OpenSTA Repository with sample Profiles, Scripts and Tests:
Note: IPADR is the IP address on which the Test Manager or Test Executer was executed (with dots replaced by underscores).
When a Test is executed, any files that need to be compiled are compiled into the \Objectcode folder of the Repository. If compilation is successful - or not required - all the files required to execute the Test - not the Task Groups within it - are copied to the \Engines folder of the Repository Host, e.g.:
When a Test Manager initiates the execution of a Task Group, all the files required by the Test Executer are copied to the \Engines folder of the Test Executer Host. These files are temporary and can be deleted when test execution is complete.
SNMP Profiles
The Simple Network Management Protocol (SNMP) Model and protocol allow state information to be retrieved from nodes in a computer network that are running SNMP agents or are served by proxy agents. A 'Network Management Station' sends an SNMP request to the SNMP agent, or proxy agent, which returns the requested data.
The OpenSTA SNMP Package allows SNMP data to be collected from Host computers and other devices running SNMP agents using OpenSTA SNMP Profiles. These are created and maintained by the OpenSTA SNMP Configuration Plug-in invoked by Commander. This uses the data in the Management Information Block (MIB) files, supplied with OpenSTA, to present the data for selection. A list of the IP addresses of Hosts within a user selected range of IP addresses may also be scanned to identify Hosts running SNMP agents or proxy agents. This list may be used to select an SNMP Profile Host and to view the object data from that Host.
An OpenSTA SNMP Profile defines the data to be collected from one or more Hosts. An OpenSTA SNMP Profile is stored in a portable format object file with the .SMP file extension. SNMP data is retrieved and recorded at user specified intervals throughout each Test-run by the OpenSTA SNMP Test Executer (TExecuter_smp.exe). A different interval may be specified for each SNMP object.
OpenSTA's SNMP Test Executer uses Net-SNMP from the University of California at Davis.
SNMP data collected by the OpenSTA SNMP Test Executer can be monitored as it is retrieved using the SNMP monitoring Plug-in from within Commander. The data is stored in local files - one data file per Test Executer. These files are closed and copied to the Test-run folder of the OpenSTA Repository on the Repository Host when Test execution is complete, e.g.:
The SNMPHeader.txt file contains descriptions for the data collected and is used in Results display. There is one header file for each data file.
NT Performance Profiles
Windows NT and Windows 2000 include a graphical tool - the Performance Monitor - for viewing performance data on these systems. The OpenSTA NT Performance Package allows the performance data displayed by this tool and collected by the NT Performance facility to be recorded within OpenSTA.
OpenSTA allows NT Performance data to be collected from Hosts running Windows NT or Windows 2000 using OpenSTA NT Performance Profiles, created and maintained by the OpenSTA NT Performance Configuration Plug-in invoked by Commander. An OpenSTA NT Performance Profile defines the data to be collected from one or more Hosts. An NT Performance Profile is stored in a single portable format object file with the .NTP file extension.
NT Performance data is collected at user specified intervals throughout each Test-run by the OpenSTA NT Performance Test Executer (TExecuter_ntp.exe).
OpenSTA's NT Performance Test Executer uses the Windows API to retrieve the required data.
Windows Performance data collected by the OpenSTA NT Performance Test Executer can be monitored as it is collected using the NT Performance Plug-in within Commander. The data is stored in local files, one data file per Test Executer. These files are closed and copied to the Test-run folder of the OpenSTA Repository on the Repository Host when Test execution is complete, e.g.:
The NTPHeader.txt file contains descriptions for the data collected and is used in Results display. There is one header file for each data file.
Architecture Package Installed Files
The following files are installed by the OpenSTA Architecture Package:
Script-Based Package Installed Files
The following files are installed by the OpenSTA HTTP Package. They are listed as an example of the files installed by a Script-based OpenSTA Package:
SNMP Package Installed Files
The following files are installed by the OpenSTA SNMP Package.
NT Performance Package Installed Files
The following files are installed by the OpenSTA NT Performance Package.
Error Reporting and Tracing
OpenSTA creates and maintains a number of Log and Trace files for recording Test-run information. These are described below.
The Audit, Report and History Logs
OpenSTA maintains an Audit Log of its activity and related events for each Test-run. This file contains informational, warning and error messages from the Test Manager, Test Executers and, optionally, messages from Scripts written using the SCL LOG command.
All messages in the Audit log are time-stamped and indicate the name of the Script being processed and the associated User ID, if applicable. All Time-stamps in the Audit Log and elsewhere are based on the time on the Repository Host. This makes it easier to analyze the results of Tests executed on Hosts with different system clock settings or in different time-zones.
The Audit Log can be viewed from the Results tab in Commander. The Audit Log is stored in the Test-run results folder in the OpenSTA Repository. For example, the Audit Log for the Test MYEST initiated on 23-Jan-2001 at 14:27:55 would be held in the following file (if OpenSTA was installed in the default location):
In addition to the Audit Log, OpenSTA may also create two further Test-run logs that may be written to from a Script - a Report Log and a History Log.
The purpose of the Report Log (TestRep.txt) is to record transient information relating to the execution of a Test. Test Executers may write messages to this Log, for example to record test-case failures and passes. Messages may also be written to the Log from a Script using the SCL REPORT command. The Report Log can be viewed from the Results tab in Commander.
The purpose of the History Log (TestHis.txt) is to record a history of the executions of a test. Messages are written to the Log from a Script using the SCL HISTORY command. No OpenSTA process - Test Manager or Test Executer - writes messages to this Log. The History Logs for a Test can be viewed from the Results tab in Commander. A separate History Log is maintained for each Test-run. However, all the History Logs for a Test are concatenated to form a single Log when viewed within Commander.
Report and History Logs are stored in the Test-run results folders. Messages within them are time-stamped and indicate the name of the Script being processed and the associated User ID, if applicable.
Test Manager and Test Executer Trace Logs
For each Test-run, a Trace Log is created for the Test Manager and each Test Executer. These Logs contain informational, warning and error messages logged by the Test Manager and Test Executers respectively - error messages will also be written to the Audit Log. They are created in the same folder as the corresponding executable images - i.e. \Engines - and are copied to the Test-run results folder on test completion. The Log file names have the following format:
Note: IPADR is the IP address of the Host on which the Test Manager or Test Executer was executed (with dots replaced by underscores), PID is the Process ID of the Test Manager or Test Executer and EID is the Executer Identifier (e.g. SMP for the SNMP Test Executer).
Each message that may be written to the Test Manager and Test Executer Trace Logs has an associated category and level. The level is a number between 1 and 1000 and indicates the importance, or severity, of the message - 1 is the most important and 1000 the least important. These levels are used to control the level of messages that are recorded for each category for a Test-run.
The highest level of messages to be recorded for each category during a Test-run may be specified on the Trace Settings dialog within Commander. These settings are saved in the file TraceSettings - held in the Repository folder. These settings will apply to all subsequent Test-runs, or until the Trace Settings are modified. Initially, tracing is switched off - i.e. the trace levels for all message categories is set to zero - in order to make execution as fast as possible. Tracing is typically enabled during Script, Test and Profile development, in order to help resolve problems when Tests are not running as expected.
Other Trace Logs
In addition to the Logs described above, OpenSTA also maintains the following Logs, which may be of use in diagnosing problems:
OpenSTA Daemon Log, records the activity of the OpenSTA Daemon. PID is the process ID of the OpenSTA Daemon process.
A higher level of tracing my be set for the OpenSTA Daemon by checking the "Turn on Tracing" check box on the Name Server Configuration utility's "Configuration" dialog.
Tracing Script Activity
Script activity may also be traced at run-time using the SCL NOTE command within Scripts. This command allows a message to be associated with a thread. The messages associated with a thread - if any - may be viewed within Commander, through the Monitoring tab. By including NOTE commands within Scripts it is possible to trace the flow of execution of threads at run-time.
Starting OpenSTA
When OpenSTA is installed, the Name Server Configuration utility (DaemonCFG.exe) is configured to startup automatically at boot time and to start the Naming Service (omninames.exe) - if the local Host is the Repository Host - and then the OpenSTA Daemon (cyrdmn.exe) on the local Host.
Before a Test can be executed within OpenSTA, the Naming Service - omninames - must be running on the Repository Host. The Naming Service is used to hold the names and types of OpenSTA CORBA objects. The Naming Service provides the means by which a program can locate the object reference for an object and thereby reference it.
By default, after a Host on which OpenSTA has been installed is booted, the following images will be running:
These images will continue to run until, either they are explicitly shutdown by the user - using the OpenSTA Name Server - or they terminate abnormally. If "Automatic Notification" is enable, the Name Server Configuration utility displays a warning dialog box, if either the Naming Service or OpenSTA Daemon terminates abnormally.
Omninames and the OpenSTA Daemon can be started from the command lines follows:
They must be started in the above order.
The Name Server Configuration Utility
The Name Server Configuration utility (DaemonCFG.exe), accessible from the Windows Programs menu, provides a "Configure..." option. This displays a configuration dialog containing four fields:
This identifies the name, or IP address, of the Host holding the OpenSTA Repository to be used by the local Host. Tests must be initiated from the Repository Host and the Naming Service must run on the Repository Host.
This is a readonly field identifying the Repository path on the local Host. This is configured from Commander. It is not relevant and not used, unless the Repository Host is the local Host.
If this check-box field is checked, additional tracing data will be logged to the OpenSTA Daemon log file cyrdmn.log. The file will always be created and have data written to it even if this field is not checked.
If this check-box field is checked, a timer is initiated to 'fire' every five seconds. When the timer fires, the system is checked to see that the Naming Service and OpenSTA Daemon process are still running - if either is not, a warning dialog is displayed.
The Name Server Configuration utility also provides the following menu options:
Starts the Naming Service (omninames.exe) and the OpenSTA Daemon (cyrdmn.exe) on the local Host. The Naming Service, will only be started if the Repository Host is configured to be the local Host.
Shuts down the Naming Service (omninames.exe), the OpenSTA Daemon (cyrdmn.exe) and the Name Server Configuration utility (DaemonCFG.exe) on the local Host. The Naming Service, will only be shutdown if the Repository Host is configured to be the local Host.
Lists the OpenSTA CORBA objects registered with the Naming Service. The omniORB nameclt utility (held in the Server folder) may also be used to view the list of registered objects - command "nameclt list".
· The Repository is located on the Repository Host in the folder identified by the "Repository path", which can be configured through Commander.
· An OpenSTA Daemon (cyrdmn.exe) must be running on each Host on which a Task Group is to be executed.
The OpenSTA Daemon
The OpenSTA Daemon process starts a CORBA Factory object and loads a set of 'provider' modules, used to provide 'services' to OpenSTA components. These modules are held in the OpenSTA Server folder and have file names of the form "CyrVDKnnn.dll", where "nnn" is numeric identifier.
Below is a list of the OpenSTA provider modules, together with the names that they register with the Naming Service:
CyrVDK001.dll - Thread provider
IPADR_CyrThreadPrivider_001.CyranoProviderCyrVDK002.dll - Repository interface
IPADR_CyrStProvider_001.CyranoProvider
IPADR_RegistryProvider_001.CyranoProvider
IPADR_CyranoVDK002.CyranoLogCyrVDK003.dll - Naming Service cleaner
CyrVDK004.dll - Time service
TimeService.TimeServiceWhere IPADR is the Host's IP address (with dots replaced by underscores).
After the OpenSTA Daemon has been started, there will be seven CORBA objects registered with the Naming Service:
IPADR_CyranoDaemon.CyranoLog // Daemon logger
IPADR.CyranoFactory // CORBA object factory
IPADR_CyrThreadProvider_001.CyranoProvider // Thread provider
IPADR_CyrStProvider_001.CyranoProvider // Repository provider
IPADR_RegistryProvider_001.CyranoProvider // Registry provider
IPADR_CyranoVDK002.CyranoLog // Repository logger
TimeService.TimeService // Timer service
These objects should always be registered with the Naming Service - they should only disappear if the OpenSTA Daemon shuts down or terminates abnormally.
Command Line Formats
There is a command line interface to most OpenSTA executable images. The formats of those that may be useful to OpenSTA users are listed below.
Test Initiator (TestInit.exe)
The Test Initiator utility may be used to start, monitor, stop and kill tests.
TestInit -start -n Node -P Repository -T Testname
The -start switch initiates execution of a specified test (Testname) from the specified Project "Repository" and on the specified node (Node). For example:
TestInit -start -n myHost -P Repository -T mytest
Note: the TestInit utility is invoked by Commander to initiate a Test.
The -monitor switch displays a 'snapshot' of the current state of the OpenSTA CORBA objects to standard output. The output is not refreshed.
The -stop and -kill switches cause TestInit to prompt for the Test Manager to stop or kill respectively. The former attempts to stop a Test-run gracefully and will wait for the Test Manager and Test Executers to close down before returning. The latter terminates the Test Manager and Test Executers immediately - it does not wait for confirmation that the processes have terminated.
OpenSTA Daemon (CyrDmn.exe)
The OpenSTA Daemon process must be running before a Test can be executed. It is normally started at machine startup by execution of the Name Server Configuration utility (DaemonCFG.exe). It may also be started by selecting Start > Programs > OpenSTA > OpenSTA Name Server or by selecting Start Name Server within the Name Server Configuration utility.
cyrdmn --help Display utility help
--trace n Trace logging level (0 to 1000)
The --trace switch is not required.
Script Compiler (scl.exe)
The script compiler (SCL) is used to compile Scripts and Tests. The former are created using a Package-specific Plug-in. The latter are generated automatically by Commander. Scripts are held in the Scripts folder and have an application specific extension (.HTP for HTTP Scripts). Tests and Test Scripts are held in the Tests folder and have a file extension .TSS and .SCL respectively.
SCL generates portable format object files which can be executed by a Test Manager (for Tests) and a Test Executer (for Scripts). A dependency file is also created for each Script and Test that is successfully compiled, this is used to identify items required to compile and run the Script or Test. SCL has the following command line format:
-I inc_path --include=inc_path
-V v1+v2{+...} --variant=v1+v2{+...}
Compile the Script myscript.htp and output compilation messages to standard output - for a successful compilation these will identify the names of the files created. SCL will look for any include files in the local folder and, if not found there, in the Include subfolder. The following files will be created:
myscript.tof - Portable format object file (for execution)
myscript.scd - Script dependency file
Compile the Test myscript.tss. The following files will be created:
| CYRANO Customer Support: France +33 (0) 1 56 33 40 00 USA +1 (978) 462-0737 UK +44 (0) 1274 761024 support-fr@cyrano.com support-us@cyrano.com support-uk@cyrano.com |
| |
|
|
|