RAPIDS Manual

RAPIDS Configuration Files

RAPIDS Client Interface

• Monitoring
  • Start monitoring

    Connect to the RAPIDS server and start receiving the stream of events. At this point a number of RAPIDS displays will be opened.

  • View current settings

    Sometimes it is important to view the monitoring setup that is being currently used. If you choose this item, the monitoring setup tabbed panel that shows the current setup will be opened. Note that while viewing the setup you are not allowed to change it. Go to the "Change Current Settings" menu item in order to modify the current setup.

    The tabbed panel that shows the currently used monitoring setup includes the following sections:

    • Nodes. Information on the SOCC and Sensor nodes: nicknames, machines etc.

    • System. Commands used to start/stop processes on the nodes. A set of commands might be provided for every node.

      In fact, you can consider such a set of commands as a script which will be executed on a node.

      In order to view commands used to start processes on the node, say, SOCC2 you should choose this node from a drop-down list at the top of the pane titled Processes running on a node and then press Show start commands button. A window that contains a list of commands will be opened (the nickname of the node, that is SOCC2, will appear in the title of the window). Similarly, if you would like to see commands used to stop processes on the node SOCC2 you should choose SOCC2 and press Show stop commands button.

      Note: 1. Meanwhile, there exists only one method to execute commands on different nodes in a certain order. For example, Rotation algorithm should be started only after feature repository and optimization algorithms have been already started. This can be achieved by inserting sleeps where it is necessary: in the example above - before starting feature repository and optimization algorithms.

      2. A script called killer.pl is used for stopping processes. This script receives the name of the process which should be killed, finds PIDs of all processes that have this name and kills them.

    • Network. Settings for the network simulation: packet delays and/or drop rates can be simulated on network connections between Sensor nodes and the SOCC.

    • Monitoring. Metrics and processes that will be monitored. In order to view system metrics (such as CPU or memory utilization) that are collected on the node SOCC2, for example, you should choose this node from a drop-down list at the top of the Node settings pane and then press System... button. Similarly, if you would like to view process metrics collected on SOCC2 and processes monitored on this node you should choose SOCC2 and press Processes... button.

      When you press Processes... button, a window that contains a list of all monitored processes on a certain node will be opened. All these processes will appear in the Message diagram. If you choose a process and press Show details... button you will see various settings for this process (such the color code of its events, metrics collected for the process etc.)

      On the Node settings pane you will also find a Refresh rate spinner. It shows how frequently the monitoring information is being updated.

      For more details on events and metrics available for monitoring see the "Events and Metrics" section.

    Note: It is possible to specify in the experiment configuration file processes that you would like to monitor and that will not appear on the Message diagram. These processes and the metrics collected for them will appear on displays showing process metrics. However, the current version of GUI shows only processes that appear on the Message diagram.

    • General. Creation and modification dates for the currently viewed configuration, nickname of the configuration etc.

  • Change Current Settings. This feature has not been implemented in the current version of RAPIDS.
• Log.
  • Replay... Allows to view log files using RAPIDS displays. More information on replay feature can be found here.
  • Convert to ASCII... RAPIDS log files are stored in a binary format. In some cases, however, a conversion into the plain ASCII format is required (for example, if statistical processing of collected events and metrics should be performed). This feature allows to convert a chosen log file into ASCII format. The textual file will have the same name as the source log file and the "txt" extension. It will be placed into the same directory where all log files are located.
  • Resource utilization...
    • CPU utilization. Mean and standard deviation will be calculated. Histogram seeds will be printed that show how much time (as a percentage of the total recorded time) a specific percentage of the CPU has been utilized. Also, high activity periods will be printed: periods of time when the CPU has been utilized more then 40%. And finally, data files that can be used by gnuplot to create CPU utilization graphs will be generated.
    • Memory utilization. Data files that can be used by gnuplot to create RAM and swap space utilization graphs will be generated.
    • LDM transfer times. Data files that can be used by gnuplot to create LDM transfer time histograms will be generated. The histograms will show how many LDM messages were sent for a specific amount of time. All delayed messages will be printed separately. A message is considered delayed if it took more than one minute to transfer it.
• Window. A checkbox list of all currently available RAPIDS displays.

  Sometimes a user might want to close RAPIDS displays; for example, if the user is not currently interested in the information being shown on a display, or just would like to avoid cluttering the screen. When the user closes a display, the correspondent item in the list of all RAPIDS displays will be unchecked. The closed display can be restored by checking it back in the list.

• Exit. Exit RAPIDS client (alternatively press Ctrl-X). Note that exiting the client does not mean that the experiment will be stopped (if there is one that is currently running). If you would like to stop the experiment, you should stop the RAPIDS server.
• Help. Show this manual. This feature has not been implemented in the current version of RAPIDS.

RAPIDS Displays

• Message diagram. Shows Messaging and Application events as well as messages formed by these events in real time.

  The X-axis of the Message diagram shows the absolute time, while the Y-axis shows processes running on different system nodes. Messaging and Application events generated by a process P appear on the line that corresponds to P on the diagram. As time passes the diagram scrolls to show the current time frame.

  

  Figure 2: RAPIDS message diagram showing 30 seconds system's cycle.

  • 19:21:48 Task generation module (featureRepo) sends a set of tasks to the optimization module (Optimization). These are tasks that should be completed during the upcoming cycle.
  • 19:21:48 - 19:21:51 Optimization module generates control signals and sends them to the radars.
  • 19:22:05, 19:22:07, 19:22:08 Radars (streamer_KRSP, streamer_KSAO etc.) send data to the SOCC. Note that streamer_KRSP, streamer_KLWE and streamer_KCYR have incomplete commands from the previous cycle at the time they receive new ones. For example, streamer_KLWE sends data at 19:21:53 finishing a scan according to the command received at the previous cycle.
  • Radar data is received at the SOCC and inserted into a linear buffer (lb_cat). Detection algorithms get the data out of the linear buffer (for example, lowRefCYR at 19:22:08) and process it. Detected meteorological features are sent to the task generation module (for example, at 19:22:09 lowRefCYR sends detected low reflectivity areas).
  • 19:22:17 Task generation module sends new set of tasks generated basing on the meteorological features detected during the cycle.

  Note that in the example above the detection algorithm generates application events (rather than messaging events) every time it sends detected features to the task generation module. Different types of application events are depicted on the diagram using different shapes: in our example detection algorithms throw simple debug events which are depicted as a vertical bars.

  Initially a 10 minutes long time interval is shown on the diagram but you can change it to 3 or 1 minute long one. In order to do it choose Time axis menu item on the diagram and then choose the desired resolution. Note that time axis ticks for 10 and 3 minutes long intervals correspond to minutes while for 1 minute long interval they correspond to seconds.

• Event Details. Shows various attributes of the events on the Message Diagram, such as a specific message associated with the event, PID of the process that generated the event, the exact time when the event was generated etc. In order to obtain additional information about an event you should click the right mouse button while positioning the mouse on a dot that represents this event on the diagram.

  You may see that details of more than one event appear when you click on the diagram. It means that more than one event has been generated almost at the same time, and RAPIDS was unable to figure out which of them you have chosen. You may want to change the resolution of the diagram in order to obtain more precise results.

  

  Figure 3: Examples of attributes associated with various events:

  • An Application event generated by the detection algorithm lowrefCYR on 10/26/2006 at 15:22:54. The event has the following message attached to it: "Features sent".
  • Two LDM Messaging events generated by the streamer streamer_KCYR on 10/26/2006 at 15:23:07. Both events have the pqinsert type. The messages associated with the events clarify that Reflectivity and Velocity products have been sent.

• RAPIDS Info.

  Shows various diagnostic messages as well as statuses of network connections (see Figure 4).

  

  Figure 4: This window is opened along with the main RAPIDS display; the first two messages notify the user that the RAPIDS client has successfully connected to the proxy server. The third message is printed when the active user starts an experiment. The rest of the messages show the statuses of network connections between system nodes.

• System Metrics.

  Shows current values of various system metrics.

  

  Figure 5: CPU and memory utilization on system nodes.

• Windows that show color codes of monitored processes. These windows also show process metrics collected for the processes. The number of these windows is equal to the number of machines involved in the experiment.

Events and Metrics

• Messaging. Events of this type occur when data is being sent/received through one of the system's communication channels. Table 1 shows which communication channels are used for transferring different kinds of data.

  Data Type	Communication Channel
  Radar scans (NetCDF files)	LDM product queue
  Meteorological features	WDSS-II linear buffers
  Radar tasks	TCP connection
  Radar control signals	TCP connection

  Table 1.Communication channels used for transferring different types of DCAS data.

  When two Messaging events relate to the transferring of the same item between processes P1 and P2, they form a message from P1 to P2.

  Messaging events are depicted on the message diagram as rhombuses. Messages are depicted as arrows going from the sending event to the receiving event. The color of an arrow shows which communication channel was used for the transferring: red color corresponds to LDM product queues, green corresponds to WDSS-II linear buffers, while blue corresponds to TCP connections.

• Application. Events of this type are generated by software modules running in the system. In contrast to Messaging events which RAPIDS collects transparently to the software modules, Application events require changes in the software's code in order to be seen by RAPIDS. RAPIDS API that can be used to generate Application events is described here. All Application events are shown on the message diagram. Here is the list of various Application events that can be generated:
  • Error. Different levels of importance (warning, error etc.) can be used to describe various faulty situations that might occur in the software. Events of this type are depicted on the message diagram as crosses.
  • Value of a variable. These events are generated when a current value of a specific variable in an application should be reported. Events of this type are depicted on the message diagram as five-pointed stars.
  • Algorithm stage. Events of this type should be generated to report various stages in an application's execution, such as entrance to a subroutine. Currently only two subtypes of this event are available:
    • Begin. Events of this type are depicted on the message diagram as small arrows pointing up.
    • End. Events of this type are depicted on the message diagram as small arrows pointing down.
  • Debug message. Events of this type can be generated to report any information that does not fall into the above categories. For example, some meteorological applications generate events of this type to report that detected features have been sent to the task generation module (see Figure 2). Events of this type are depicted on the message diagram as vertical bars.
• Metric. RAPIDS allows real-time monitoring of utilization of various system resources, such as CPU, memory etc. In an experiment setup a user specifies which system resources should be monitored; RAPIDS periodically calculates utilization of these resources (according to the refresh rate specified by the user) and displays it on the system metrics display (see Figure 5). Every time RAPIDS calculates utilization of a specific system resource an event of Metric type occurs.

  Metric events are classified into System and Process events.

  • System. CPU, memory, workload over the last 1, 5 and 15 minutes: shown on the system metrics display (see Figure 5). Status of the network connections between the nodes: shown on the RAPIDS info display (see Figure 4).

    Note that the in order to monitor the network status RAPIDS starts small software probes on each system node in the beginning of an experiment. These probes periodically send "I'm alive" signals to one of the nodes, say S, who forwards them to RAPIDS. If such a signal stops coming from one of the nodes, say C, RAPIDS reports that connection between S and C has been broken. Obviously, this monitoring method creates a possibility of a false alarm: if the software probe on C crashes and stops sending signals to S, RAPIDS will be reporting a broken connection between S and C still the same.

  • Process.

How to replay an experiment

1. Start the RAPIDS Client.

   >run_gui.pl

2. Select the log file of the experiment you would like to replay. When the main RAPIDS window is opened choose the Log item from the menu and then the Replay... item. Choose the log file of the experiment you would like to replay. Press the Replay button.

   Note: The name of a log file starts with the prefix log followed by the date and time when this log file was created. For example, the following log file:

   log_02242005_172230

   was created on February 24^th, 2005 at 17:22:30.

3. Replay the experiment. In addition to standard RAPIDS windows described in Section “RAPIDS Displays” another window titled Replay functions will be opened. Replay functions include the following:

   • Play. Replay the experiment at the real time speed.
   • Pause. When the experiment is paused an additional time resolution (milliseconds) becomes available at the message diagram. See Magnify function for more details.
   • Stop.
   • Fast forward. Replay the experiment at the speed 5 time higher than the real time one.
   • Magnify. This function becomes available only when the experiment is paused. The function allows to view the message diagram at millisecond resolution. While being in this time mode you can scroll the diagram forward or backward using corresponding buttons on the tool bar at the bottom of the diagram (see Figure 6).

     Figure 6: Message diagram that shows a one second long time frame. The diagram can be scrolled forward or backward using corresponding buttons on the tool bar at the bottom of the diagram.

     An Event Details display will be opened along with the diagram. Figure 7 shows an example of such a display with details of events that appear on the message diagram from Figure 6.

     Figure 7: Event Details display that shows detailed descriptions of events that appear on the Message Diagram from Figure 6. For example, Messaging event generated by streamer_KCYR: radar control signals received on 10/26/2006 at 15:23:19. In the line providing details of this event there is a number that appear in brackets after the date/time when the event has occurred. This is merely the same date/time of the event occurrence shown in another format: as number of milliseconds since the standard base time known as "the epoch", namely January 1, 1970, 00:00:00 GMT. This number shows that this event occurred on the 993th millisecond of the 19th second which corresponds to the very first blue arrow on the diagram from Figure 6. Also, there are several Application events that have been generated by streamer_KCYR at the same time. They provide details for the received control signals, such as which sector and how many elevations should be scanned, how much time it will take to perform the scan etc.

4. Stop replaying the experiment. Press Stop button in the “Replay functions” window.

Back to top

Troubleshooting

1. Log in to all machines used in the experiment and stop all DCAS applications started during the experiment (streamers, detection algorithms, task generation and optimization module etc.)
2. Log in to all machines used in the experiment as rapids and stop LMMs:

   >/usr/share/rapids/utils/bin/lmm_remover

3. While logged in as rapids remove RAPIDS message queue - a data structure used for collecting events:

   >/usr/share/rapids/utils/bin/rmq_remover

Back to top

The RAPIDS API

Provided with the RAPIDS framework is a set of API functions that users can use in their applications to report different kinds of messages/events to the RAPIDS monitoring system. Below are links to the RAPIDS C, C++ and Java API.

• C API

• C++ API

• Java API, Example showing the use of the API. Also, here is the source code for a dummy server and client applications (along with the corresponding makefile) showing how to use the API to generate Socket Messaging events. Please make sure that LD_LIBRARY_PATH variable is set appropriately as shown in the How to Install RAPIDS section prior to running the application.

Back to top

Connecting to the Real-Time Stream of RAPIDS Events

Back to top

Reporting Bugs

Please, send an e-mail to Anna (abekkerm@ecs.umass.edu) in case RAPIDS has crashed, you have found a bug or there is anything else you would like to tell :)

Back to top