Sipconf performance measurement

Problem statement

Measure QoS parameters, delay, jitter and loss, for our conference server. Write tools or setup test environment for this so that it can be used with other conference servers also for comparision.

1. Which software are we extending? sipconf
2. Features to be supported and priorities? TBD
3. Existing libraries or modules? sipconf, rtptools, sipstone, sipua.
4. Programming language? C/C++, (and Tcl optional)
5. Operating system? Solaris, Win2000 and Linux.
6. GUI tool kit? N/A.
7. Component testing? This itelf is a test tool.
8. Error logging? TBD.
9. Installation mechanism? TBD
10. Milestones? TBD

Reading: Read the paper on Centralized conferencing using SIP to get a general idea of how sipconf works.

Running sipconf: Use the instructions to check out the cinema code from CVS. See the compilation instructions in README.build and compile sipconf on your favorite platform. See the sipconf documentation. Test sipconf using sipc. Sipc is installed at /proj/irt-gc2/irt/sipc.unix and /proj/irt-gc2/irt/sipc.linux. You will need a new license string if you want to test it on a non-columbia machine.

RTP is used over UDP to send audio and video data. RTCP (Real-time Transport Control Protocol) is used along with RTP, to provide some QoS information. For a point-to-point case the RTCP information can provide delay, jitter and loss information in each direction to both the participants. For a centralized conferencing case (no multicast) the RTP channels are effectively between the conference server and each of the participants. So the conference server and each of the participant can know the loss, delay and jitter information on the link between the participant and the server. However, because each RTP channel to a participant is considered independent of the other participant, the information does not take into account the delay and loss introduced at the conference server.

Reading: Read the RTP RFC 1889, and find out how the QoS information are exchanged using RTCP packets.

Rtptools: Download and install the RTP Tools, and write a simple shell script to general RTP packets using rtpsend, and receive them using rtpdump. Can you write a script that will look at the rtpdump output, analyse the RTCP reports and print out the QoS information in each direction.

As the number of participants increase in a conference, the number of audio streams to be handled by the server also increases. Due to inherent limitation in terms of bandwidth and CPU resourses, the over-all performance of the conference reduces. This can be seen by the participants as delay, jitter or loss in audio streams. The goal of this project is to measure quantitatively the performance of sipconf with increasing load.

The parameters that define quality in this case are listed below.

Delay

This is the total delay in the audio stream. That is the time it takes for a audio data from a participant to go to the server, get mixed, sent back to the participants and finally received by the participant. This can include the application level delay in the participants' user agent, the network delay, the delay in the conference server. What we are interested in is measuring the delay in the conference server. On a typical LAN network delay will be very less. Application level delay in the participants' user agent is minimized by using no play out delay. The conference server delay consists of the play out delay (the time a audio data packet has to wait for other packets from other streams before it gets mixed) and the other delay (socket, thread switching, packetization, mixing, etc.). Note that the mixing delay may be significant if the codec implementation has inherent delays (algorithmic plus implementation). However we will be using G.711 codec that has minimum processing overhead and no algorithmic delay. The playout delay is adaptive so as to minimize the value while keeping the packet loss (late packets) as minimum. The two delay component we are interested in are the over-all delay (this is roughly equal to the conference server delay) and the play-out delay at the conference server. Packetization delay is usually 20ms or 40ms.

Loss

Packet loss can happen due to the network or due to a packet arriving too late for getting mixed with other streams. We usually do not have much control on the packet loss happening due to the network. However, if a packet is dropped at the server because it arrived too late, means that our play out delay was too small for that packet. Usually a packet loss of up 5% is tolerable for audio.

Jitter

Jitter is defined as the variation in delay. Consider source A sending audio stream to source B. If the delay is fixed, say 100 ms, for every packet, then A keeps on sending packets every 20ms (for example) and B keeps on receiving packets every 20 ms. Although the delay is there, but since it is fixed, the inter-packet arrival time is constant at the receiver (same as the sender). This means we do not need to buffer, the packets before playing if the delay is fixed. No consider what happens if the delay varies. For example, A sends packets at 20ms interval but due to the network, B received them at timestamps 0, 20, 45, 70, 80, 107,... Note that the variation in delay are 0, 0, 5, 10, 0, 7,... If B is playing out packets every 20ms, and the first packet was played out at timestamp 0, then when the third packet is supposed to be played (at timestamp 40), the packet hasn't arrived. So it is skipped. Similarly the fourth packet also does not arrive at timestamp 60, so skipped too. This can be avoided by keeping a playout delay of say 10ms. So instead of playing the packets at timestamps 0, 20, 40, 60, 80, 100. they are played at timestamps 10, 30, 50, 70, 90, 110. Note that with higher playout delay, you can absord higher jitter. But that introduces more delay overall to the end user. A delay of more than 200ms is percievable and annoying. If the packet can not be played due to late arrival it is effectively lost. So there is a tradeoff here between delay and loss.

In our environment, the jitter for audio stream from the participant to the server is absorbed at the server (using its own playout delay). But the jitter from the server to the participant will affect the play out delay buffer size at the participant's user agent. This will effectively add to the over-all delay.

The performance metrics can be roughly divided into two categories: quality and efficiency. Delay, loss and jitter are the quality metrics. Efficiency can be measured in terms of CPU load, bandwidth utilization, thread switching overhead, number of open file descriptors, etc. While most of the efficiency metrics depend on the platform and implementation, it is nice to see the effect when the load on the server increases. Apart from comparision with other implementations, these metric is useful in load-balancing decisions also, e.g., when to start a secondary server. Moreover, its always a selling point if we show the performance numbers to the customers of our software. It should be noted that the efficiency metrics affect the quality metrics. Generally the system performs well if the efficiency metrics are kept within a threshold. Beyond the threshold, the quality starts degrading. For example, the packet loss at the mixer may be negligible if the CPU utilization is within 70%, but after this the packet loss may increase due to excessive time spent in other things and the thread that is suppose to handle the packet does not get scheduled and the packet is lost.

Metrics: List all the metrics you can think-of for measuring the performance of sipconf.

We will start by running the test on one platform and then extend it to other platforms. A general test setup will have a single conference server, sipconf, and a number of test clients (sip user agents). Each of these user agents connect to sipconf, and send and receive RTP audio data. Our initial goal is to measure performance only for the audio media type.

We have some test tools that can be used for testing the performance. In particular the SIP test user agent, sipua can be used as a test client for the participants. Sipua can make the SIP call to the conference server and can invoke external audio tool to send and receive audio packets.

Running sipua: Compile sipua from CVS. See the documentation and understand the basic commands in sipua. Run sipconf. Connect using sipua. Read the example setup to similate multiple calls. That setup allows invoking multiple sipuas on different hosts and connect to the conference server. various parameters, e.g., call duration, number of hosts, number of participants per host, etc can be changed in the various script files shown in the example. If it asks for the license file, you can get it at sipua.license.

We already have a file called, rtpsend.tcl. This can be used as an external audio tool in the test above. Basically it uses the rtpsend and rtpdump tools to send and receive audio packets. A modified version of rtpsend.tcl is available at rtpsend_jitter.tcl that also analyses the jitter in the received RTP stream. It uses the script analyze.tcl to process the received RTP stream and calculate the jitter. A more direct approach would be to use RTCP information to calculate the jitter.

Using rtpsend: Read rtpsend.tcl, rtpsend_jitter.tcl and analyze.tcl. Use sipua with -a option specifying rtpsend_jitter.tcl. Connect to the conference server. And find out the jitter variation in the received audio stream.

Up to this point you have already experimented with the existing test setup. Now we will build our test environment using these components. The following ascii diagram illustrates the test setup.

     +--------+       SIP INVITE               +-----------+        
   A | sipua  |------------------------->------|           |        
     + - - - -+                                |  sipconf  |        
     |        |                                |           |        
     | rtptool|                                |           |        
     |(rtpsend|                                |           |        
     |rtpdump)|       RTP/RTCP                 |           |        
     |        |<------------------------------>|           |        
     +--------+                                |           |        
                                               |           |        
        . . .                                  |           |        
                                               +-----------+        
     +--------+       SIP+RTP                   /  /                  
   B | sipua  |----------->--------------------/  /                    
     +--------+                                  /                   
        . . .                                   /                    
                                               /                     
     +--------+       SIP+RTP                 /                      
   C | sipua  |----------->------------------/                       
     +--------+                                                     

The first step is to write the rtpaudio tool that can be used for measuring all the QoS metrics. Some of the metrics, e.g., CPU utilization, number of threads, etc. are available only on the server side so we may need to fetch that information from the server. This can be done either by periodically printing out these metric values in sipconf, or by defining a special RTCP packet (QUERY,RESPONSE) that will allow the rtptools to QUERY the metrics and the sipconf server to return them in the RESPONSE. The first approach is straightforward but requires integrating the numbers from both client and server. The second approach will make it easier to automate and all performance metric collection will be done by the client. However, it require changes in the RTP library at the server side to handle such special RTCP reports. Using the QUERY/RESPONSE in SIP can also be done however, will not be useful for other components that do not use SIP, like RTSP media server. (It would probably make more sense to put this in a log (SQL/file) and display on web interface. Investigate this option? TBD)

Measuring the audio end-to-end delay is tricky as we need to compare the audio samples sent and received. Because of the lossy nature of audio compression, the sample values will not match exactly. Secondly, due to mixing of the audio from all the participants, a sender will not get only his own audio back but a mixed stream from all the participants. One workaround is to use linear samples instead of any codec so that there is no compression/decompression involved in the media path. And use sipconf with -b option to send audio back to the participant. And also keep only one active sender, so that there is no mixing involved at the server. In any event, usually only one participant is active in any conference since otherwise it would be difficult to follow what is going on if many participants speak at the same time. Using this setup, the rtptool can send linear samples with all sample values as 1 in first packet, 2 in second packet and so on. When it get backs the packets from the server, it can appropriately match the received packets with the sent packets.

There are other possibilities also: for instance, (1) use some signal processing to compare the audio samples; (2) use an IP options in the sent RTP packet. When sipconf receives this, it puts the same options in the mixed RTP packet back to the participant; (3) modify sipconf to send back the audio packet sent by the sender instead of the mixed stream (loopback mode). This will be after the mixing is done, but the mixed data is ignored for this participant. This will anyway be useful for testing purpose.

Rtptools: Using the rtpsend.tcl or in your favorite language, write a program that invokes rtpsend to send out RTP/RTCP packets and receives the RTP/RTCP packets. You should be able to configure the following parameters:

1. Codec: G.711 Mu Law, Linear 16.
2. Sampling rate: we will use 8kHz initially. Possible values should be 8kHz and 16kHz.
3. Packetization interval: we will use default of 20ms. It should be possible to configure it for 40ms or 30ms also.
4. Jitter: a random jitter value is choosen for sending out the packets. The jitter will affect the playout delay at the server and hence the total playout delay.
5. Talkspurt distribution: a random talk-spurt and silence distribution is choosen for sending out the packets. Since playout delay is adapted only during silence period, varying jitter won't have any affect if silence suppression is not on.

The tool should be able to print the RTCP QoS information: delay, loss and jitter in each direction. In addition it should be able to print out the end-to-end delay using the technique described earlier.

Extend the tool to send out the RTCP QUERY and receive RTCP RESPONSE and display the metrics.

Sipconf modifications

1. Modify the sipconf server to allow sending back only the participants audio instead of the mixed stream (as in loop back mode). This will allow testing even if more than one participants are active at any time.
2. Modify sipconf to display/log the performance metrics using our Logging mechanism. These metrics include CPU utilization, memory utilization, number of threads, number of open file descriptors, playout delay for the participant, any late packet for the participant and the conference, number of participants and number of conferences, delay due to mixing (decoding, adding, encoding).
3. Modify sipconf to return the RTCP RESPONSE messages for the participants that send RTCP QUERY with all these metrics.
4. Modify the cinema conference web interface to display the current log information (metrics) for the conference and the server. Or use the code from sipstone to plot graphs and analyze the metrics.

Plot graphs Plot various performance graphs, e.g., CPU utilization vs number of participants, Memory utilization vs number of three party conferences, etc. Do this for sipconf running on Sun/Sparc, Win2000, Linux, and Tru64 (with decreasing priority).