\documentclass[11pt]{article}
\usepackage{times}
\usepackage{graphics}
\usepackage{tight}
\usepackage[dvips]{changebar}
\usepackage[USletter]{vmargin}
\input{../base/defs}
\newcommand{\ps}[4]{\begin{figure}[htbp!]
\centerline{\resizebox{#1}{#2}{\includegraphics{#3.eps}}}
\caption{#4}\label{fig:#3}
\end{figure}}

\newcommand{\MUST}{\textsc{must}}
\newcommand{\MUSTNOT}{\textsc{must not}}
\newcommand{\REQUIRED}{\textsc{required}}
\newcommand{\SHALL}{\textsc{shall}}
\newcommand{\SHALLNOT}{\textsc{shall not}}
\newcommand{\SHOULD}{\textsc{should}}
\newcommand{\SHOULDNOT}{\textsc{should not}}
\newcommand{\MAY}{\textsc{may}}
\newcommand{\OPTIONAL}{\textsc{optional}}
\newcommand{\RECOMMENDED}{\textsc{recommended}}
\newcommand{\NOTRECOMMENDED}{\textsc{not recommended}}

\setpapersize{USletter}

\begin{document}

\title{SIPstone - Benchmarking SIP Server Performance\thanks{This work
was supported by grants from SIPcomm, Clarent Communications and Lucent
as well as equipment grants from Sun Microsystems and Compaq.}}

\author{Henning Schulzrinne 
\and Sankaran Narayanan 
\and Jonathan Lennox 
\and Michael Doyle\\
Columbia University, Ubiquity\\
{\sf \{hgs,sankaran,lennox\}@cs.columbia.edu, {\sf mdoyle@ubiquity.net}}
}

\maketitle
\centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{SIPstone1.ps}}}

\begin{abstract}
SIP-based Internet telephony systems need to be appropriately
dimensioned, as the call and registration rate can reach several
thousand requests a second.  This draft proposes an initial simple set
of metrics for evaluating and benchmarking the performance of SIP proxy,
redirect and registrar servers.  The benchmark SIPstone-A expresses a
weighted average of these metrics.
\end{abstract}

\section{Introduction}
\label{sec:intro}

SIPstone is a benchmark for SIP (Session Initiation
Protocol~\cite{rfc2543}) proxy and redirect Servers (SIP Server).  The
benchmark attempts to measure the request handling capacity of a SIP
server or a cluster of SIP servers.

SIPstone is designed to be a benchmark with a single standardized
implementation and workload.  The implementation performs a series of
tests that generate the pre-configured workload.  This workload is
performed in a controlled IP telephony test bed that simulates the
activities of multiple users initiating SIP calls.  The workload is
intended to be simple and repeatable, and exercises a breadth of system
components associated with this environment such as concurrent
initiation of calls by multiple users, random call arrivals and the
forwarding of requests.

The benchmark currently is limited to evaluating the sustainable rate of
what we believe to be a representative workload, consisting of
registrations, successful forwarding and unsuccessful call attempts. 
The relative importance of these operations will differ significantly
between different installations.  For example, in a call center, each
registered user will register once an hour, but probably receive on the
order of 20 calls during that time.  On the other hand, for a
residential or office phone, the call rate will be less than one per
hour, while maintaining the same registration interval.  Unsuccessful
calls, where the user is temporarily unreachable, will play a role for
networks with primarily wireless end systems that are frequently turned
off.

This benchmark is an initial attempt at characterizing server
performance in a way that is useful for dimensioning and provisioning. 
However, it intentionally omits a number of more complicated scenarios,
such as parallel or sequential forking or the processing of sip-cgi
\cite{rfc3050} or CPL \cite{draft-ietf-iptel-cpl} services.  Given lack
of operational experience, it is hard to predict what type of
programmable services will find widespread use or how common forking
proxies will be in the core of the network.  (For many enterprise and
residential proxy servers, performance issues are of secondary concern.)

Note that the benchmark has a number of self-imposed limitations.  The
benchmark does not evaluate protocol compliance or robustness.  There
are separate tests for parser robustness
\cite{draft-ietf-sip-call-flows}, while testing behavior under signaling
overload is the subject of on-going work.  To limit the time it takes to
perform the test suite, individual tests run 15 minutes, making it
possible that a server that leaks memory, for example, could evade
detection.

Finally, a benchmark is only one metric out of many in evaluating
servers and does not reflect functionality.  For example, a server could
be fast, but not support SNMP-based management and request counting.

The remainder of this document is organized as follows: 
Section~\ref{sec:sip} presents an overview of SIP servers, and
illustrates SIP call setup with several examples.  Issues involved in
designing a benchmark for an Internet telephony system are discussed in
Section~\ref{sec:sipbenchmark}.  Section~\ref{sec:sipstone} presents the
architectural features of the benchmark such as the test bed components
and load generation mechanisms.  Performance metrics for SIP Servers are
defined in Section~\ref{sec:performance}.  Section~\ref{sec:performance}
also details the measurement methodology used for benchmarking. 
Section~\ref{sec:limitations} discusses issues that are to be addressed
in the next revision.  Section~\ref{sec:conclusion} presents
conclusions.

\section{SIP Servers}
\label{sec:sip}

Before we look at benchmarking mechanisms, it is necessary to be
familiar with the categories of SIP servers and their operation. 
Readers familiar with SIP may skip this section.  SIP supports user
mobility by allowing addresses in the form of {\em user@domain} to be
used when making Internet calls.  For instance, {\em alice@home.com} can
call {\em bob@office.com} no matter what communication device, IP
address or phone number bob is using currently.  The locations where
users could be reached are maintained by the SIP location or
registration servers. 

When Alice, with address {\em sip:alice@home.com}, wants to call Bob,
{\em sip:bob@office.com}, her SIP phone contacts the server at {\em
office.com}.  The server knows where Bob can be reached and can either
return Bob's location to Alice's phone (so-called redirect mode) or can
itself try to contact Bob at his current location (proxy mode).  In the
former case, Alice's phone retries the new location, while in the latter
case the server proxies the request, transparent to the caller.  It is
possible to encounter multiple SIP servers (either in redirect or proxy
mode) in a given call attempt.  A {\em forking proxy} can fork the call
request to more than one locations, so that the first phone that is
picked-up gets the call, while all the other phones stop ringing.

The list of audio/video algorithms supported and the transport addresses
to receive them, are described using Session Description Protocol
(SDP~\cite{rfc2327}) messages, carried as the body of SIP requests and
responses.

\section{Issues in Designing a SIP Benchmark}
\label{sec:sipbenchmark}

The goal of any benchmarking system is to help understand how a system
will perform, and to help compare different implementations.  The
primary SIP performance metric is the number of requests that a system
can process successfully within a given delay constraint, while the
request processing delay itself is of somewhat secondary importance as
long as it remains below the acceptable ``post-selection delay'' and
``answer signal delay'' \cite{E.721}.  For informational responses, the
delay needs to be sufficiently short to avoid triggering UDP
retransmissions, i.e., about 500\,ms minus any network delays.  The
performance of a SIP server depends not only on the implementation but
also the size of the user population serviced, the number of DNS lookups
required, the transport protocol used (UDP or TCP), and the type and
statistical arrival process of requests submitted.  We discuss some of
these considerations in more detail in the sections below.

\subsection{User Population}
\label{sec:population}

The performance of a SIP server that has per-user information is likely
to depend on the size of the user population being reached by the
server, since a typical SIP transaction involves looking up the contact
addresses of a given user in a database of some kind.  There are at
least three different approaches for implementing such a database,
including an in-memory table without persistent storage, in-memory
caching with disk write-through and a database located on a networked
server.  Depending on the cache size and the number of users
registering, the performance will differ dramatically and will be
determined largely by the back-end database, less by the server.  Since
it appears likely that the user population served by carrier-scale SIP
servers is likely to be large, measured in thousands, our benchmark
attempts to model this scenario.  However, the evaluation of a server
needs to ascertain whether the user population in the benchmark is a
fair representation of the actual user population size and whether the
server performs in-memory caching that may decrease in effectiveness for
a larger population.

However, readers should be aware that different database architectures
have trade-offs other than performance.  For example, web-based user
management may require the use of networked databases.  Also, in-memory
databases, while faster, generally cannot maintain registration
information across system reboots.

Finally, for outbound proxy servers, database interactions are limited
to authentication lookups, rather than registration lookups.  However,
we believe that outbound proxy servers are typically installed in
corporate networks and thus have a lower call volume.  (For example, if
we estimate that during the busy hour, one fourth of the IP phones are
in use and that a business call generally lasts three minutes, we would
expect a call volume of about 5 BHCA (busy hour call attempts) per user,
or about 100,000 outbound calls during an hour for a large campus with
20,000 phones.  This is about 28 call attempts per second, which is
likely to be well within the capacity of most servers.) We thus treat
outbound proxies, with authentication, as a separate case.

\subsection{Request Modeling}

Traditionally, a Poisson process\footnote{A Poisson arrival process is
equivalent to an exponential interarrival time.} is used to characterize
PSTN call arrivals~\cite{Bell91:Digital}.  For system sizing purposes,
generally the busy hour call volume is used as a requirement, indicating
the time-average arrival rate during the busiest hour of the busiest day
or an average of the $n$ busiest days of the year
\cite{Schm9904:Analytical}. 

The time between setup and teardown is the conversation time or the call
holding time.  The exponential distribution is the traditional method of
approximating call holding times in a switch, although it is only
modestly accurate \cite{FCC0012:Trends}.  Since SIP proxy servers mostly
maintain transaction state, not call state, we can largely ignore the
call duration statistics.  More important for SIP servers is the time
between call arrival and answer, as it determines the number of
transaction records that the server needs to maintain. Using the
statistics from \cite{Duff7801:Study}, about 70\% of the calls are
answered, in roughly 8.5 seconds, while unanswered calls ring for 38
seconds. Thus, an average {\INVITE} transaction duration of 20 seconds
is chosen.

For call stateful (user agent) servers, the shape of the distribution of
call durations may have some influence on the number of calls present in
the system, but is not likely to affect the call throughput.

\subsection{Standard Call Transaction}

Some Internet services, such as HTTP, DNS and LDAP, have a very simple
request-response structure, where measuring the response time is fairly
straightforward.  However, SIP requests often generate multiple
provisional responses and may even generate, when forking, multiple
final responses.  The ``100 Trying'' response needs to be returned soon
enough to avoid retransmissions of the request, as excessive
retransmissions will quickly lead to server congestion collapse.  Hence,
in the request mix used to benchmark a server, we need to consider the
response time of a full transaction such as {\INVITE}, ``100 Trying'' or
``180 Ringing'' provisional response and the ``200 OK'' final response. 

\subsection{Other SIP Issues}

Other SIP specific issues include the use of different transport
protocols such as TCP and UDP.  When using TCP, issues such as the
ability of the server to handle a large number of open connections and
its use of persistent connections need to be considered.  We currently
do not have a good estimate as to the typical number of upstream servers
reaching a high-volume proxy server.  If the number is low, almost all
such servers will use persistent connnections, while, for example, an
outbound proxy server serving a typical office may see a new TCP
connection for every request.  Until we have more operational
experience, we have each thread in our load generator maintain a
persistent connection to the server under test.

When UDP is used, we can ignore the effect of network packet losses for
requests, since they do not affect server load, but need to consider
retransmissions of requests and final responses from downstream servers
if the initial response or the {\ACK} request are excessively delayed by
the server under test.  For simplicity, we ignore all network packet
losses, since it appears likely that operational networks will have to
have very low losses, typically below 5\%, to achieve good voice quality
and acceptable call setup delays. 

Mechanisms such as forking and \header{Via}-header processing impose
higher processing demands on the proxy server.  Forking in SIP Proxy
Servers was discussed in Section\ref{sec:sip}.  When a chain of proxy
servers are used, handling of the Via header becomes critical in
processing the message.  The workload should be able to make the proxy
server perform such computations.  In this version of this document, we
do not consider forking and Via-header processing.  In addition to the
above, size of the SDP used with SIP requests has the potential to
affect performance.  A large SDP may result in several datagrams when
UDP is used. 

\subsection{General Benchmarking Considerations}

Other relevant issues to consider when designing a benchmark include
repeatability, measurability, and scalability.  Repeatability refers to
the ability to replicate the results by a third-party when running the
test under the specified environment.  Measurability refers to the ease
with which the measured test results can be quantified in a way to
present a picture of the system being measured.  The ability of the
benchmark to run on different configurations such as simulating a large
number of users, different arrival distributions refers to scalability. 
We elaborate on these issues further in the next section.

\section{SIPstone}
\label{sec:sipstone}

SIPstone describes a workload for SIP requests in proxies using a set of
tests that exercise various components of typical SIP servers.

\subsection{Architecture}

The ``server under test'' (SUT) is a SIP proxy, redirect or registrar
server whose performance is to be estimated.  The benchmark consists of
a set of SIPstone load generators that create the SIP request load, a
call handler that simulates a user agent server and a central benchmark
manager (``coordinator'') that coordinates the execution of the
benchmark, and the SUT.  Actual measurements are based on the controlled
workload profile described in Section~\ref{sec:performance}.  The call
handlers may run along with the load generators or on different systems.

\subsubsection{Server under Test}
\label{sec:sut}

The SUT consists of the host system(s), including hardware and software,
required to support the SIP telephony server and any other components
including database(s), if applicable.  All network components (hardware
and software) between host machines which handle intra-SUT application
communications are part of the SUT.  The software consists of the SIP
redirect and/or proxy server.  This software is referred to as the SIP
server in the discussion below.  For simplicity, this document
considers the registrar server to be co-located with the SIP server.

For the test, file-based logging must be enabled, so that each request
is written to a log file. If the system does not support file-based
logging, an equivalent logging mechanism must be enabled and this fact
be noted in the description.

\subsubsection{User Agent Client (UAC)}
\label{sec:uac}

SIP requests for the benchmark are generated by user agent clients
(UACs).  Depending on the request rate needed, one or more clients may
be needed to generate the requests. The requests are directed to the UAS
(Section~\ref{sec:uas}). If only one UAC is used, it is configured with
the following information:
\begin{itemize}
\item the interarrival time for requests for each host;
\item the type of request ({\INVITE} (implies {\BYE}) or {\REGISTER}), 
\item the transport protocol (UDP or TCP),
\item the number of requests to generate.
\end{itemize}

The user names are chosen as counting from \textsf{A000000}.

\subsubsection{Coordinator}
\label{sec:coordinator}

If more than one test UAC is needed, a coordinator reads the input
benchmark configuration file.  The benchmark configuration file contains
\begin{itemize}
\item the names of the UAC hosts;
\item the names of the UAS hosts;
\item the interarrival time for requests for each host;
\item the type of request ({\INVITE} (implies {\BYE}) or {\REGISTER}), 
\item the transport protocol (UDP or TCP),
\item the number of requests to generate,
\item the starting user name (e.g., \textsf{A013878}).
\end{itemize}

The coordinator invokes the UACs and UASs on each of the participating
computers using mechanisms such as rsh/rcmd or Java RMI.  Each UAC
generates SIP requests with a Poisson interarrival distribution and
records the response delay, i.e., the time between sending the request,
the arrival time of the 1xx response, if any, and the arrival time of
the 2xx response.

\subsubsection{User Agent Server (UAS) -- Call Handler}
\label{sec:uas}

The UAS is needed for the ``Proxy 200'', ``Outbound proxy'' and
``Redirect'' tests (see Section~\ref{sec:tests} below).  For the
``Outbound proxy'' and ``Proxy 200'' tests, the UAS answers incoming
{\INVITE} and {\BYE} requests.  One or more UAS are used.  Upon receipt
of an {\INVITE} request, they answer immediately with a ``180 Ringing''
response, followed instantly by ``200 OK''.  For the ``Redirect'' test,
the UAS only registers its location.

The test UAS {\MUST} be able to respond, under load, to an incoming
{\INVITE} or {\BYE} within 100\,ms.

Prior to the benchmarking run, the UAS registers with the registrar. 
For reproducibility, the UAS registers the names \textsf{A000000},
\textsf{A000001}, \textsf{A000002}, etc., with the number of
registrations large enough that each user name will be used only once.

\subsection{Description of Tests}
\label{sec:tests}

The benchmark currently consists of five tests.  Each test is performed
using both UDP and TCP, with results reported separately for each
transport protocol.  Thus, a total of ten results is to be reported for
servers that support both transport protocols. The individual tests are:

\begin{description}
\item[Registration:] Registrations of the form shown in
Fig.~\ref{fig:register1} and Fig.~\ref{fig:register2} are sent by the
load generator, using digest authentication, corresponding to call flow
2.1.1 \cite{draft-ietf-sip-call-flows}.  The user accounts have been
established previously in a manner depending on the SUT.  For
simplicity, account name and user secret are typically chosen to be the
same.  The message flow is shown in Fig.~\ref{fig:register200}.  The
transaction delay is measured from sending the first {\REGISTER} request
to receiving the final 200 (OK) response, including the 401
``Unauthorized'' message. 

\ps{!}{0.5\textheight}{register200}{Registration test}

\item[Outbound proxy:] The load generator sends an {\INVITE} request to
the SUT acting as an outbound proxy.  The message has the format shown
in Fig.~\ref{fig:invite} and is destined to the UAS. The message flow is
the same as for the ``Proxy 200'' test below.

\item[Redirect:] The load generator sends an {\INVITE} request to the
SUT acting as a redirect server, using the message format in
Fig.~\ref{fig:invite}.  It is assumed that all call destinations have
registered with that server.  The message flow is shown in
Fig.~\ref{fig:r3xx}. The transaction delay is measured from sending the
{\INVITE} request to receiving the 3xx response.

\ps{!}{0.5\textheight}{r3xx}{Redirect test}

\item[Proxy 480:] The load generator sends an {\INVITE} request to the
SUT acting as a redirect server, using the message format in
Fig.~\ref{fig:invite}.  The call destinations are known to the server,
but have not registered, so that the server returns a 480 (Temporarily
unavailable) response.  The request is not authenticated.

\item[Proxy 200:] The load generator alternates between {\INVITE} and
{\BYE} transactions to the SUT acting as a non-forking proxy server. 
The {\BYE} transaction is sent immediately after the {\INVITE}
transaction completes.  The message flow is shown in
Fig.~\ref{fig:r200}. The requests are not authenticated.

The server should insert a \header{Record-Route} to reflect real
operational conditions, however, the operation of the system does not
depend on this since the caller load generator always sends the request
to the SUT. The delay bound noted in Table~\ref{tab:response} applies to each
individual request, i.e., {\INVITE} and {\BYE}, individually.

The TRT is measured only for the {\INVITE} request.

\ps{!}{0.5\textheight}{r200}{Proxy 200 test}

\end{description}

The first {\REGISTER} request is shown in Fig.~\ref{fig:register1}, the
second, authenticated {\REGISTER} in Fig.~\ref{fig:register2} while the
{\INVITE} request is shown in Fig.~\ref{fig:invite}.

\begin{figure}[htpb!]
\begin{tt}
REGISTER sip:\emph{registrar} SIP/2.0\\
Via: SIP/2.0/UDP \emph{origin}\\
From: $<$sip:$>$;tag=1\\
To: sip:\emph{registrand}\\
Call-ID: \emph{call-id value}\\
CSeq: 1 REGISTER\\
Contact: $<$sip:\emph{IP address}$>$\\
Expires: 7200\\
Content-Length: 0
\end{tt}
\caption{Initial {\REGISTER} request}
\label{fig:register1}
\end{figure}

\begin{figure}[htpb!]
\begin{tt}
REGISTER sip:\emph{registrar} SIP/2.0\\
Via: SIP/2.0/UDP \emph{origin}\\
From: $<$sip:\emph{registrand}$>$;tag=1\\
To: sip:\emph{registrand}\\
Call-ID: \emph{call-id value}\\
CSeq: 1 REGISTER\\
Contact: $<$sip:\emph{IP address}$>$\\
Expires: 7200
Authorization:Digest username="\emph{registrand}",\\
~ realm="SIPstone",\\
~ nonce="ea9c8e88df84f1cec4341ae6cbe5a359", opaque="",\\
~ uri="\emph{registrar}", response="dfe56131d1958046689cd83306477ecc"\\
Content-Length: 0
\end{tt}
\caption{Authenticated {\REGISTER} request}
\label{fig:register2}
\end{figure}

\begin{figure}[htpb!]
\begin{tt}
INVITE sip:\emph{destination} SIP/2.0\\
Via: SIP/2.0/UDP \emph{caller address}\\
From: Alice $<$sip:\emph{destination}$>$;tag=17\\
To: Bob <sip:\emph{origin}>\\
Call-ID: 602214199@mouse.wonderland.com\\
CSeq: 1 INVITE\\
Contact: Alice $<$\emph{origin}$>$\\
Subject: SIP will be discussed, too\\
Content-Type: application/sdp\\
Content-Length: 187\\
\\
v=0\\
o=user1 53655765 2353687637 IN IP4 128.3.4.5\\
s=Mbone Audio\\
t=3149328700 0\\
i=Discussion of Mbone Engineering Issues\\
e=mbone@somewhere.com\\
c=IN IP4 \emph{caller IP address}\\
t=0 0\\
m=audio 3456 RTP/AVP 0\\
a=rtpmap:0 PCMU/8000\\
\end{tt}
\caption{{\INVITE} request}
\label{fig:invite}
\end{figure}

\section{Benchmarking Methodology}
\label{sec:performance}

\subsection{Environment}

SIPstone benchmarking consists of a series of test runs, with increasing
load levels generated by the load generators, and targeted at the SIP
server being tested.  

\subsection{Definition of Terms}
\begin{description}
\item[Measurement Interval (MI):] The \textbf{measurement interval} is
defined as the steady state period during the execution of the benchmark
from which the reported performance metric is derived.  During the
measurement interval, the UACs generate SIP requests.

\item[Transaction Response Time (TRT):] The \textsf{transaction response
time} is defined as the time elapsed from the first byte sent by a UAC
to initiate a transaction until the last byte received by the UAC that
ends the transaction.  For the ``proxy 200'' test case, a transaction
can be either the {\INVITE} or the corresponding {\BYE} transaction, but only
the {\INVITE} transaction is measured. 

\item[Registrations per second (RPS):] \textbf{Registrations per second}
is defined as the average number of successful registrations per second
during the measurement interval.

\item[Calls per second (CPS):] \textbf{Calls per second} is defined as
the average number of calls per second completed with a 2xx or 4xx
response during a measurement interval.  For the ``proxy 200'' test
case, a single call includes both the {\INVITE} and corresponding {\BYE}
transaction.

\item[Transaction failure probability (TFP):] The transaction failure
probability is the fraction of transactions that fail, i.e, where the
server does not return a provisional or final response within the time
limit or where the response is a 5xx response.

\end{description}

In the future, we may define a cost metric, \$/CPS or \$/RPS, which
measures the cost, in U.S.  dollars, per call or registration per
second, including all server hardware and software.  (A similar metric
is commonly used for transaction processing systems.  SIP server
functionality can be spread relatively easily across a number of network
hosts, so that this metric is useful for network planning.)

\subsection{Measurement Methodology}

To determine the RPS and CPS values, the request rate is increased until
the TFP increases to 5\%, evaluated across a measurement interval of 15
minutes.  The highest sustained throughput is reported as the benchmark
number.  To speed up the benchmarking process, the measurements that
probe for the capacity can be shorter than 15 minutes, with five minutes
generally sufficient.  For more detailed benchmarks, the TRT as a
function of the transaction rate should be plotted, with at least four
measurements recommended, including one at a load below 10\% of the
maximum RPS or CPS.  The latter value is useful to estimate the
idle-server TRT.

Prior to each measurement interval, the SUT is initialized, as
necessary, with the user database and registrations. 

The test operates in an ``open loop'' mode, where the arrival of the
$n+1$st request does not depend on the completion of the $n$th request
for the same UAC\footnote{In a closed-loop test, each new request is
only issued after the completion of the previous one and a suitable
waiting time.}

\subsection{Scaling Requirements}

The size of the user population used for the tests should scale with the
request handling capacity of the server.  Assuming an average of one
hour for registration intervals and inter-call intervals yields a user
population of 3.6 million for a server capable of handling 1000 calls or
registrations per second.  However, for reasonable measurement
intervals, we cannot reach this number of destinations, and are likely
to spend an inordinate amount of time priming the system with user data. 
Thus, instead we simply require that each request during the measurement
interval has a different user name, except that the {\BYE} request in
the ``Proxy 200'' measurement corresponds to a previous {\INVITE}
request.

\subsection{Response Time Constraints}

During each measurement interval, at least 95\% of call setup requests
of each type must have a TRT of less than the constraint specified (in
milliseconds) in Table~\ref{tab:response}.  The values except for the
time-to-200 threshold are designed such that they are less than the SIP
retransmission default ($T1$) timer of 500\,ms. 

The time-to-200 threshold depends on the call model.  E.721 \cite{E.721}
suggests a 95th percentile value of 6.0 seconds for local ISDN calls
under normal load (8.0 seconds for toll and 11.0 seconds for
international connections), assuming between one and 10 nodes.  One
model is that SIP calls will traverse at least two proxy servers
(outbound and remote) and possibly more (e.g., in 3GPP), so that a
conservative upper bound of four servers appears reasonable, for both
``toll'' (i.e., inter-provider) and international calls.  This leaves a
delay budget of 2.0 seconds per server for each request ({\BYE} and
{\INVITE}).

Instead of triggering a retransmission when the timer elapses, the
worker marks it as failure and continues with the next iteration.  The
average TRT of a measurement interval is computed by averaging over the
successful (timely) responses.

\begin{table*}[hbtp!]
\begin{center}
\begin{tabular}{|l|r|r|p{3in}|} 
\hline
Scenario & Response & Response Time (ms)  &  Description \\
\hline
Figure~\ref{fig:register200} & 401 & 200 & Final response before retransmission occurs \\
\hline
Figure~\ref{fig:r200},\ref{fig:r3xx} & 1xx & 100 & Initial 1xx in all scenarios \\
\hline
Figure~\ref{fig:r3xx} & non-1xx &   400 & For redirect servers, before retransmission occurs \\
\hline
Figure~\ref{fig:r200} & 200 & 2,000 & For a simple SIP call through the proxy.\\
\hline
\end{tabular}
\caption{Allowed Response Times}
\label{tab:response}
\end{center}
\end{table*}

\subsection{Metrics and Parameters to be reported}

When presenting SIPstone results, the following data has to be provided.
The configuration parameters are constant over a test run.

\begin{itemize}
\item Description of the clustering configuration, including the number
of hosts and the load balancing mechanism used (e.g., DNS SRV or stateless
proxy);

\item All aspects of the server hardware, in particular 
\begin{itemize}
\item the CPU count, type and speed,
\item the memory configuration,
\item the network interface type and speed,
\item the disk and disk controller configuration;
\end{itemize}

\item the server operating system and version and any non-standard
tunings or settings, e.g., for network parameters;

\item the server software configuration, including 
\begin{itemize}
\item the number of servers used,

\item the type of location server (e.g., a local database, in-memory, or
a network server) and whether it was located on the same host as the
proxy server, 

\item request logging used (note that basic file-based logging is
required by default),

\item network management features in use;
\end{itemize}

\item the number and type of SIPstone load generators that participate
in the run, and the number of worker threads created in each generator;

\item the number and type of network segments connecting the load
generators to the SIP server;

\item the arrangement of load generators on the network segments;

\item the workload parameters except the load level, which may be
increased across several test runs.

\end{itemize}

To remove one variable from the measurement results, the server and load
generators should be connected by a switched 100\,Mb/s Ethernet that
only carries measurement-related traffic and no other LAN traffic.  This
ensures that network loading does not become a bottleneck.  (At 100
calls a second, with 1500 bytes each, the data rate for signaling is
about 1.2\,Mb/s.)

The following benchmark results should be reported:

\begin{itemize}
\item The number of connections requested by the clients and accepted by
the SUT per second.  The intent is to count only the number of new
connections made successfully by the clients in generating the load for
the benchmark.

\item CPU and memory utilization of server at various loads;

\item A curve plotting the TRT as a function of request arrival rates,
with at least four plot points and one value at approximately 10\% of
the capacity;

\item CPS and RPS.
\end{itemize}

When reporting results for server clusters, both the total, aggregate
CPS/RPS and the per-server rate should be reported. For clusters, it is
suggested to include a reasonable sample of cluster sizes, e.g., in
geometric progression (1, 2, 4, 8, \ldots).

We also define an initial composite benchmark called \textbf{SIPstone-A}
that weighs the ten processing rate metrics as follows:

\begin{flushleft}
\begin{tabular}{lrr}
Test & weight UDP & weight TCP\\
\hline
Registration   & 0.20 & 0.05\\
Outbound Proxy & 0.10 & 0.05\\ 
Redirect       & 0.10 & 0.05\\
Proxy 480      & 0.10 & 0.05\\
Proxy 200      & 0.20 & 0.10\\
\end{tabular}
\end{flushleft}

The number is to be reported as a unit-less number, for example, ``1234
SIPstone-A''.

\section{Limitations}
\label{sec:limitations}

Given the large number of test results, it may be worthwhile to create
two composite numbers, namely a ``call center'' number and a ``mobile
users'' throughput.  For the call center metric, we assume that there
are 20 calls per registrations, while mobile users have one registration
per call.  Separate metrics are to be reported for redirect and proxy
mode, since many servers are operating as one or the other.

It remains to be evaluated whether call metrics can be combined
linearly.  For example, a server that can handle 1000
registrations/second or 200 redirections/second should also be able to
handle a mixed load of 500 registrations/second and 100
redirections/second or 800 registrations/second and redirections/second.

More complicated scenarios, such as invocations of SIP servlets, sip-cgi
and CPL scripts, need to be considered.  However, given that processing
requirements for each script can be effectively unbounded, depending on
the internal computation and external references needed, a set of simple
services will need to be proscribed.  For example, a simple time-of-day
call forwarding mechanism may be indicative of the efficiency in script
invocation and processing.

Also, future tests will need to incorporate SSL and IPsec scenarios if
these become practically important. Currently, few systems appear to use
these security mechanisms.

\section{Summary of Requirements}

Below, $R$ is the request handling rate.

\begin{flushleft}
\begin{tabular}{lrl}
Requirements & value & unit\\
\hline
User population               & 900 max(RPS, CPS) & \\
Call holding time             &   0 & s\\
Maximum ``100'' response time & 0.1 & s\\
Maximum ``200'' response time &   2 & s\\
Run time                      & 900 & s\\
Transaction failure prob.     &   1 & \%\\
\end{tabular}
\end{flushleft}

\section{Conclusions}
\label{sec:conclusion}

This draft summarizes an early benchmarking scheme for SIP telephony
servers.  We discussed several issues to be considered while designing
benchmarks for SIP servers, and presented an architecture and a
technique to measure server performance.

\bibliographystyle{ieeetr}
\bibliography{string,rfc,i-d,itu,net78,net91,net94,net98,net99,net00,net01}

\section{Acknowledgements}

Bertrand Farabet (Sun) provided valuable comments.

\end{document}
