A Homegrown RPC Mechanism
Ruediger R. Asche
Microsoft Developer Network Technology Group
May 9, 1995
Abstract
This article is last in a series of technical articles that describe the implementation and application of a C++ class hierarchy that encapsulates the Windows NT® security application programming interface (API). The series consists of the following articles:
"Windows NT Security in Theory and Practice" (introduction)
"The Guts of Security" (implementation of the security class hierarchy)
"Security Bits and Pieces" (architecture of the sample application suite)
"A Homegrown RPC Mechanism" (description of the remote communication implemented in the sample application suite)
CLIAPP/SRVAPP, a sample application suite that consists of a database client and server, illustrates the concepts introduced in this article series.
Introduction
In "Security Bits and Pieces," I described the architecture of the CLIAPP/SRVAPP sample application suite that I wrote to demonstrate security programming under Windows NT. The only missing piece in that article is the discussion of the mechanism that allows the client and server to communicate over the network. This article fills that hole. I will describe the C++ class, CDatabaseProtocol, which is responsible for letting the client and server call each other's functions over the network. The CDatabaseProtocol class hierarchy, in effect, implements a small remote procedure call (RPC) protocol.
However, RPC is a well-abused buzzword that has several meanings, so here's my disclaimer: This article will not teach you anything about the RPC layer that is provided by Microsoft® operating systems, neither will it teach you to code for that interface. Instead, the article will introduce you to the kinds of problems that any RPC mechanism will need to solve, and will give you the option to code small rudimentary RPC layers for yourself.
Why do I reinvent the wheel and design my own RPC scheme instead of using the Windows NT RPC layer? There are two answers to this question: One that makes me look good, and one that makes me look like a dinkeldorf.
Let's start with the charming answer: Of course I know all about RPC, and I could have easily cranked out an application based completely on built-in RPC. However, I decided that introducing RPC with all the security stuff would result in a steep learning curve for readers.
The not-so-charming answer is that I am a liar. I did not use the RPC layer in Windows NT because I really don't know how to use it. (Well, I built and ran a few RPC applications, but I decided that it was too much work.) Nevertheless, I felt that introducing RPC with all the security stuff would result in too steep a learning curve for readers.
In either case, I decided to design my own RPC scheme. In a later article, I will simply unplug the networking layer and replace it with one that uses the Windows NT built-in RPC mechanism.
What Is RPC?
To answer this question, let me quote myself from the article "Communication with Class" in the MSDN Library:
And there are only two ways for two computers to communicate: via passing data back and forth or via calling each other's routines—though the latter strategy can be viewed as nothing but a variation of the first.
In the article series that begins with "Communication with Class," I describe the first communication strategy (passing data back and forth). If you look at the definition of the CCommunication class hierarchy, you will find that all classes in this hierarchy support the Read and Write members for sending and receiving data over a communication channel; thus, we are dealing with a data-exchange mechanism only.
RPC, very roughly, is a mechanism that allows one process to invoke a function that executes in another process. Let us look at the prototype of the CDatabaseProtocol hierarchy class to see what I mean by that:
class CDatabaseProtocol : public CProtocol
{
private:
CClientCommunication *m_cBothways;
public:
CDatabaseProtocol(CClientCommunication *);
BOOL Open(const char* pszFileName, UINT nOpenFlags,
CFileException* pError = NULL);
UINT Read(void FAR* lpBuf, UINT nCount);
void Write(const void FAR* lpBuf, UINT nCount);
void Close();
};
class CClientDatabaseProtocol : public CDatabaseProtocol
{
public:
CClientDatabaseProtocol(CClientCommunication *);
BOOL AddData(int *iIndex,CHAINLIST *clElement);
BOOL RemoveData(int iIndex);
BOOL RetrieveData(int iIndex, CHAINLIST *cpResult);
BOOL GetEntries(int *);
BOOL Terminate();
};
#ifdef SERVER
class CServerDatabaseProtocol : public CDatabaseProtocol
{
public:
CServerDatabaseProtocol(CServerCommunication *);
BOOL AcceptCommand(int *iCommand, CHAINLIST *cpElement, int *iIndex);
BOOL ProcessCommand(CHAINLIST *cpElement, int *iIndex);
BOOL Fail(int iErrorCode);
BOOL Acknowledge(int iIndex);
};
#endif
Remember what the CLIAPP/SRVAPP sample application suite does: The database that both the client and server work on resides on the server, but the database operations are accessible from both the client and the server. In other words, the server can call Insert, Delete, and Retrieve operations (recall that the database itself is implemented in the ChainedQueue object, which provides the Insert, Retrieve, and Remove member functions), and the client can also call the same set of operations—namely, CClientDatabaseProtocol::AddData, CClientDatabaseProtocol::RemoveData, and CClientDatabaseProtocol::RetrieveData.
The client, when working on the database, is not aware of the location of the database; it simply uses a functional interface that hides all of the details from the client application. (Note that the client calls a function that performs an operation on a database, which resides somewhere far away from the client, possibly somewhere on the network.)
This is basically what RPC is all about—it provides a mechanism for encoding a functional interface over the network. You can implement an RPC mechanism in many ways. The one I provide is very rudimentary, but it shows two pitfalls that you must take into consideration when designing functional interfaces across networks:
The interface must be logically separated from the network transport. In other words, a change in the underlying network bindings must not require any changes to the RPC code.
The RPC layer may have to convert any compound structures passed to functions before sending them over the network. Here is a simple example: Imagine that we want to implement an RPC function with the following prototype:
void RPCFunc(WEIRDSTRUCT ws);where WEIRDSTRUCT is any structure that contains indirect references (that is, pointers). Pointers are valid only within the context of the current application, so if we simply passed WEIRDSTRUCT over the network, the pointer would not make any sense on the receiving end of the function. Even worse, for the pointer to make sense, we must pack up whatever structure it points to, send it over the network as well, reassemble both structures on the receiving end, and correctly adjust the pointer in the copy of WEIRDSTRUCT.
Uh-oh. Weird thoughts come to mind: What about calls by reference? What about functions that operate on large chunks of memory? How do we know what to copy? Will we run into situations that require copying megabytes of memory from one machine to the other? These nasty thoughts are only the tip of the iceberg: Just imagine what would happen if a structure that you copied back and forth had to be serialized between the two processes—how do you claim a "remote mutex" to ensure mutual exclusion on the shared data structure? How do you ensure that the client does not hang if the RPC layer does not claim this mysterious "remote mutex" (once we know how to implement and use it), and the network dies? How do you implement decent error handling that catches network problems as well as the expected error returns?
RPC Architecture in the Sample Application Suite
The way RPC is implemented in the sample application suite is surprisingly straightforward—there are only two C++ class hierarchies, one of which (the CNamedPipe hierarchy) has already been discussed in detail (see "Garden Hoses at Work"). By the way, nothing forces the communication object to be a named pipe except for the fact that named pipes are easiest to secure in the Windows NT security model. For the sake of the RPC implementation, the RPC mechanism could be built around any other derivative of CClientCommunication (that is, a CClientNetBIOS or CClientSocket object, or any other derivative of CClientCommunication that you care to implement).
The other group of classes that is used to implement the RPC layer is CDatabaseProtocol and its derivatives, CServerDatabaseProtocol and CClientDatabaseProtocol. I have already introduced the base class, CProtocol, in the article "Communication with Class," where I used protocol objects to implement file transfer and chat abstractions over CCommunication objects.
In fact, I like to think of CProtocol derivatives as objects that translate some kind of object-specific data exchange into raw data exchange with the communication object. Sounds confusing? Recall the prototypes for the CDatabaseProtocol class hierarchy. Note that the member functions supported by the client side of the protocol (CClientDatabaseProtocol) are identical to the functions supported by the ChainedQueue class, which implements our little homegrown database. In other words, all that the client application "sees" of the RPC mechanism is an interface that looks exactly like the one that operates on the database.
However, to the network, the data looks considerably different. No matter which communication object we use to take care of the data transport, the network level contains no such thing as a "database," but only a raw stream of data to be sent back and forth. The protocol objects are responsible for translating the database API (or whatever API the RPC layer chooses to implement) into a communication API that the CCommunication object understands (which is simply Read and Write).
Let's see how the AddData function works on the client side to get an understanding of what's going on (from PROTOCOL.CPP):
BOOL CClientDatabaseProtocol::AddData(int *iIndex, CHAINLIST *clElement)
{
int iData;
// First write the stuff out.
iData = CMD_ADDRECORD;
_try
{
Write((char *)&iData,sizeof(int));
Write((char *)&clElement->iSecuredElement,sizeof(int));
Write((char *)&clElement->iInsecuredElement,sizeof(int));
// Then wait for the response.
Read((char *)&iData,sizeof(int));
if (iData == CMD_SUCCESS)
{
Read ((char *)iIndex,sizeof(int));
return TRUE;
}
else
{
Read((char *)&m_iErrorCode,sizeof(int));
return FALSE;
};
}
_except (EXCEPTION_EXECUTE_HANDLER)
{
m_iErrorCode = GetExceptionCode();
};
return FALSE;
};
The code first writes a command identifier (in this case, CMD_ADDRECORD) to the server. The server uses this command identifier to discriminate between commands. The client "flattens out" the data record to be transmitted by sending each of the elements individually, and then waits for a response from the server. The RemoveRecord and RetrieveRecord members work in a similar way.
Let's look at the server side of the transaction (also from PROTOCOL.CPP):
BOOL CServerDatabaseProtocol::AcceptCommand(int *iCommand, CHAINLIST *cpElement,
int *iIndex)
{
_try
{
if (Read((char *)iCommand,sizeof(int))!=sizeof(int)) return FALSE;
// Fetch the command first.
switch (*iCommand)
{ case CMD_ADDRECORD:
if (Read ((char *)&cpElement->iSecuredElement,sizeof(int))
!=sizeof(int)) return FALSE;
if (Read ((char *)&cpElement->iInsecuredElement,sizeof(int))
!=sizeof(int)) return FALSE;
return TRUE;
case CMD_DELETERECORD:
if (Read((char *)iIndex,sizeof(int))
!=sizeof(int)) return FALSE;
return TRUE;
case CMD_RETRIEVERECORD:
if (Read((char *)iIndex,sizeof(int))
!=sizeof(int)) return FALSE;
return TRUE;
case CMD_GETENTRIES:
return TRUE;
}; // switch
return TRUE;
}
_except (EXCEPTION_EXECUTE_HANDLER)
{
m_iErrorCode = GetExceptionCode();
};
return FALSE;
};
The AcceptCommand member function listens on [to?] the associated communication object and "decodes" an incoming request. The parameters for the AcceptCommand member specify addresses of variables that will receive the command identifier as well as provide additional information. For example, for an add record request, the CHAINLIST member stores the record that the client wants to add.
How is AcceptCommand used? Let us look at SRVVIEW.CPP:
long WINAPI PipeThreadFunction(CNpscesrvView *cvTarget)
{ char szDiagnosticMessage[255];
CServerDatabaseProtocol *cpProt;
BOOL bFinished;
ServerChainedQueue *cqTheQueue = cvTarget->m_cqQueue;
if (!cvTarget->m_cpPipe->AwaitCommunicationAttempt())
{
cvTarget->DisplayTextErrorMessage("Open named pipe failed -- %s",
cvTarget->m_cpPipe->m_iErrorCode);
goto ErrorExit;
}
else
{
sprintf(szDiagnosticMessage,"Open named pipe succeeded");
cvTarget->AddStringandAdjust(szDiagnosticMessage);
}
// Acknowledge communication to the UI.
cpProt = new CServerDatabaseProtocol(cvTarget->m_cpPipe);
if (!cpProt->Open("",CFile::modeReadWrite)) // We are server...
// Log an error here.
goto ErrorExit;
int iCommand,iIndex;
CHAINLIST cpElement;
bFinished=FALSE;
while (!bFinished) // We'll break out of this loop later...
{
if (!cpProt->AcceptCommand(&iCommand,&cpElement,&iIndex))
{
cvTarget->DisplayTextErrorMessage("accepting command from named pipe failed
-- %s",cpProt->m_iErrorCode);
bFinished = TRUE;
continue;
};
switch (iCommand)
{ case CMD_EXIT:
cvTarget->AddStringandAdjust("Client terminated connection!");
bFinished=TRUE;
break;
case CMD_GETENTRIES:
cpProt->Acknowledge(cqTheQueue->GetEntries());
break;
case CMD_ADDRECORD:
if (!cqTheQueue->SafeInsert(&iIndex,&cpElement))
{
cvTarget->DisplayTextErrorMessage("Remote insert failed; propagating
error code -- %s",
cqTheQueue->m_iErrorCode);
cpProt->Fail(cqTheQueue->m_iErrorCode);
}
else
{
cpProt->Acknowledge(iIndex);
cvTarget->AddStringandAdjust("Remote insert succeeded!");
};
break;
case CMD_DELETERECORD:
if (!cqTheQueue->SafeRemove(iIndex))
{
cvTarget->DisplayTextErrorMessage("Remote remove failed; propagating
error code -- %s",
cqTheQueue->m_iErrorCode);
cpProt->Fail(cqTheQueue->m_iErrorCode);
}
else
{
cpProt->Acknowledge(0);
cvTarget->AddStringandAdjust("Remote remove succeeded!");
};
break;
case CMD_RETRIEVERECORD:
if (!cqTheQueue->SafeRetrieve(iIndex,&cpElement))
{
cvTarget->DisplayTextErrorMessage("Remote retrieve failed;
propagating error code --
%s",cqTheQueue->m_iErrorCode);
cpProt->Fail(cqTheQueue->m_iErrorCode);
}
else
{
cpProt->Acknowledge(cpElement.iInsecuredElement);
cpProt->Acknowledge(cpElement.iSecuredElement);
cvTarget->AddStringandAdjust("Remote retrieve succeeded!");
};
break;
}; // switch
}; // while
cpProt->Close();
delete (cpProt);
cvTarget->m_cpPipe->CloseInstance();
ErrorExit:
CloseHandle(cvTarget->m_hThread);
cvTarget->m_bThreadIsActive = FALSE;
return 0;
}; // Thread fn
On the server side, the logic for responding to client requests is located in a secondary thread whose thread function we just saw. The thread repeatedly calls AcceptCommand on its associated CServerDatabaseProtocol object until it encounters the CMD_TERMINATE command identifier. Each command is dispatched to the associated database object. Note that by using the SafeInsert, SafeRemove, and SafeRetrieve members of the ChainedQueue database object, the security check is made in these members; if access to the database is denied, the code sends the appropriate error code to the client.
After processing a command, the server either sends a failure code for the client to scrutinize, or sends a success code, possibly including more information (such as the values of the retrieved element in response to a Retrieve request). The server side of the CDatabaseProtocol class supports the two members Fail(int iErrorCode) and Acknowledge(int iVal) for this purpose.
Note that the "serialization" of data structures (that is, the conversion from an arbitrary data structure to a "raw" stream of bytes and vice versa) can be anything between a no-brainer and a nightmare, depending on the complexity of the data structures to be transferred between the server and client processes. In the CLIAPP/SRVAPP application suite, the serialization is a no-brainer, because I kept my sample database very simple. More complex data structures naturally involve more work. One way to simplify the task would be to delegate serialization to, say, a Serialize member function.
Summary
The database architecture that is used to demonstrate security in the CLIAPP/SRVAPP sample application suite is rather primitive. As a result, an RPC layer that implements remote database operations on this architecture can be designed rather easily. The separation between communication objects and protocol objects provides an elegant and clean framework for translating complex and "raw" interfaces back and forth.
Derivatives of the protocol object may have to be considerably more complex to accommodate more elaborate interfaces.
Although the protocol/communication RPC architecture I provided is not as powerful as the built-in RPC mechanism in Windows NT, you can use this architecture to implement simple functional client-server interactions.