The FreePastry Tutorial.
This tutorial is designed to get you cooking quickly with the FreePastry
API and software toolkit.
Version @tutorial_version@; @tutorial_date@. For FreePastry version @freepastry_version@. Maintained by @maintainer@.
Transport Layers
Modify low level details of FreePastry's network stack.
This tutorial will show you how to use the new TransportLayer interface in the package org.mpisws.p2p.transport. It will also show you how to add your TransportLayer to the SocketPastryNodeFactory. We will start with the goals of the the Layered Transport Layer. Versions of FreePastry before 2.1 had a unified transport layer. When features needed to be added or modified, it was very complicated to get all of the parts to work properly. The new version rearranges the transport layer into a stack of layers where each has it's own small task. To give you an idea of what a typical task of a layer, we will describe the layers that are assembled when creating the SocketPastryNodeFactory in FreePastry 2.1, beginning with the lowest layer:
SocketPastryNodeFactory's Layers:
- Wire -- Opens/Accepts Sockets, Sends/Receives Datagrams
- MagicNumber -- Throws away sockets/datagrams for other applications (if it doesn't match the application specific magic number)
- MultiInetAddressTransportLayer -- Handles multi-addressing (ex, when using a NAT a node may have more than 1 address:port, the internal address, and the NAT's external address that is forwarded
- SourceRoute -- sends messages/opens sockets along a source route, this layer manages both the endpoints and the intermediate nodes, note that this layer does not determine the optimal route, to an end host, that is done by another layer, the SourceRouteManager.
- LowerIdentity -- This layer, in conjunction with the UpperIdenity maintains the "intention" of the sent/received message. For example, if a node has restarted with a different NodeId this layer drops pings/sockets if they were intended for the previous node at this address.
- Liveness -- Adaptively pings nodes when requested to determine liveness/proximity, implements 2 new interfaces: LivenessProvider, Pinger
- SourceRouteManager -- Chooses the appropriate SourceRoute based on the liveness/proximity of the lower layer, implements another new interface ProximityProvider
- Priority -- Uses a single socket to send messages. Can select the order of the messages based on the priority.
- UpperIdentity -- (See Lower Identity) This layer keeps track of the intended destination of the message so that the lower layer can properly encode that intention.
- CommonAPI -- Serializes/Deserializes messages from a RawMessage to a ByteBuffer.
Other interesting layers:
- DirectTransportLayer -- (Currently in rice.pastry.direct) This implements the discreet event simulator.
Upcoming layers:
- SSL -- Provides Crypto/Authentication (typically goes above the SourceRoute layer to provide end-to-end crypto/auth)
- BandwidthLimiting -- Limits the Bandwidth of a node (typically goes between MagicNumber/SourceRoute Layers)
- PeerReview -- Provides protocol accountability (typically goes near the top, such as between CommonAPI/Priority layers)
- STUN -- would likely replace the Wire Layer and provide NAT hole-punching
The basic interface:
public interface TransportLayerEach transport layer operates on an Identifier (InetSocketAddress, SourceRoute, NodeHandle etc), and a MessageType (ByteBuffer, RawMessage etc) The most common operations are sending messages and opening sockets:extends Destructable {}
public MessageRequestHandleTo do so, you need the Identifier of the remote node, and the message to be delivered. Additionally, you may specify some transport-layer specific options such as Guaranteed/Unguaranteed/Encrypted etc. We will describe these options later. Finally you provide a callback to deliver notificaiton of success or failure when the operation completes. The call is non-blocking and returns a RequestHandle. The RequestHandle is like a receipt or a tracking number. You can also use the RequestHandle to cancel an existing request if it is no longer necessary. For example if the operaiton takes too long. The TransportLayerCallback provides the inverse operations and must have the identical Identifier/MessageType:sendMessage(Identifier i, MessageType m, MessageCallback deliverAckToMe, Map options); public SocketRequestHandle openSocket(Identifier i, SocketCallback deliverSocketToMe, Map options);
public interface TransportLayerCallbackThe P2PSocket is similar to the AppSocket.{ public void messageReceived(Identifier i, MessageType m, Map options) throws IOException; public void incomingSocket(P2PSocket s) throws IOException; }
Other calls in the TransportLayer:
This method returns the Identifier of the local node for this layer.public Identifier getLocalIdentifier();These methods can control flow by rejecting new messags/sockets if the local node is being overwhelmed.
public void acceptSockets(boolean b); public void acceptMessages(boolean b);This method sets the callback
public void setCallback(TransportLayerCallbackThis method sets the ErrorHandler which is usd for notification of unexpected behavior. Ex: The acceptor socket closes, or an unexpected message arrivescallback);
public void setErrorHandler(ErrorHandlerThis method cleans up the layer, for example closing down the AcceptorSocket.handler);
public void destroy();
Download the tutorial files: BandwidthLimitingTransportLayer.java NotEnoughBandwidthException.java DistTutorial.java MyApp.java, MyMsg.java into a directory called rice/tutorial/transportlayer/.
For this tutorial, we'll create a new tranport layer that caps the peak outgoing bandwidth. We will use a bucket sysetem with configurable bandwidth and bucket-length. For example if we want to limit bandwidth to 10K/second, we can allow 10K for any second, or 1K for 1/10th of second. For simplicity, we won't distinguish between socket and datagram traffic. The obvious place for this layer will be just above the Wire layer, but to provide maximum flexibility, we would like this layer to work with any Identifier. We will have to specify a message type that we can determine the size. In this case, we'll use the ByteBuffer as our Message type. To insert our new layer between 2 existing layers, we also need to be a TransportLayer callback of the same types so we can insert ourself between two existing layers. Here is the definition of our new class.public class BandwidthLimitingTransportLayerHere is the constructor that specifies the bucket size, and bucket time length. We also want an environment for later and the TransportLayer below us.implements TransportLayer , TransportLayerCallback { }
public BandwidthLimitingTransportLayer( TransportLayerYou can look at the code to see the declaration of these fields. The last thing we have to do is set ourself as the lower level's callback. This will cause it to deliver messages/sockets to us. This variable is the bucket.tl, long bucketSize, int bucketTimelimit, Environment env) { this.environment = env; this.tl = tl; BUCKET_SIZE = bucketSize; BUCKET_TIME_LIMIT = bucketTimelimit; logger = env.getLogManager().getLogger(BandwidthLimitingTransportLayer.class, null); tl.setCallback(this); }
/** * When this goes to zero, don't send messages */ protected long bucket;Now let's create a task to refil the bucket.
environment.getSelectorManager().getTimer().schedule(new TimerTask(){ @Override public void run() { // always synchronize on this before modifying the bucket synchronized(this) { bucket = BUCKET_SIZE; } } }, 0, BUCKET_TIME_LIMIT);If the TimerTask is unfamiliar, please review the timer tutorial. Now let's limit the outgoing message bandwidth. We are going to throw a
NotEnoughBandwidthException
if there isn't sufficient bandwidth. There are two things to show here. a) subtracting from the bucket or throwing the exception, b) setting up the message receipt properly so the task can be cancelled.
This shows the important code for our example. (We retun null for now.)
public MessageRequestHandleThe code we have so far is not allowed because we must return a proper MessageRequestHandle, as well as return the MessageRequestHandle when the message succeeds. The release already includes a generic implementation of the MessageRequestHandle: org.mpisws.p2p.transport.util.MessageRequestHandleImpl. Let's take a look at it: The constructor initializes these 3 fields:sendMessage(Identifier i, ByteBuffer m, MessageCallback deliverAckToMe, Map options) { boolean success = true; synchronized(this) { if (m.remaining() > bucket) { success = false; } else { bucket-=m.remaining(); } } if (!success) { deliverAckToMe.sendFailed(null, new NotEnoughBandwidthException(bucket, m.remaining())); return null; } tl.sendMessage(i,m,deliverAckToMe,options); }
Identifier identifier; MessageType msg; MapAnd there are also getters. However, we still need to be able tooptions;
cancel()
the operation in the next transport layer. This uses the 4th field:
Cancellable subCancellable; public boolean cancel() { return subCancellable.cancel(); }It is initialized with a call to
setSubCancellable()
Here is the code that now properly returns a MessageRequestHandle:
public MessageRequestHandleNote how we call returnMe.setSubCancellable with the call to the lower transportLayer. There is one more problem in the code. Because we simply pass through the deliverAckToMe field, whensendMessage(Identifier i, ByteBuffer m, MessageCallback deliverAckToMe, Map options) { MessageRequestHandleImpl returnMe = new MessageRequestHandleImpl (i, m, options); boolean success = true; synchronized(this) { if (m.remaining() > bucket) { success = false; } else { bucket-=m.remaining(); } } if (!success) { deliverAckToMe.sendFailed(returnMe, new NotEnoughBandwidthException(bucket, m.remaining())); return returnMe; } returnMe.setSubCancellable(tl.sendMessage(i,m,deliverAckToMe,options)); return returnMe; }
deliverAckToMe.ack()
or deliverAckToMe.sendFailed()
is called, it will have the wrong MessageRequestHandle. Thus, we need to create our won MessageRequestHandle which wraps deliverAckToMe
. The first thing we need to do is declare deliverAckToMe
and returnMe
final. Then we will create an anonymous inner class of the MessageCallback.
public MessageRequestHandleThe MessageCallback simply calls deliverAckToMe'ssendMessage(Identifier i, ByteBuffer m, final MessageCallback deliverAckToMe, Map options) { final MessageRequestHandleImpl returnMe = new MessageRequestHandleImpl (i, m, options); ... returnMe.setSubCancellable(tl.sendMessage(i,m,new MessageCallback () { public void ack(MessageRequestHandle msg) { deliverAckToMe.ack(returnMe); } public void sendFailed(MessageRequestHandle msg, IOException reason) { deliverAckToMe.sendFailed(returnMe, reason); } },options)); return returnMe; }
ack()/sendFailed()
methods with returnMe.
Here is the full code for the method:
public MessageRequestHandleThis may seem a bit overwhelming right now, but this code provides a lot of flexibility for the transport layer while still keeping the calls simple. The last major step is making the Sockets also respect the bandwith limitations. Rather than throwing an exception when we exceed the bandwidth, we just need to throttle the traffic, by only sending what we are allowed. To do this, we will create a P2PSocket that decrements the bucket on each write. Like the MessageRequestHandleImpl, we already have an implementation of a P2PSocket that wraps another. It is called the org.mpisws.p2p.transport.util.SocketWrapperSocket. The generic parameters are the 2 kinds of Identifiers that it Adapts. Since we are importing and exporting the same Identifier, we declare our Socket like this:sendMessage(Identifier i, ByteBuffer m, final MessageCallback deliverAckToMe, Map options) { final MessageRequestHandleImpl returnMe = new MessageRequestHandleImpl (i, m, options); boolean success = true; synchronized(this) { if (m.remaining() > bucket) { success = false; } else { bucket-=m.remaining(); } } if (!success) { deliverAckToMe.sendFailed(returnMe, new NotEnoughBandwidthException(bucket, m.remaining())); return returnMe; } returnMe.setSubCancellable(tl.sendMessage(i,m,new MessageCallback () { public void ack(MessageRequestHandle msg) { deliverAckToMe.ack(returnMe); } public void sendFailed(MessageRequestHandle msg, IOException reason) { deliverAckToMe.sendFailed(returnMe, reason); } },options)); return returnMe; }
class BandwidthLimitingSocket extends SocketWrapperSocketNow we need to set up the constructor. We will provide it the wrapped socket's identifier and options. Also, we must pass it our logger.{ }
public BandwidthLimitingSocket(P2PSocketNow we need to override these two methods:socket) { super(socket.getIdentifier(), socket, BandwidthLimitingTransportLayer.this.logger, socket.getOptions()); }
@Override public long write(ByteBuffer srcs) throws IOException {} @Override public long write(ByteBuffer[] srcs, int offset, int length) throws IOException {}Let's start with the first method. Here is the easy case:
if (srcs.remaining() <= bucket) { long ret = super.write(srcs); if (ret >= 0) { // EOF is usually -1 synchronized(this) { bucket -= ret; } } return ret; }The rest is complicated, because it is important to leave the incoming ByteBuffer's position in the proper place. We create a ByteBuffer temp with the amount of space left in the buffer:
// we're trying to write more than we can, we need to create a new ByteBuffer // we have to be careful about the bytebuffer calling into us to properly // set the position when we are done, let's record the original position int originalPosition = srcs.position(); ByteBuffer temp = ByteBuffer.wrap(srcs.array(), originalPosition, bucket);Now we try to write the data by calling super. This method may throw an IOException or return EOF. Fortunately, the srcs buffer is in its original position, so we don't need to do anything but return/throw exception:
// try to write our temp buffer long ret = super.write(temp); if (ret < 0) { // there was a problem, // reset the position, return return ret; }If we made it here, there were no problems, allocate the bandwidth, and update the position on the srcs ByteBuffer:
// allocate the bandwidth synchronized(this) { bucket -= ret; } // the lower layer couldn't write as much as we wanted to // we need to properly set the position srcs.position(originalPosition+(int)ret); return ret;The code for the other write method is similar, but setting the position is more complicated. The first part is similar, add up how much to send, and call super if we aren't overflowing.
// calculate how much they are trying to write: int toWrite = 0; for (int i = offset; i < offset+length; i++) { toWrite += srcs[i].remaining(); } int tempBucket = bucket; // so we don't get confused by synchronization if (toWrite <= tempBucket) { long ret = super.write(srcs, offset, length); if (ret >= 0) { // EOF is usually -1 synchronized(this) { bucket -= ret; } } return ret; }Note: We use the variable tempBucket so we don't get confused if the bucket is refilled asynchronously. The strategy now is to make a copy of the srcs array. This code finds the ByteBuffer that causes the overflow. We will replace that item with a temporary buffer as in the above example.
// we're trying to write more than we can, we need to create a new ByteBuffer // in the overflowing position, and set the length properly // we have to be careful about the bytebuffer calling into us to properly // set the position when we are done ByteBuffer[] temp = new ByteBuffer[srcs.length]; System.arraycopy(srcs, 0, temp, 0, srcs.length); int myLength = length; // we'll pass this to the call that we want int myIndex = 0; toWrite = 0; // reset this one for (int i = offset; i < offset+length; i++) { int next = srcs[i].remaining(); if (next+toWrite > tempBucket) { //we have the problem at this slot // set the myLength myLength = i-offset+1; myIndex = i; // replace it with a temporary byteBuffer srcs[i] = ByteBuffer.wrap(srcs[i].array(), srcs[i].position(), tempBucket-toWrite); break; } toWrite+=next; }This code looks like the previous example, but we may need to advance the position of the buffer we replaced.
// try to write our temp buffer long ret = super.write(temp, offset, myLength); if (ret < 0) { // there was a problem return ret; } // allocate the bandwidth synchronized(this) { bucket -= ret; } // we need to properly set the position on the buffer we replaced // the idea here is that we are advancing the srcs[i].position() with the // amount that was written in temp[i].position() srcs[myIndex].position(srcs[myIndex].position()+temp[myIndex].position()); return ret;The next step is to use our BandwidthLimitingSocket. There are two ways to get a socket. When the upper layer opens a socket, and when the lower layer accepts a socket.
openSocket():
We don't get the socket right away. We have to wait until it has completed opening. This is similar to a continuation. The first thing we have to do is create a SocketRequestHandle for the same reasons as we did in the above code with MessageRequestHandle.SocketRequestHandleImplNow, we ask the lower transport layer to open the socket, and then we will wrap it with our BandwidthLimitingSocket which we will return to deliverSocketToMe. If there is an Exception, we just pass it up to the previous layer.returnMe = new SocketRequestHandleImpl (i,options); returnMe.setSubCancellable(tl.openSocket(i, ... , options)); return returnMe;
tl.openSocket(i, new SocketCallback(){ public void receiveResult(SocketRequestHandle cancellable, P2PSocket sock) { deliverSocketToMe.receiveResult(returnMe, new BandwidthLimitingSocket(sock)); } public void receiveException(SocketRequestHandle s, IOException ex) { deliverSocketToMe.receiveException(returnMe, ex); } }, options)
incomingSocket():
First, we need a callback to deliver the socket to. This will be set later, but we will implement the setCallback() method in TransportLayer.TransportLayerCallbackNow it is simple to overridecallback; public void setCallback(TransportLayerCallback callback) { this.callback = callback; }
incomingSocket()
public void incomingSocket(P2PSocketThe last step is to add all of the rest of the methods in TransportLayer, transportLayerCallback. These are just going to forward the calls down or up as appropriate:s) throws IOException { callback.incomingSocket(new BandwidthLimitingSocket(s)); }
public void acceptMessages(boolean b) { tl.acceptMessages(b); } public void acceptSockets(boolean b) { tl.acceptSockets(b); } public Identifier getLocalIdentifier() { return tl.getLocalIdentifier(); } public void setErrorHandler(ErrorHandlerWhere should we put this layer? We will show two options. Because we are at the Java level, we already kon we can't fully account for the TCP overhead of the bandwidth (retransmission etc). However to get the maximum effect, we should place it just above Wire. Here, we show how to do this by extending SocketPastryNodeFactory. The SocketPastryNodeFactory in FreePastry version 2.1 is much more extensible than before. There is a get...TransportLayer() call for each layer that it constructs. To insert a different layer, simply override one of these calls and wrap the default layer with the new one. When the SocketPastryNodeFactory tries to construct the lowest layer, it calls getWireTransportLayer(). We will first construct the default layer by calling super.getWireTransportLayer(). However, we will return our Bandwidth-Limiting layer that wraps the wire layer.handler) { tl.setErrorHandler(handler); } public void destroy() { tl.destroy(); } public void messageReceived(Identifier i, ByteBuffer m, Map options) throws IOException { callback.messageReceived(i, m, options); }
public static PastryNodeFactory exampleA(int bindport, Environment env, NodeIdFactory nidFactory, final int amt, final int time) throws IOException { PastryNodeFactory factory = new SocketPastryNodeFactory(nidFactory, bindport, env) { @Override protected TransportLayerYou can replace the construction of the SocketPastryNodeFactory at the beginning of any of the existing tutorials with this code to add the "bandwidth-limiting" feature. Perhaps we don't want to include some of FreePastry's overhead in our bandwidth lmitation. If we put it above the SourceRouteManager, we don't include bandwidth for liveness checks, nor overhead from constructing source routes. However, we will have to forward some additional calls to do this:getWireTransportLayer(InetSocketAddress innermostAddress, TLPastryNode pn) throws IOException { // get the standard layer TransportLayer wtl = super.getWireTransportLayer(innermostAddress, pn); // wrap it with our layer return new BandwidthLimitingTransportLayer (wtl, amt, time, pn.getEnvironment()); } }; return factory; }