What is D-Orb

Dimsim Buffer(D-Orb) provides a mechanism for buffering real-time data produced by CIMA sources. The buffer ensures that CIMA is fault-tolerant and provides sinks the ability to recover from data errors caused by corrupt or lost transmission, power failure, software reboot and likes.

Who else uses Ring Buffers

Ring Buffers are useful data structures for management and distribution of real time data.

Object Ring Buffers are used extensively by the seismic community¹ for monitoring and visualization of seismic events. Agencies like NASA and Military use ring buffers to help increase productivity in real time decision making and simulation using archived data².  There are even patent claims on Ring Buffer based network bus management system .

Ring Buffer usage varies from project to project. But most implementations use a combination of both hardware (dedicated RAMS, server machines) and software tools.

1. Antelope Real-time Systems uses ORBs for data acquisition, transport, buffering, processing, archiving and distribution information.

2.  DataTurbine Project uses ring buffer based network bus to provide a networked file system for cached streaming data

What functions an ORB can provide

Data Acquisition   : -  act as a temporary sink to a source that produce real-time data. The ORB can then be queried by clients(permanent sinks) for specific subsets of data spanning time and source(s). This concept can be extended to include many to one or many to many relationships between ORB, sources and clients.

Data Transport     : -  provide a reliable mechanism for transferring large files to clients. For example, clients to a source can include limited bandwidth and low memory devices like mobile phones which cannot handle/afford to deal with large files. ORB frees the sources from decision making and takes the responsibility of deciding whether or not to transfer files and how to package them.

Data Buffering  :-  In addition to providing a temporary cache of recent data from sources, ORBs can also help data processing by acting as a temporary buffer to hold data until it is ready for processing.

Data Processing :-  ORBs could make use of callback hooks and code patterns like visitor  to provide data processing functions. An example in the crystallography area would be the generation of JPEG images from raw OSC files. Rather than having a separate plug-in for generating JPEG images, the visitor pattern can be used to auto-generate and insert a jpeg file every time a new OSC file is added to the buffer.

Data Archiving :-  ORBs can be linked to persistent data stores (which could be (or have been) an ORB client as well) to provide seamless access to historical data. An example would be a portal application transitioning from real-time to historical data for playback. Rather than depending on different sources for input, the application can rely on the ORB to provide both real-time and historical data.

Manage Distributed Data Sources :- A Single ORB can act as an entry point for buffered data from distributed data sources. An example would be an ORB buffering real-time data from multiple reef-sensors. On the other end, a single orb can be composed of multiple parts ranging from in-memory buffer to dedicated persistent data stores.

DIMSIM Orb Functional Requirements

Hardware   : Currently there are no specific requirements for hardware components

Software     : The buffer will be implemented as a java package providing the following functions

1. Data Acquisition Methods : Ability to register with sources running on local machine and receive data of arbitrary type and size. This data will be inserted into ring buffer packaged with source name and time-stamp.

• Implementation Note : Arbitrary type is restricted to the lower levels of Data Store. At higher levels, the interfaces are specific to IRegisteredParcel Objects, which are the main objects of interest for CIMA.
  

2. Data Access Methods :

Query : Clients must be able to query ring buffer the following :

• Number of objects available for a given source and a start time-stamp
  
• Number of objects available for a given source and time-range
  
• Number of objects available for a given source and object data type
• A directory like listing of buffer contents ; listing producer id, parcel types, access restrictions  and creation time.
  

Fault-tolerance : ORB must provide clients the option of

• requesting a guaranteed delivery of objects for a given source and time range.
  
• use case : Client wants to playback data for the first ten minutes of the hour. Client first queries for number of objects, then sends a request for guaranteed delivery  of these objects. ORB marks those objects as locked (i.e., cannot be deleted from buffer) until it is delivered to the client. 

Implementation Note: Guaranteed delivery is now part of CIMA transport layer. To ensure objects are locked until delivery, the buffer generates a copy (soft link in the form of pointers to objects, thereby delaying garbage collection in the event they are removed from buffer before delivery)

3. Data Processing Interfaces :   

• Allow objects to register with the Ring buffer for notification events. In particular for add new and delete events. To prevent race conditions and delays, the notify process must be non-blocking .  This leads to the  problem of identifying  the object for which notification event is being sent. 
• Implementation Note : Spring Event notification methods are used to implement the above requirement. Suggestions made during the course of Dimsim development work has resulted in Spring Developers extending Spring Event notification in the next release. For more information  see SPR-4690 
  
• Allow registered objects to walk-thorugh the Ring buffer collection and carry out processing. This can be implemented by visitor pattern.
• Implementation Note : Dimsim Buffer methods  provide  guaranteed access to restricted cloned views of buffered data based on input  producer type, creation time and object type. In addition, Event Notification methods for addition and removal of Buffer objects provide access to producer and the buffer object in focus. Together, the above functions remove the need for Buffer to allow direct access to its content for processing. Such an approach frees the buffer to focus on buffering and responding to consumer requests for objects.
  

4. Data Transport

• Not clear what the requirements will be and if they cannot be done by other functions. 
  
• For example, changing image size for different displays can be carried out by plug-ins registered with ORB for notification events.
• Implementation Note : CIMA transport layer deals with transmission of data. Buffer implementation is restricted to responding to remote consumer requests for buffered data and ensuring that CIMA transport methods are invoked to respond to data requests.
  

5. Data Buffer

• A Ring buffer, by default should continue to erase the buffer in a circular fashion, but what happens if the current object slated for deletion is being processed for transmission or conversion ?. ORB needs a temporary cache for holding objects slated for deletion, but not available for delete yet.
• Implementation Note : Providing cloned versions of data objects buffered resolves this problem.
  

6. Multiple ORB implementations

• The underlying data store can be in-memory, file/folder based or database based. It is recommended that (if time permits) all three implementations be made available and allow the user to configure which one(s) they would like for their project.

•  Implementation Note : Two implementations are available Memory Based and database based using Hibernate Object Relational Mapping. Memory Based has been tested and works fine. ORB based implementation suffers from problems associated with fetching of database Session object when Spring and Hibernate are used together. More details can be found in this blog
  

7. Security :

• Data requests to Buffer must be authenticated to ensure consumers access to buffer is restricted to producers they have access to.
• Implementation Note : This is taken care by allowing static and dynamic configuration of allowed security tuples <consumer-id, consumer-url, producer-id>. More information is available in the Dimsim Security document.
  

Implementation

Source Code and Documents related to Dimsim Buffer implementation is available for viewing at Dimsim Trac.

References

1. Harvey, D., D. Quinlan, F. Vernon, and R. Hansen (1998),    “ORB: A New Real-Time Data Exchange and Seismic Processing System”, ,” Seis. Res. Lett. 69, 165.

2. RBNB White Paper 11/27/01 at http://outlet.creare.com/rbnb/ .