What is VDSF?

The virtual Data Space Framework (vdsf) is a tool to simplify the development of software projects. It provides an alternative to complex multi-threaded applications by simplifying the process and by making it easier to add additional elements later on.

The way it works: you don't need to design mechanisms for threads to exchange data and synchronize accesses to it. What you should do instead is to design your tasks and the data required for communication by the different tasks. Easier to explain using an example (not a very realistic one - just to demonstrate how the whole thing works).

A stock exchange - a list of some of the tasks:

• Receive buy/sell orders (and of course, send a receipt for these orders to the customers once validated).
• Process these orders. A transaction can either be concluded (selling price is equal or lower than the asking price) or the orders remain pending.
• If an order is completed, save the data to a backend database (which will enable other parts not covered here like billing).
• Send a confirmation of the transaction to both the seller and the buyer when their orders are completed.
• Update the summary of activities for the day, for each stock: current highest bid price, current lowest ask price, volume for the day, etc.

Based on this, our preliminary list of data containers will be:

• An input queue to store the incoming requests.
• One or more ordered lists to store pending orders (ordered by price). One possibility would be to have two such lists for each stock, one for selling requests, one for buying requests. Note: ordered lists are scheduled for version 0.4.
• An output queue for confirmation messages.
• An output queue for the database.
• A hash map for our summary of activities.

Note that pending requests could be stored in the database instead. The current design is based on the premise that data in the VDS can be recovered if a crash occurs. Furthermore, RDBMS real strength (IMHO) is in their ability to run complex queries but this is not the case here - all we are doing is trying to match the prices of buying and selling requests.

The programs that we need to write are:

• An input program. It will read requests from the communication channel, validate them, store them in the input queue and send an acknowledgement to the client.
• A processor program. It will read from the input queue and try to find matching order. If yes, it inserts records in the confirmation queue and in the database queue. It will also update the summary map as needed.
• A confirmation program. To send confirmations of the transaction to both seller and buyer.
• A program to insert transaction records into the database.

Once this basic structure is in place, it is easy to see how additional programs and data structures can be written to beef up the system:

• Additional input programs could be written if there are multiple sources of input requests.
• One could add a monitoring program, for example to warn the administrators if the system is under stress (one could even imagine a monitoring program lauching more versions of the processor program if only the input queue is getting quite large and if the computer itself is not under stressed (multi-core system)).
• Other type of monitoring could easily be added as well, sending alerts if a stock is undergoing major changes (up or down or if the volume of sales is abnormally high or...).
• Configuration options for these programs can be put in a hash map, allowing the behaviour of some of these programs to be modified in real time (the frequency at which the monitor programs rescan the data for changes, for example).

In conclusion, using VDSF allows a more modular approach to solve complex situations. Each program/module will be simpler, less error prone, and the programming team will be able to concentrate on the actual logic needed to satisfy the requirements of the system instead of putting a lot of effort on unrelated tasks (managing concurrency, for example).

Last updated on May 22, 2008.