Blitz Tuning Guide

Principle of Locality (POL)

Many JavaSpace applications follow the same 'principle of locality' (referred to as POL throughout the rest of this text) as their single-process, non-remote brethren. The POL essentially amounts to a rule of thumb - 'a program will access 10% of a program 90% of the time'. This same principle also applies to data access, though not quite as strongly. What does this mean for JavaSpace applications?

  1. Recently written entry's might well be taken soon.
  2. Recently written entry's might well be read soon.
  3. Certain entry's will be 'favourites' accessed many times for some reason or another.
  4. Some applications may be entirely random access. These are difficult to tune for and usually respond best to brute force methods such as more memory, larger caches, faster disks etc.

Clearly, not all entry's written to a space fall into any of the above categories. Those that do form the working set.

Taking Advantage

Microprocessors and operating systems exploit the POL to boost performance using caching. In any given situation, the perfect cache is one that:

  1. Can be indexed in a fashion well suited to the information it contains.
  2. Is big enough to contain the whole working wet.

So, for Blitz to perform at it's best, we'd like the entire working set of entry's to be held in cache. Clearly, in cases where the working set is large, we may not be able to fit it all in cache. When this happens, we rely on the cache replacement policy to make best use of the cache.

Blitz Architecture

Blitz stores segments Entry's by type storing them in separate repositories. Each respository contains two kinds of cache:

The Entry Cache is designed to accelerate repeated access to Entry's such as queue heads/tails or tokens. Write Buffer is designed to take advantage of the fact that many JavaSpace applications use flow-type models which results in many recently written Entry's being taken almost immediately afterward. It caches recently written entry's anticipating that they will be almost immediately taken/deleted. Everytime this happens, Write Buffer saves us disk accesses.

Blitz provides various storage models offering different tradeoffs between level of persistence and performance - see Appendix A for more details.

Basic Tuning

Tuning starts with the creation of a "work profile" for the application followed by adjustments in Blitz's configuration:
  1. Determine, via observation (using static analysis, program instrumentation and Blitz statistics), what the approximate working set of entry's will be for your application and configure the entry cache size to be at least that big, if possible. In cases, where it isn't possible, the cache replacement policy will handle the "swapping" that will occur. In many cases, a cache size which is 25% of the size of the working set gives reasonable performance. For high throughput systems (especially if take is commonly used), size the cache as size_of_working_set + (25% of size_of_working_set). The Write Queue should then be configured to at least 2 * entry_cache_size.
  2. Use the Berkeley DB cache statistics to size the Berkeley DB cache (Blitz can be configured to display these statistics at each checkpoint interval).
  3. Determine the persistence needs of your application and configure the appropriate storage model.

Other Recommendations

If at all possible, leave lease reaping turned off (lease reaping is the term we use for the process of actively trawling through all entries in a Blitz instance deleting those that have expired).

If your application requires significant RemoteEvent throughput (as the result of using notify) or needs to handle dispatching of a large number of blocking queries (as the result of a large number of clients waiting for entry's using read* or take* with non-zero timeouts), increase the number of task threads.

Optimizing Logging

Both persistent and time-barrier persistent storage models make use of a log to provide their persistence guarentees. In the case of persistent, once all the above steps have been taken, Blitz's performance will most likely be determined by the speed with which it can log sequentially to disk. This aspect of the persistent model should be tuned as follows:

  1. Determine the time required for a single operation to be forced to disk (unlike many other situations, any writes that Blitz puts in the log must be flushed immediately to disk in order to ensure the ACID properties) and set the batch write window size to that time plus a little more (say 5-10ms). This will slow down benchmarks which use a single thread to do, for example, a simple loop performing writes and take*s (if you wish to tune for these situations set the window to zero) but will provide a performance boost at any reasonable concurrent load.
  2. If you have a multi-channel disk controller with multiple disks, consider putting the Blitz logs on one disk and the persistent storage directory on another.
  3. Use some kind of disk array or buy faster disks. Once you've done all the other tweaking, this is the last and most expensive resort. You may need to add more processing power but remember that disk performance in most systems is considerably less than that of even the cheapest of desktop processors. The same can be said of memory so, when buying additional hardware for your system, ensure you've done some profiling to correctly determine which components need upgrading.
  4. Enable concurrent batching of log writes.
  5. Size the log buffer such that all writes within a batch are held in memory before being written to disk as one big chunk.
  6. Determine, via observation, the expected throughput of transactions within the application and set the checkpointing interval accordingly. Note that take*s, read*s or writes performed with a null transaction are treated as transactions containing a single operation. Adjusting this value changes the balance between speed of recovery versus overall throughput.

The time-barrier persistent storage model doesn't support concurrent batching (this is achieved naturally by the design) but usually requires a much larger log buffer than the default persistent storage model because it will tend towards writing larger numbers of log records in batches.

Appendix A - Storage Models

As of Blitz 2.0, it is possible to configure a number of different persistence profiles. They are currently:

  1. Persistent - the default setting. In this mode, Blitz behaves like a fully persistent JavaSpace such as the persistent version of Outrigger.
  2. Transient - causes Blitz to act like a disk-backed cache. In this mode, Blitz behaves like the transient version of Outrigger. No logging is performed and, when Blitz is restarted, all state (including Join state etc.) is lost. Memory-only transient implementations can halt with OutOfMemoryError's if they are over-filled. Blitz avoids this problem by swapping to disk when the number of stored Entry's overflows it's available cache space. Obviously, performance will degrade relative to the amount of swapping Blitz is required to perform. When the caches are sufficiently large, Blitz will make minimal use of disk, yielding maximal performance.
  3. TimeBarrierPersistent - provides a performance versus persistence QoS tradeoff. In this mode, changes made more than a certain number of milliseconds ago are guarenteed to be persistent. More recent changes are not guarenteed persistent but may be persistent. This mode provides the developer with a means of balancing persistence needs against performance.

The actual model used is determined by the value of the configuration variable storageModel. The standard configuration file contains example settings for all three modes which should provide reasonable starting points for more precise tuning. For more details on the exact settings for each model, including tuning options, see the Javadoc for org.dancres.blitz.config.Persistent, org.dancres.blitz.config.Transient and org.dancres.blitz.config.TimeBarrierPersistent.

Appendix B - Global Configuration Options related to Performance

Appendix C - Statistics Gathering Options for Performance Analysis

Back to Documentation