Elitism Improves Productivity

The Realeyes IDS application includes multiple plugins that interact with each other. The basic means of communication is a structure with information about the status of a network session, put on a queue by one plugin and taken off by the next one to process the session.

At the lowest level, this is a Data structure, which defines the packet captured by the Collector. The Data structure is then taken by the Stream Handler which determines which session it belongs to and sets some information, such as the start time, and then puts a Stream Analysis Work Element (SAWE) on another queue. The Stream Analyzers perform matching operations on the packets based on the rules defined for each one. Then the Action Analyzer and Event Analyzer perform correlation on the results of the Stream Analyzers.

This works very smoothly, except for the fact that there are multiple Stream Analyzers and one Action Analyzer. The Action Analyzer can free Data structures, and it must not free any that are still being processed. Because all of this analysis is happening asynchronously, the fields that indicate the state can change while being tested.

To handle this, I created a separate field that is set once when the session is ready for the Action Analyzer. Initially, I tried to wait briefly for the Stream Analyzers to update these fields. Of course, briefly is in the eye of the beholder. I set the wait value to 1 microsecond, which is 0.000001 second.

But the standard clock in most Intel computers is actually ticking once per 0.1 millisecond, or 0.0001 second. This is like saying, "Give me a second," and then taking over a minute and a half. The result was that work piled up waiting on the Action Analyzer. Buffers could not be freed and the application could not run for more than a couple of hours in the pilot environment.

I finally realized that instead of waiting for the first SAWE on the queue, the Action Analyzer should try to find one that was ready. In other words, it should ignore the structures that didn't meet its standards, and only choose that of the highest quality. In still other words, it should be an elitist.

And low and behold, buffer usage became almost a non-issue. The application now runs for days without running out of buffers. (In fact, it usually crashes from a bug before it runs out of buffers, but I'm working on fixing those.)

This demonstrates that being described as an elitist can be a compliment.

Later . . . Jim