There has been an interesting thread in one of the Apama technical forum, regarding the notion of time in Apama, that can be more broadly applied to the world of CEP in general.
Timers and pauses are part of those matters that could possibly make your application racy and undeterministic. You don’t want your logic to be tied up to runtime systems differences and load. For example, you have two input piece of data that are 0.9 second apart when they arrive to your system input. First event arrives to your CEP engine, and as a consequence you set a timer of 1 second to collect all the data within a window of a second within the first event. Obviously you expect the timer to fire after the second is consumed. Right? Right! But what if after the first event a heavy calculation fires, and keeps your engine so busy to slow down consuming more events. Second event may or may not reach processing after one second. But should you worry about your application to give different outputs depending of your load?
Architectural considerations around time handling (but more in general around determinism) will give you a good representation on how sophisticated a CEP engine is. Best frameworks around will handle at core level such type of problems, as a second level OS, so that at application level you don’t need to take care of such issues.
Apama, the CEP engine I am most familiar with – unsurprisingly given the maturity of the platform – handles the time in a brilliant way. Time is effectively just another input event that the CEP engine sends to its input queue. In this way a timers cannot fire while you are processing information because events are processed sequentially together with all the other inputs. So to mention the above example, if you create a timer for 1 second, then you are effectively reacting to the time event that has been merged into the input stream after the second piece of data. In this way there is no possibility of a race, those two guys will be processed in diligent order.
This is just brilliant, and it has a number of advantages. You can disable sending such timing events and send it manually or take them from a recorded input flow (for example yesterday market prices), and just drive your CEP identically to when those events were received in first time. Time displayed in your clients will be artificially set to the past (identical to what those clients would have seen when those events were recorded in first place). Corollary is that you can now take liberties in terms of how the time passes (in the internal CEP representation) by just changing the speed you send your input. Nothing will be different than in your original real time execution (and here is where the determinism comes handy) because time is exactly another input of your application incidentally contained in a file or database, or whatever is your source.
You want a cup of tea, not to worry, just stop sending input data and time will freeze. The ability of going as slow as you wish is important when you are trying to understand what happened in relation of a certain situation like debugging. On the other side the ability of going as fast as possible is paramount when you are backtesting, to have a quick turnaround and understand how your new algo would have acted against hours, days or weeks of your recorded data (think infra night optimization of your algo parameter).
Unit testing is another area that highly benefit from determinism. As you can drive time, you can write unit tests that reflect perfectly the artificial situation you want to recreate.
This is exploited also in CEP farms. When you have a cluster of CEP engines you want them to have a time representation that is in sync between each other and not driven by the clock of each box. A way to achieve this is to have only one CEP (possibly the entry point of all the data) that produces time events and propagates the information to the others.
Not to mention the ability of decoupling your logic from your time source. I worked quite extensively in the Surveillance area at my friends of Turquoise and NYSE. If you want to detect an abuse that is, let’s say, caused by submitting two orders within a certain time, then you don’t want to mess around. The time you want to take in account needs to be identified by the time those orders arrived to the stock exchange (and audited in the order as a field), not time they arrived and processed in your downstream Surveillance system that could suffer of drift.
Oh men… I forgot that, do I need to rewrite my application?
Nope, just set the time using the field included in your data, so your CEP will think the time is exactly the time of when the event arrived in the exchange. No need of changing your application, everything related to timing will diligently adapt to your new definition of time.
