[Sigia-l] Massive data

[Listera]

For the last few weeks, I've been designing a new platform prototype for a
key financial institution that should change how (non-trading) financial
information is displayed and consumed.

There's one interaction flow I keep coming back to that would be an
unprecedented boon to user productivity, but, unfortunately, depends on
real-time parsing of massive quantities of data/info. Every which way I
maneuver around the notion, I'm stymied by the fact that RDBMs (disk or RAM
based) just don't have the throughput to get me the data I need to massage
it into an otherwise interesting widget design. So I haven't been able to
include this uber-widget into my grand strategy.

Then comes Streambase:

"In a recent pilot program, Streambase was able to analyze 140,000 messages
per second, while a leading relational database -- Stonebraker won't say
which one -- could handle only 900 messages per second. Streambase has 12
customers now testing its software, all of them financial services companies
that need to analyze rapid-fire ticker feeds and other streaming data.
Unlike traditional database programs, Streambase analyzes data without
storing it to disk, performing queries on data as it flows. Traditional
systems bog down because they first store data on hard drives or in main
memory and then query it, Stonebraker says."

<http://www.forbes.com/technology/2005/01/18/cz_dl_0118streambase.html>

That's more than a 100X throughput increase, "on a $1,500 PC." Sounds
incredible. Needless to say, the ability to process massive data streams on
the fly enables many different kinds of applications:

<http://www.infoworld.com/article/05/01/07/HNstreambasestone_1.html>

Question: Anyone has seen a demo of this beast?

Ziya
Nullius in Verba