I gave a talk entitled “How to Index Massive Data Sets Quickly” at the Morrelly Homeland Security Center. The event was hosted at the Long Island Forum for Technology (LIFT) and was jointly supported by NYSTAR and the Stony Brook Sensor CAT. The audience was composed primarily of technologists in the area. While many of the people in the audience were not necessarily MySQL users, the problem of working with massive data sets is one that many MySQL users face.
One topic we discussed in detail was the data-ingest problem common to three letter agencies: sensors generate thousands or millions of data items per second. The sensors could be recording mouse-click data, taking temperature or chemical readings, or measuring network performance. The data must be stored so that analytical queries can be made on live data in real time.
From the talk, we know that people are interested in indexing and querying live data. Fractal Tree indexes, with insertions that run up to two orders of magnitude faster than B-tree insertions, are an optimal technology for solving the data-ingest problem on large data sets. (Here are more details about Fractal Tree algorithm performance.) Through this blog and ongoing discussions, we look forward to hearing more about these applications and welcome comments and feedback.
