Modern databases and text analytics

A few database management systems (DBMS) are in use: Microsoft’s SQL (Structured Query Language), Linux’s MySQL, Apache Open Office Base, IBM DB2, MarianDB, Oracle and Oracle Express, PostgreSQL, and SQLite. Note that Microsoft Access is kind of a “poor man’s” database system, in the sense that it is very simplistic, with the entire database being placed in a single file. Microsoft Excel is an electronic spreadsheet, not a database system, since all users share a common view of the data and in the form in which the data is physically stored.

Although RDBMS are still the most predominant database systems today, at least for the average database proprietor, many of the new “high tech” data that we want to process are much more complicated and cannot be organized into any meaningful rows-and-columns configurations. Thus, in addition to the rather structured data for which RDBMS and the models mentioned above were intended, today’s target data for processing also contain unstructured types such as texts in contracts, manuals, newspapers, tweets, metadata, e-mail contents, books, webpages, photos, videos, and what have you. These kinds of information cannot and should not be disassembled into rows and columns so that sentences and phrases are kept intact for meaningful context analysis. The government may want to analyze the contents of billions of e-mails for hints on possible terrorist links.

The next-generation database systems suitable for storage and retrieving unstructured data are collectively referred to as NoSQL, for “Not only SQL.” Examples include Amazon’s SimpleDB, 10 gen’s MongoDB, Google’s Apache HBase, and Facebook’s Apache Cassandra. These databases are also ideally suited for massive data (cluster application). Note that HBase is an open source, distributed, column-oriented database system based on Google’s BigTable.  (Big Table is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Big Table.)

A University of Toronto benchmark conducted last year found Cassandra to be better than HBase. End Point Corporation, a database company, benchmarked the top NoSQL databases –Cassandra, HBase, and MongoDB. Again, Cassandra led in all the metrics, followed by HBase. Developers of Cassandra attribute its superior performance to the “fundamental architectural choices.”

The other high tech big-boost to modern surveillance operation is the ability to analyze texts (text analytics) – as in analyzing the texts in e-mail contents, tweets, webpages, etc., for the purpose of extracting relevant information, which could then be transformed into simple bits of information that can go into 2D tables, for example. The texts represent unstructured data.

The techniques for analyzing textual contents have their origin from the fields of Natural Language Processing (NLP), data mining, applied mathematics, and statistics. Note that text analytics is not a search operation where you merely look for the existence or otherwise of words in a text. In text analytics, you are trying to discover information. Obviously, the two operations can be combined for added capabilities.

In a recent 2013 book: “Big Data for Dummies,” by Judith Hurwitz, Alan Nugent, Fern Halper, and Marcia Kaufman, in the “For Dummies” series, the authors discussed text analytics, a piece that I find to be easy to read. They identify four levels of NLP analysis: In lexical/morphological analysis, the characteristics of an individual word is examined, to obtain bits of information that will help you understand what the word means within the context in which the word is used. The use of an electronic dictionary is an inherent part of the analysis. In syntactic analysis, grammatical structures are used to “dissect” the text and put individual words in the proper context. In semantic analysis, the possible meanings of a sentence are determined. According to Hurwitz, “this can include examining word order and sentence structure and disambiguating words by relating the syntax found in the phrases, sentences, and paragraphs.” In the discourse-level analysis, attempts are made to determine the meaning of a text beyond the sentence level.

Note that a set of rules might need to be established to aid in the extraction of information from various document sources. An example of a rule, a trivial one at such, is that the name of a person must start with an upper-case letter. Very elaborate rules might be necessary to significantly enhance the analysis. Hurwitz and her co-authors also describe how the extracted information could be understood. This is too lengthy to present in this article, and you can consult the reference. However, the end result of your unstructured data analysis is a structured data, which can now be combined with other structured data that you may have.

The long and short of this article is that unstructured data in e-mails, tweets, and so on, can only be stored using modern database systems (NoSQL), and processed using computer-based procedures such as Natural Language Processing (NLP).  The end result will significantly enhance a government’s ability to “catch a thief,” if not a terrorist!