Batch Import

The first thing you may want to do when getting started with the RAPID™ database is to import existing data from an external source. This data may be in flat files (for example CSV format), or in existing databases, like MySQL or MongoDB.

Best practices

The RAPID™ DAL API is transaction based, therefore is a trade-off between performance and ensuring that data has been committed to the database.

CSV Files

Measurement and log data is often kept in text files in comma separated or space/tab delimited formats. It usually looks like this:

Timestamp,Unit3- SENSOR1_ABC_AVG_PV,Unit4- SENSOR2_ABC_AVG_PV
2014-03-04 00:01:01.000, 1353.300049, 1353.199951
2014-03-04 00:01:10.859, 1353.199951,  1353.199951
2014-03-04 00:01:19.546, 1353.500000,  1353.000000

The order of values may be different (e.g. maybe the timestamp is last), but generally each line will have a timestamp and 1 or more measurement values. There might also be some sort of identifier string in the line, and the first line of the file may contain column headers.

To ease the import of data, Automsoft provides a tool called LoadEvents that can read a specially formatted CSV file and automatically store the data in the database. Since the RAPID™ database uses an item to represent each measurement source, the data needs to be formatted in a slightly different format. An example of this is provided below:

Unit3, SENSOR1_ABC_AVG_PV, float32
2014-03-04 00:01:01.000, 1353.300049, 0xC0
2014-03-04 00:01:10.859, 1353.199951, 0xC0
2014-03-04 00:01:19.546, 1353.500000, 0xC0
Unit4, SENSOR2_ABC_AVG_PV, float32
2014-03-04 00:01:30.390, 1353.199951, 0xC0
2014-03-04 00:01:30.390, 1353.199951, 0xC0
2014-03-04 00:01:39.797, 1353.000000, 0xC0

This example file contains the two items from the first CSV example, reformatted for import. Before each set of data, we define where in the database we will store the data. The first entry is the node (collection of items), the second is the item and the third is the type of the item. After that, each line corresponds to an event to store in the item.

Now that you've got the basic idea, check out the sample LoadEvents from gitHub. This uses the LoadEvents tool to import some sample data into the database. You can then read out the data, using either the RAPID™ Configuration tool or the TestData example. If you have any questions, drop us a mail to support@automsoft.com

Automsoft is a technology pioneer. In 2001, Automsoft launched the world’s fastest and highest performing data historian, completely re-thinking conventions on technology and the management of process data. Today, Automsoft’s RAPID™ database technology is at the forefront in complex asset management, and provides the data backbone for managing the massive volume of data generated by sensor networks, control systems and devices across industries from smart grid to medical sensing technology. Automsoft is headquartered in the Republic of Ireland.

Contact

EMEA

Automsoft
24 Lansdowne Road
Dublin 4
Ireland
Tel:  +353 1 449 1100
Fax:  +353 1 449 1199
E-mail:  info@automsoft.com

North America

Automsoft
5 Independence Way, Suite 300
Princeton
NJ 08540
USA
Tel:  +1 650 585 4338
E-mail:  info@automsoft.com