Introduction to DB-All.e version 2.5 13/09/2006
DB-All.e is a fast on-disk database where meteorological observed and forecast data can be stored, searched, retrieved and updated.
This framework allows to manage large amounts of data using its simple Application Program Interface, and provides tools to visualise, import and export in the standard formats BUFR, AOF and CREX.
Main characteristics of DB-ALL.e
- Free Software licensed under the terms of the GNU General Public
- License.
- Fortran, C, C++ and Python APIs are provided.
- To make computation easier, data is stored as physical
quantities,
- that is, as measures of a variable in a specific point of space and time, rather than as a sequence of encoded reports.
- Internal representation is similar to BUFR and CREX WMO standard
- (table code driven) and utilities for import and export are included (generic and ECMWF template).
- Representation is in 7 dimensions: latitude and longitude
geographic
- coordinates, table driven vertical coordinate, reference time, table driven observation and forecast specification, table driven data type.
- It allows to store extra information linked to the data, such as
- confidence intervals for quality control.
- It allows to store extra information linked to the stations.
- Variables can be represented as real, integer and characters,
with
- appropriate precision for the type of measured value.
- It is based on physical principles, that is, the data it contains
are
- defined in terms of homogeneous and consistent physical data. For example, it is impossible for two incompatible values to exist in the same point in space and time.
- It can manage fixed stations such as synops and moving stations
such
- as airplanes or ships.
- It can manage both observational and forecast data.
- It can manage data along all three dimensions in space, such as
data
- from soundings and airplanes.
- Report information is preserved. It can work based on physical
- parameters or on report types.
- The Fortran API supports data in real, double precision, integer,
- byte and character types
- Optimized for read-write access
- Developed using an ODBC programming layer over a MySQL relational
- database.
- Includes an interactive graphical navigation tool to visualize,
- search, modify and delete the database contents.
It is a powerful and flexible alternative to Netcdf for punctual data.
Introduction to the APIs
datum: any data can be expresses as a floating point (single or double precision), integer, character or byte, keeping in mind that some conversions are not possible; a datum is univocally described by a table similar to the B tables of the WMO BUFR and CREX formats.
attribute: an attribute is a datum that is referred to another datum; a datum can have several attribute; a confidence interval is normally an attribute of a datum.
coordinates: every datum is univocally defined by a system of coordinates; the dimensions to define are 7:
- 'network': an homogeneous group of stations with the same kind of
- measures and management
- 'x' and 'y': the geographical coordinates
- 'z': the vertical coordinate in one of its various aspects; it is
- defined univocally by a table code and two numerical values
- 'datetime': date and time of validity of the datum, be it
observed,
- forecast or estimated
- 'timerange': the time period to which the datum refers, as it can
- be, for example, a cumulated or averaged value over an interval; it is defined by a table code and two numerical values
- 'variable': defines the physical parameter and is identified by a
- numeric code according to the B table
A particular instance of datum is the station information, which is used to convey special information about a station. It is the equivalent of any other datum, but the available coordinates have been reduced to 4, collapsing z, datetime and timerange to a special value. The network of a station information is also usually set to a special network which only reports station information, so that station information can be referred to just x, y and variable. Station information can have attributes.
It is thus possible, according to the above definition, to have infinite station information and infinite data with infinite attributes, of course within the limits of hardware and software performance.
Fortran API
The routines are divided in three categories:
- setting input parameters
- reading output parameters
- action routines
Action routines perform operations such as queries, insertions and deletions; input routines set the parameters to the action routines, and output routines read the results of the action routines.
The API uses two main parameter storage areas:
- input: defines all the input parameters used by the action routines
- output: defines all the parameters output by the action routines
The input generally represents what to write or what to look for in a query; the output generally represents the results of a read or some useful parameter that can be read after an action routine and can be used later, such as shortcuts to data just written.
To write data it is necessary to define all the 7 coordinates that identify the data; in the case of station information, the required coordinates are just 3.
To query data, every parameter is optional and for some of them (such as latitude, longitude and datetime) it is possible to specify minimum and maximum values. Every parameter that is set produces a restriction of the resulting dataset, and the results of the following query are those values that fit inside the restriction.
Input, output
It is possible to use variables that are integer, strings, real, double precision and byte, as is appropriate according to the datum that is manipulated.
These routines set the input parameters:
idba_seti(handle, "param", intvalue)
idba_setc(handle, "param", charvalue)
idba_setr(handle, "param", realvalue)
idba_setd(handle, "param", doublevalue)
idba_setb(handle, "param", bytevalue)
These routines read the output parameters:
idba_enqi(handle, "param", intvalue)
idba_enqc(handle, "param", charvalue)
idba_enqr(handle, "param", realvalue)
idba_enqd(handle, "param", doublevalue)
idba_enqb(handle, "param", bytevalue)
These routines unset input parameters:
idba_unset (integer handle, param)
idba_unsetall (handle)
Connections, sessions, handles
The first thing needed to work with DB-All.e is to create a connection to a DSN (Data Source Name) ODBC that corresponds to a MySQL database. Every connection defines a group of data with which it is possible to work. It is possible, although rarely required, to have open connections to many different databases at the same time.
It is then necessary to create one or more working sessions, each of which has a different input and output storage area and allows to perform multiple operations on the database at the same time.
The action routines to open and close a connection are:
call idba_presentati(dbhandle, "dsn", "username", "password")
......
call idba_arrivederci(dbhandle)
The action routines to open and close a session are:
call idba_preparati(dbhandle, handle, "read", "read", "read")
......
call idba_fatto(handle)
Writing data
There are three groups of data:
- Station identification (latitude, longitude, identifier if mobile)
- Data (including station information)
- Attributes
To write any data you need to refer to the station identification. To write attributes you need to refer to existing data.
You can write station identification and data at the same time using the action routine idba_prendilo; attributes are read and written separately with the action routine idba_critica.
Data and station information cannot be written at the same time, and the distinction is made using the input routine idba_setcontextana.
The station identification can also be univocally identified by an integer number (ana_id); it is thus possible to write the data separately from the station identification providing the ana_id value as a shortcut to it.
Data (including station information) is identified by a number (context_id) and the variable code, which need to be specified when writing attributes using the input values *context_id and *var_related.
Reading and writing station identification information
Station identification data is written together with other data by setting all required input parameters and calling:
call idba_prendilo(handle)
Station identification data and station informations are read together using the action routines:
integer idba_quantesono (integer handle, integer count)
integer idba_elencamele (integer handle)
Reading and writing data (including station information)
These data can normally be written together with station identification information. If they are written separately, they need to be referred to an ana_id value previously obtained when writing station identification information or reading from a idba_elencamele.
If the idba_setcontextana input routine is called, then the data will be written as station information, else as normal data.
It is possible to write many variables (all having the same coordinates) with the same action routines: after setting all the input data, call:
call idba_prendilo(handle)
Data is read using the action routines:
idba_voglioquesto (integer handle, integer count)
idba_dammelo (integer handle, character parameter)
Reading and writing attributes
Attributes can normally be written after writing the data (including station information) they refer to.
If they are written separately, it is necessary to provide values for *context_id and *var_related to define univocally the datum to which they refer.
*context_id can be read as context_id after a previous idba_dammelo or idba_prendilo.
*var_related describes which variable, among the ones present in the same space-time coordinates, will have the attributes attached.
It is possible to insert many attributes, all attached to the same data, with one single invocation of the action routine: after setting all the input data, call:
call idba_critica(handle)
To read the attributes, call:
call idba_voglioancora(handle, countancora)
call idba_ancora(handle, param)
Deleting data
The action routines to deleting data read their input like the routines that read data: replacing query routines with deletion routines causes the data selected by their input to be deleted instead of read.
When a datum is deleted, all attached attributes are deleted as well.
To delete data, use:
idba_dimenticami (integer handle)
To delete attributes, use:
idba_scusa (integer handle)
To completely empty and reset the database, use:
idba_scopa (integer handle, character repinfofile)
References
Please refer to the Fortran API manual for detailed specifications: