Xobas Software'> ]> ProLinga-Data Project Release: &PRODUCT;-&version; - &release-date; Bas Driessen &xobas;
bas.driessen@xobas.com
 Original author and current maintainer. 2002-©right-year; The ProLinga Team Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". This document describes the ProLinga-Data Project, how to build the software components and documents the API reference of the prolinga-data library. ProLinga-Data Project ProLinga-Data is a project that uses the latest libgda technologies provide access to the data providers PostgreSQL, MySQL, Oracle, Sybase, ODBC data sources, mSQL, IBM DB2, FireBird/Interbase, FreeTDS, SQLite, LDAP, MS SQL Server, Berkeley DB, MDB (MS Access) and xBase (dBase, Clipper, FoxPro) files. Since libgda is under active development itself, we have to introduce this technology in seperate steps. The first (and implemented) step is that SQL queries can be passed on to a provider using its native language dialect (SQL92 or SQL99 for instance). ProLinga-Data is in this instance "just" the traffic agent of passing on a SQL statement and returning a result or data model (cursor). The next step is to introduce an easy way to modify schemas allowing the management of database objects (tables, views, users, procedures, etc) in the provider environment, without using any SQL at all. This feature will be available in upcoming releases of libgda and will then get implemented in ProLinga-Data. This feature gives us the advantage for instance to build and re-build data files from within the ProLinga Environment, without having to know if the data provider is PostgreSQL or MySQL for instance. The ability to create tables and indexes is available in the mean-time and the rebuild ability is not too far away. Fully functional updatable data models are available for the PostgreSQL and MySQL providers. Future enhancement steps will include: XML Queries. Queries in a generic form that are put into an XML document are send to a provider where this query is transformed into a, for the data provider, understandable (SQL) query and where results are returned to the caller. Reports ProLinga-Data works according the Request-Response mechanism. A request can be for instance to execute a "SELECT * ..." in a data provider environment. ProLinga-Data will send out this request to the data provider and returns a result or data model (cursor) as a response. ProLinga-Data can be configured as a Web Service (daemon) on the machine. A sophisticated SOAP layer on top controls the flow and processing of incoming and outgoing requests/responses over the TCP/IP network. Using this concept, programs can run locally on a machine while accessing a remote database engine centrally located in an intranet environment or anywhere in the world via the internet. Main purpose for the creation of this project, is to provide an easy way to access and store user data from the ProLinga environment. Since the result library of this project is very open, simple and easy to implement, it can be used in other non-ProLinga related projects. ProLinga-Data has been built and tested on RedHat Fedora, Mandriva Linux, Debian, Ubuntu and openSUSE. Since all code is written in C++, it can be ported to all modern UNIX systems, Linux variants as well as Windows and Mac O/S platforms. Data Commands Data Commands are used to access a data provider. A request XML document containing a command is presented to the provider. This document is processed and a response XML document is returned. Data Commands have the following format: . . . . . . ]]> The following Data Commands are available: Connect Disconnect ListProviders ListDataSources OpenDataSource CloseDataSource ExecuteQuery ExecuteNonQuery ExecuteQueryDataModel ExecuteQueryTableDataModel DataModelGetRow DataModelAppendRow DataModelUpdateRow DataModelRemoveRow DataModelClose Transaction ManageData MetaStore Ping Connect With the connect command, a session is created. A valid session is required to be able to send Data Commands. Hhandles, cursor positions etc will all be held internally in a session record. Example Connect Request: ]]> Example Connect Response: ]]> Disconnect With the disconnect command, a session is destroyed. Example Disconnect Request: ]]> Example Disconnect Response: ]]> ListProviders Administration/debug command. With the ListProviders command, providers for which libgda has been configured for, will be shown. This does not necessarily mean that the providers themselves are installed and are available. Example ListProviders Request: ]]> Example ListProviders Response: ]]> ListDataSources Administration/debug command. With the ListDataSources, datasources for which libgda has been configured, will be shown. Example ListDataSources Request: ]]> Example ListDataSources Response: MySQL ProLinga MySQL. PORT=3306;USER=prolinga;DATABASE=stock;HOST=localhost PostgreSQL Stock database in PostgreSQL HOSTADDR=127.0.0.1;PORT=5432;USER=prolinga;DATABASE=stock;HOST=localhost admin ]]> OpenDataSource With the OpenDataSource command, a connection to a data source is established. Example OpenDataSource Request: prolinga False False True ]]> Example OpenDataSource Response: ]]> CloseDataSource With the CloseDataSource command, a connection is destroyed. This command is not required when followed by a disconnect command, since a disconnect command will close all data sources linked to that session. Example CloseDataSource Request: ]]> Example CloseDataSource Response: ]]> ExecuteNonQuery With the ExecuteNonQuery command, SQL queries as a DELETE or UPDATE can be executed that do not return any data. Only the number of rows affected will be returned. Example ExecuteNonQuery Request: INSERT INTO Parts VALUES ('12345','Nail U12','2.95','54') ]]> Example ExecuteNonQuery Response: 1 ]]> ExecuteQuery With the ExecuteQuery command, SQL queries can be executed that return data. All data is returned in a single XML response file. For large responses, ExecuteQueryDataModel can be used instead to "just" load the datamodel (cursor), which can be processed using DataModel commands. Example ExecuteQuery Request: SELECT * FROM Parts ]]> Example ExecuteQuery Response: 12345 Nail U12 2.95 54 12346 Nail U15 2.90 14 ]]> ExecuteQueryDataModel With the ExecuteQueryDataModel command, SQL queries can be executed that return data. All data is loaded in a DataModel. DataModel commands can then be used to retrieve this data. Example ExecuteQueryDataModel Request: SELECT * FROM Parts ]]> Example ExecuteQueryDataModel Response: 5 12 ]]> ExecuteQueryTableDataModel With the ExecuteQueryTableDataModel command, a table can be loaded in the data model for update, so records can easily be changed, appended or removed resulting in the underlying data provider to be updated. Example ExecuteQueryTableDataModel Request: Parts
where price > '10.00' ]]> Example ExecuteQueryTableDataModel Response: 5 12 ]]> DataModelGetRow With the DataModelGetRow command, a row can be retrieved from a DataModel. Example DataModelGetRow Request: ]]> Example DataModelGetRow Response: 12346 Nail U15 2.90 14 ]]> DataModelAppendRow With the DataModelAppendRow command, a row can be appended to the DataModel generated with the ExecuteQueryTableDataModel command. At time of writing, this functionality has not been implemented yet with all data providers supported by the libgda project. Please check the provider specific notes or mailing list (archive) for more info. Example DataModelAppendRow Request: 12346 Nail U15 2.90 14 ]]> Example DataModelAppendRow Response: ]]> DataModelUpdateRow With the DataModelUpdateRow command, a row can be updated in the DataModel generated with the ExecuteQueryTableDataModel command. This results in the underlying database row's values being changed. At time of writing, this functionality has not been implemented yet with all data providers supported by the libgda project. Please check their provider specific notes or mailing list (archive) for more info. Example DataModelUpdateRow Request: 12346 Nail U15 2.95 14 ]]> Example DataModelUpdateRow Response: ]]> DataModelRemoveRow With the DataModelRemoveRow command, a row can be removed from a DataModel generated with the ExecuteQueryTableDataModel command. This results in the underlying database row being removed from the database. At time of writing, this functionality has not been implemented yet with all data providers supported by the libgda project. Please check their provider specific notes or mailing list (archive) for more info. Example DataModelRemoveRow Request: ]]> Example DataModelRemoveRow Response: ]]> DataModelClose With the DataModelClose command, one or all DataModels can be closed and resources will be freed. Example DataModelClose Request: ]]> Example DataModelClose Response: ]]> Transaction With the Transaction command, update SQL commands can be safeley executed within a transaction environment. Transactions will only work if the underlying tables are transaction safe tables. With MySQL for instance Transactions will only work when using the type InnoDB or BDB. The default (My)ISAM type does not have a transaction engine. Example Transaction Request: * Serializable ** *) Available Modes are Begin, Commit and Rollback **) Available IsolationLevels are Unknown, ReadCommitted, ReadUncommitted, RepeatableRead and Serializable. ]]> Example Transaction Response: ]]> ManageData With the ManageData command the following actions can be performed: Create Drop Reformat (to do) Dump to text file (to do) Load from text file (to do) ... Example ManageData Request: Parts ]]> Example ManageData Response: ]]> MetaStore With the MetaStore command the following actions can be performed: Update ... (to do) Example MetaStore Request: Parts * *) If ExternalTableName is left empty then all tables will be updated. This process can take a while depending on the number of tables involved. ]]> Example MetaStore Response: ]]> Ping The ping command can be used to test if a data host is on-line and responding to requests. Example Ping Request: ]]> Example Ping Response: ]]> Additionial error information from the underlying SOAP layer will be presented as well in the response document in case of an error. Error Handling As soon as something goes wrong when processing a request, an error is generated and put into the response document. The status of the response will be set to "Error" or "Warning" depending on the severity of the error. In case of an error, processing will abort immediately, while in case of a warning processing continues. An error has the following format: Error/Warning Short description An external error text if available ]]> Example response when trying to connect to a non-existent data source. Error Can not open GDA connection. ]]> The "External" element will contain any error text passed on from the O/S or 3rd party products. The following errors are defined: Errors Error Id Severity Description 70000 Ok No Error 70001 Error Can not open GDA connection 70002 Error GDA connection does not exist 70003 Error Error processing query 70004 Error Data Model does not exist 70005 Error Invalid value RowId 70006 Error Other transaction already in progress 70007 Error No transaction in progress 70008 Warning UNDEFINED 70009 Warning Invalid Session ID 70010 Error UNDEFINED 70011 Error Error parsing file 70012 Error UNDEFINED 70013 Error Invalid Build Option 70014 Warning GDA connection already open
Multiple Data Commands It is possible to combine DCs in one single request. There will be one response file, but with results for every command, unless an error has been encountered before the end of processing. The format is as follows: ... ... ... ... ]]> Combining multiple DCs will result in better performance. Storage Types The the following storage types are defined in libgda: null bigint biguint binary blob boolean date double geometric_point (point) gobject integer list money numeric single smallint smalluint string time timestamp tinyint tinyuint type uinteger unknown Not all storage types are implemented (yet) and/or supported in all the various providers libgda can link with. Within the ProLinga 4GL environment, only the basic data storage types are defined: Character Integer Long (Integer) Single (Precision Number) Double (Precision Number) Date ProLinga 4GL -> libgda mapping ProLinga Storage Type LibGDA Storage Type Character string Integer integer Long Integer bigint Single Precision Number single Double Precision Number double Date date
libgda -> ProLinga 4GL mapping LibGDA Storage Type ProLinga Storage Type null Character bigint Long Integer biguint (not implemented yet in libgda) -- binary Character blob Character boolean Character date Date double Double Precision Number geometric_point (point) N/A gobject (not implemented yet in libgda) -- list N/A money (not implemented yet in libgda) -- numeric Double Precision Number single Single Precision Number smallint Integer smalluint (not implemented yet in libgda) -- string Character time Character timestamp Character tinyint Integer tinyuint (not implemented yet in libgda) -- type (not implemented yet in libgda) -- uinteger (not implemented yet in libgda) -- unknown (not implemented yet in libgda) --
For libgda <-> data provider mappings, please look at provider specific documentation at http://www.gnome-db.org. ProLinga-Data Programming Notes To test ProLinga-Data, the best and easiest way is to use the binaries that come with the product. Configure so the product is running as a Web Service and use the client binary to send and receive Data Commands. See the user guide section for more details. To access the Data engine from a C++ application, either network (SOAP) calls to send and receive Data Commands over the network or the C++ API to link with the Data library can be used. Network Client To be able to send and receive Data commands over the network as SOAP calls from a C++ application, the application must link with the ProLinga-Soap Client Library (libprolingasoapclient). Details of how to do so are in the section "Programming Notes" of the ProLinga-Soap Project documentation. In the example you have to change the example XML request document with a valid Data Command. Alternatively, have a look at the source code file DataClient.cpp in the src directory of ProLinga-Data. When using this method, ProLinga-Data must be running as a Web Service. Data Library To be able to link directly with Data from a C++ application, the Data Library (C++ API) can be used. A very basic example program using the Data Library looks like this: int main(int argc, char **argv) { /*Init */ xmlDocPtr docReq, docRes; xmlNodePtr curReq, curRes; xmlBufferPtr bufRes; PlData dat; /* Open Data Environment */ dat.dataOpen("./prolingadatcfg.xml", argc, argv); /* Create Data Command */ docReq = xmlNewDoc((const xmlChar *)"1.0"); docReq->children = xmlNewDocNode(docReq, NULL, (const xmlChar *)"ProLinga", NULL); curReq = xmlDocGetRootElement(docReq); curReq = xmlNewTextChild (curReq, NULL, (const xmlChar *)"Data", (const xmlChar *)""); xmlNewProp (curReq, (const xmlChar *)"Version", (const xmlChar *)"1.0"); curReq = xmlNewTextChild (curReq, NULL, (const xmlChar *)"Command", (const xmlChar *)""); xmlNewProp (curReq, (const xmlChar *)"Name", (const xmlChar *)"Connect"); xmlNewProp (curReq, (const xmlChar *)"Mode", (const xmlChar *)"Request"); /* Process Data Command */ docRes = dat.executeCommand(docReq); /* Print Response */ bufRes = xmlBufferCreate(); curRes = xmlDocGetRootElement(docRes); xmlNodeDump(bufRes, docRes, curRes, 0, 1); printf("%s\n", (char *)xmlBufferContent(bufRes)); /* Close Data Environment */ dat.dataClose(); /* Free */ xmlBufferFree(bufRes); xmlFreeDoc(docRes); xmlFreeDoc(docReq); /* Return */ return 0; } ]]> When running this application, a connection is made to the Data environment. When opening this connection an argument with the path and name of the data configuration file can be passed as an argument. The system then knows environment details. The configuration file has to be in a certain format, which is explained further on. The Data Command is then presented to the Data process engine and a response is returned. This response is printed to to stdout (screen). The connection to Data is closed and the program exits. After creating a connection using the openData class function, more than one VC can be executed without the need to close and open the Data environment for every call. In fact performance wise this is the prefered method. The C API of the libxml2 toolkit has been used in this example, while it is probably better to use a C++ implementation of this toolkit as libxml++. Internal Data code will be changed as the project progresses. To compile the program use the following command: g++ data_test.cpp -o data_test `pkg-config --cflags --libs prolinga-data` ProLinga-Data Library libprolingarepository has to be available for this option as well as the libgda libraries. You may need to set the environment variable PKG_CONFIG_PATH to point to the directories where prolinga-data.pc is located. This can be directory /usr/lib/pkgconfig or directory /usr/local/lib/pkgconfig on Linux for example. ProLinga-Data is built using various 3rd party open source packages. These packages need to be available at compile time for a sucessful result. See the dependencies and prerequisites section for more details. Download ProLinga-Data There are two types of product versions available for download: Production Version Development Version Production Version The production version is an official product release. This is a stable completed version which should be used in production environments. At this stage the product is only available as a source archive. Binary packages for a wide range of platforms will be available for download as the project progresses. Please visit the download page for the latest details. Development Version The development version is a snapshot of the latest source files. Since it can contain bugs as part of "work in progress", this version should not be used in production environments. Only developers who want to help with further development of the ProLinga Projects should download this version. Please read the details at the Subversion section of how to obtain the latest development version. Build ProLinga-Data Prerequisites ProLinga-Data &version; has been built and tested on various popular GNU/Linux platforms as RedHat Fedora, Mandriva Linux, Debian, Ubuntu and openSUSE. The product should compile and run successfully on other Linux distributions as well as UNIX. GNU gcc/g++ >= 3.3 compilers/tools are required for a successful build. Required 3rd party package(s) for a successful build: libxml2. This product provides XML libraries. libgda This product provides a data abstraction layer. ProLinga-Soap This product provides a SOAP/XML engine. Optional 3rd party package(s) for a successful run-time/test environment: Data providers like PostgreSQL or MySQL xsltproc. Command line processor. This package is only required to re-generate the HTML pages after making changes to the documentation. General Download and Installation Instructions 3rd Party Products. Assuming the directory /opt/builds will be used for 3rd party products, then in general the following steps need to be taken to download and extract a 3rd party product source archive. Download source tar/gz archive to /opt/builds/[product] Example: place libxml2-x.x.x.tar.gz in /opt/builds/libxml2 Use 'gunzip' to unzip the archive Use 'tar xvf [product.tar] to extract the archive. Example: tar xvf libxml2-x.x.x.tar. A new sub directory will be created with the new version of the product. Example: /opt/builds/libxml2/libxml2-x.x.x This way several versions of the same products can be installed next to each other. For product specific installation instructions see following sections. Libxml2: Product root dir: /opt/builds/libxml2 Current version: 2.7.3 Download: ftp://xmlsoft.org/libxml2/libxml2-2.7.3.tar.gz Build Instructions: cd /opt/builds/libxml2/libxml2-2.7.3 ./configure --prefix=/opt/builds/libxml2/libxml2-2.7.3 --disable-gtk-doc make make install LibGDA: Product root dir: /opt/builds/libgda Current version: 4.1.4 Download: ftp://ftp.gnome.org/pub/GNOME/sources/libgda/4.1/libgda-4.1.4.tar.gz Build Instructions: cd /opt/builds/libgda/libgda-4.1.4 ./configure --prefix=/opt/builds/libgda-4.1.4 make make install ProLinga-Soap: Product root dir: /opt/builds/prolinga Current version: 0.0.2 Download: http://download.prolinga.org Build Instructions: cd /opt/builds/prolinga/prolinga-soap-0.0.2 ./configure --prefix=/opt/builds/prolinga/prolinga-soap-0.0.2 make make install See the ProLinga-Soap Project documentation for more installation information. Build Instructions Extracting archive The product archive can be extracted to any desired directory using unzip (.zip file extension) or gunzip (.gz file extension)/ bunzip2 (.bz2 file extension) and tar. After extracting, the following directories will be created: Directory Structure. Directory Description &PRODUCT;-&version; Product version root directory. &PRODUCT;-&version;/config Config build information. &PRODUCT;-&version;/doc Documentation. &PRODUCT;-&version;/src All source and internal header files. &PRODUCT;-&version;/tests Self tests. &PRODUCT;-&version;/prolinga External (API) header files.
Compiling and Linking To build the product go to the product root directory and run the configure script. For default installation in /usr/local type: ./configure To install into another directory type: ./configure --prefix=/any/dirname/ For all other configure options, type: ./configure --help After running the configure script, the product can be build with: make To run the optional self-test type: make check After compilation, the binaries, libraries and header files can be installed with: make install You may need root access for this last option. By default, HTML documentation pages are available in the doc/html directory. These pages are generated from DocBook XML file format files in /doc. To re-generate the HTML pages from these files type: make html The command line XSLT processor xsltproc must be available from $PATH to be able to generate the HTML documentation. Dependencies Many of the ProLinga Projects are dependent on 3rd party libraries. These (non system) dependencies of ProLinga-Data are outlined below. If these 3rd party products are not installed in either /usr/bin, /usr/lib, /usr/include or /usr/local/bin, /usr/local/lib, /usr/local/include, the additional configure option needs to be provided when building. Dependencies Dependency Version Configure Option libxml2 >=2.6.5 --with-libxml-prefix=/path/to/libxml2 libgda >=4.1.4 --with-libgda-prefix=/path/to/libgda ProLinga Soap >=0.0.2 --with-prolingasoap-prefix=/path/to/prolingasoap
Choosing the right type of build The default configuration options provide libraries which can be used both to develop/debug as to run the product. However better builds are possible for a dedicated development or production environment. In production environments, builds are needed which contain minimal (debug) overhead, so they are fast and small. To build such binaries/libraries, the --enable-final options can be used. Example: ./configure --enable-final The enable-final flag is configured to be used with GCC environments only. If access to more compilers become available over time, the enable-flag will be ported to those compilers as well. In development environments, builds are needed producing warnings, enabling maximum debugging info etc. Several options are available here. --enable-warnings : Set all compiler warning flags --enable-debug : Enable all debug messages --enable-gprof : Enables profiling with gprof (GCC only) Further development of ProLinga-Data All ProLinga Projects are under constant active development. This includes issues related to fixing bugs, increasing performance etc, as well as expanding the various projects with new features. This chapter provides information of all these types of tasks as well as details of how to join the development team. To-do List This section contains a list of work that needs to be done on the product. A more detailed and up to date list is available on the ProLinga Bugzilla Server. Implement XML Schemas for validation. Libxml2 is finishing up XML schemas. If it is still too far away, start with DTD. ProLinga-Data is written in C++. Libxml2 however is C only. There is a C++ wrapper for this XML parser library. Details at http://libxmlplusplus.sourceforge.net/. Some strings/paths are hardcoded in the product. This all should be reviewed and be removed or made all configurable using a config file. Code review and cleanup. Make it more C++ Object Oriented. Logging. A simple logging mechanism has been implemented. Self-tests need to be developed for quality control. Porting to other platforms. (Windows/Solaris?), so we can make a distribution available on a specific port if we need to. Look into XML Queries (XQL). Locking??? Look into SQL 92 and SQL 99. Need a independent flavour of SQL if user want to in combination with XML Queries. Will always need to support native SQL for data provider. Add command line options to the binaries. Subversion A Subversion tree containing the latest project sources is available at http://svn.prolinga.org. For access please contact the ProLinga Development team. Bugzilla Bugzilla software development and bug tracking system to manage the ProLinga projects is available at http://bugzilla.prolinga.org. For access please contact the ProLinga Development team. Help to Develop Are you ready to join the ProLinga development team? There are several ways you can help: Use the product. Tell us what you think, ideas, bugs etc. Fixing bugs. Visit the development section on the ProLinga Web Site, choose a bug, fix it and send us the code changes. Enhancements. Have a look at the enhancement section of the to-do list and let us know which new feature you want to help to develop. Porting. To be successful it is essential that the ProLinga Projects are available on a wide range of platforms. We need help in porting and testing as well as providing binary releases for many different platforms. Let us know if and how you can help out. Real world examples. Are you using ProLinga products or do you know people who are happily using it. Tell us the story and provide some screenshots. ProLinga-Data C++ API This section contains the API reference for ProLinga-Data. All the public interfaces are documented here. &prolinga-data-API; ProLinga-Data User Guide This section contains the user guide for the ProLinga-Data. All the binaries are documented here. &prolinga-data-userguide; &fdl;