Xobas Software'> ]> ProLinga-Repository Project Release: &PRODUCT;-&version; - &release-date; Bas Driessen &xobas;
bas.driessen@prolinga.org
 Original author and current maintainer. 2002-©right-year; The ProLinga Team Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". This document describes the ProLinga-Repository Project, how to build the software components and documents the API reference of the prolinga-repository library. ProLinga-Repository Project The ProLinga-Repository Project forms the foundation of the ProLinga Development and Run-Time tools. A native XML database, built on Berkeley DB XML, that holds ProLinga application definitions as XML documents which can be stored and managed using an API and send over the network using the ProLinga-Soap client library. Storing and managing of data is controlled by Repository Commands (RC) using the concept of request and response. An RC request is an XML document containing a command as a read or a write action. This request is presented to the repository engine for processing. After processing, the result will be placed in an RC response document and returned to the caller. ProLinga-Repository contains a binary that can run as a Web Service (daemon) on the machine. The sophisticated prolinga-soap SOAP layer on top controls the flow and processing of incoming and outgoing requests/responses over the TCP/IP network. Using this concept, programs can run locally on a machine while accessing a remote repository centrally located in an intranet environment or anywhere in the world via the internet. ProLinga-Repository is the perfect base for platform independent multi-developer environments. Since all code is written in C++, it can be ported to all modern UNIX systems, GNU/Linux variants as well as MS-Windows and Mac O/S platforms. ProLinga-Repository is extremely fast and very reliable. It can hold up to 256 Terabytes of information with a maximum of 4 Gigabyte for a single XML document. XPath 1.0 queries can be executed for addressing parts of an XML document. Non-XML information can be stored as well, as long as it is has an XML envelope. Main purpose for the creation of this project, is to provide an easy way to store XML application documents into a database. Although the current implementation is optimized for use in the ProLinga Environment, it can be used in other Non-ProLinga related projects as well. Some minor index changes may be needed. ProLinga-Repository has been built and tested on various popular GNU/Linux platforms as RedHat Fedora, Mandriva Linux, Debian, openSUSE and Ubuntu. Since all code is written in C++, it can be ported to all modern UNIX systems, GNU/Linux variants as well as MS-Windows and Mac O/S platforms. XML Documents All data in the repository is stored as XML documents. These documents are placed in a container in the database that represents one or a collection of applications. Only 3 index fields are required for an XML document in the ProLinga environment: Application - The name of the application. Type - The type of the object. (Examples: Variable or Constant) Name - The name of the object. The combination of these 3 index fields has to be unique. Note that these three index fields are based on requirements by the ProLinga Developer and Run-Time Tools. Different indexes can be defined, but that requires some program changes. Future releases will be more flexible. The XML documents have the following format: appntypename <]]>type .... .... <]]>/type ]]> Example: Standard ]]> In the example above, the "Type" is both an attribute as an element. Using it this way has the advantage that a validation mechanism as XML schemas is much easier to implement. The type of object is not a pre-defined value. New object types can be introduced all the time. No code changes are required for this, but if validation with XML Schemas is in place to check for valid object types, this XML schema need some modifications to allow for the new types. In the current release there is no internal XML Schema validation implemented yet, but future releases will have this. Repository Commands Repository Commands are used to retrieve data from, write data to or delete data from the repository. A request XML document containing a command is presented to the repository. This document is processed and a response XML document is returned. Repository Commands have the following format: .... .... ]]> The following Repository Commands are available: List With the list command a list of object names can be retrieved, Wildcards can be used for the object name. This command is typically used whereby the user is given a selection of choices or for batch processing. Example List Request: ]]> Example List Response: ]]> List on Application or Type is not implemented as such. To achieve this you can introduce an object type "Application" and "Type" for instance. Get With the list command an XML document can be retrieved from the repository. Example Get Request: ]]> Example Get Response: Standard ]]> Wildcards can be used for "Name" to retrieve a series of XML documents with one single RC. Put With the put command an XML document can be written to the repository. The whole document needs to be presented, as no partial updates are possible. Example Put Request: Standard ]]> Example Put Response: ]]> Delete With the list command an XML document can be removed from the repository. Example Delete Request: ]]> Example Delete Response: ]]> SysInfo The SysInfo command can be used to retrieve system and enviroment details from the server where ProLinga-Repository is running. Example SysInfo Request: ]]> Example SysInfo Response: Live Server Stock Admininstration M1 8001 /home/prolinga/prolinga/system 3 /tmp/prolingarep.log 5 6 M1 ]]> Ping The ping command can be used to test if a repository host is on-line and responding to requests. Example Ping Request: ]]> Example Ping Response: ]]> Additionial error information from the underlying SOAP layer will be presented as well in the response document in case of an error. Error Handling As soon as something goes wrong when processing a request, an error is generated and put into the response document. The status of the response will be set to "Error" or "Warning" depending on the severity of the error. In case of an error, processing will abort immediately, while in case of a warning processing continues. An error has the following format: Error/Warning Short description An external error text if available ]]> Example response when trying to retieve a non-existent document: Warning Document not found ]]> The "External" element will contain any error text passed on from the O/S or 3rd party products. The following initial errors are defined: Errors Error Id Severity Description 10000 Ok No Error 10001 Error Error parsing document 10002 Error Empty Document 10003 Error Invalid root element 10004 Error Invalid repository XML file 10005 Warning No document match 10006 Warning Document not found 10007 Warning Document already deleted 10008 Error Can not parse return string
Multiple Repository Commands It is possible to combine RCs in one single request. There will be one response file, but with results for every command, unless an error has been encountered before the end of processing. The format is as follows: ... ... ... ... ]]> Combining multiple RCs will result in better performance. ProLinga-Repository Programming Notes To test the ProLinga-Repository, the best and easiest way is to use the binaries that come with the product. Configure so the Repository is running as a Web Service and use the client binary to send and receive Repository Commands. See the user guide section for more details. To access the Repository from a C++ application, either network (SOAP) calls to send and receive Repository Commands over the network or the C++ API to link with the Repository library can be used. Network Client To be able to send and receive Repository commands over the network as SOAP calls from a C++ application, the application must link with the ProLinga-Soap Client Library (libprolingasoapclient). Details of how to do so are in the section "Programming Notes" of the ProLinga-Soap Project documentation. In the example you have to change the example XML request document with a valid Repository Command. Alternatively have a look at the source code file RepositoryClient.cpp in the src directory of ProLinga-Repository. When using this method, ProLinga-Repository must be running as a Web Service. Repository Library To be able to link directly into the Repository from a C++ application, the Repository Library (C++ API) can be used. A very basic example program using the Repository Library, so it can access the Repository looks like this: int main() { /*Init */ xmlDocPtr docReq, docRes; xmlNodePtr curReq, curRes; xmlBufferPtr bufRes; PlRepository rep; /* Open Repository Environment */ rep.repositoryOpen("./prolingarepcfg.xml"); /* Create Repository Command */ docReq = xmlNewDoc((const xmlChar *)"1.0"); docReq->children = xmlNewDocNode(docReq, NULL, (const xmlChar *)"ProLinga", NULL); curReq = xmlDocGetRootElement(docReq); curReq = xmlNewTextChild (curReq, NULL, (const xmlChar *)"Repository", (const xmlChar *)""); xmlNewProp (curReq, (const xmlChar *)"Version", (const xmlChar *)"1.0"); curReq = xmlNewTextChild (curReq, NULL, (const xmlChar *)"Command", (const xmlChar *)""); xmlNewProp (curReq, (const xmlChar *)"Name", (const xmlChar *)"SysInfo"); xmlNewProp (curReq, (const xmlChar *)"Mode", (const xmlChar *)"Request"); /* Process Repository Command */ docRes = rep.executeCommand(docReq); /* Print Response */ bufRes = xmlBufferCreate(); curRes = xmlDocGetRootElement(docRes); xmlNodeDump(bufRes, docRes, curRes, 0, 1); printf("%s\n", (char *)xmlBufferContent(bufRes)); /* Close Repository Environment */ rep.repositoryClose(); /* Free */ xmlBufferFree(bufRes); xmlFreeDoc(docRes); xmlFreeDoc(docReq); /* Return */ return 0; } ]]> When running this application, a connection is made to the Repository. When opening this connection an argument with the path and name of the repository configuration file can be passed as an argument. The system then knows details as in which directory it can find the Repository files etc. The configuration file has to be in a certain format, which is explained further on. The Repository Command is then presented to the Repository process engine and a response is returned. This response is printed to to stdout (screen). The connection to Repository is closed, so data get flushed etc. and the program exits. After creating a connection using the openRepository class function, more than one RC can be executed without the need to close and open the Repository for every call. In fact performance wise this is the prefered method. The C API of the libxml2 toolkit has been used in this example, while it is probably better to use a C++ implementation of this toolkit as libxml++. Internal Repository code will be changed as the project progresses. To compile the program, use the following command: g++ repository_test.cpp -o repository_test `pkg-config --cflags --libs prolinga-repository` You may need to set the environment variable PKG_CONFIG_PATH to point to the directories where prolinga-repository.pc is located. This can be directory /usr/lib/pkgconfig or directory /usr/local/lib/pkgconfig on GNU/Linux for example. ProLinga-Repository is built using various 3rd party open source packages. These packages need to be available at compile time for a sucessful result. See the dependencies and prerequisites section for more details. Download ProLinga-Repository There are two types of product versions available for download: Production Version Development Version Production Version The production version is an official product release. This is a stable completed version which should be used in production environments. At this stage the product is only available as a source archive. Binary packages for a wide range of platforms will be available for download as the project progresses. Please visit the download page for the latest details. Development Version The development version is a snapshot of the latest source files. Since it can contain bugs as part of "work in progress", this version should not be used in production environments. Only developers who want to help with further development of the ProLinga Projects should download this version. Please read the details at the Subversion section of how to obtain the latest development version. Build ProLinga-Repository Prerequisites ProLinga-Repository &version; has been built and tested on Fedora Core, Mandriva Linux, Debian, Ubuntu, openSUSE. The product should compile and run successfully on other GNU/Linux distributions as well as UNIX. GNU gcc/g++ >= 3.3 compilers/tools are required for a successful build. Required 3rd party package(s) for a successful build: Berkeley DBXML. This product provides a native XML database. libxml2. This product provides XML libraries. ProLinga-Soap This product provides a SOAP/XML engine. Optional 3rd party package(s): xsltproc. Command line processor. This package is only required to re-generate the HTML pages after making changes to the documentation. General Download and Installation Instructions 3rd Party Products. Assuming the directory /opt/builds will be used for 3rd party products, then in general the following steps need to be taken to download and extract a 3rd party product source archive. Download source tar/gz archive to /opt/builds/[product] Example: place libxml2-x.x.x.tar.gz in /opt/builds/libxml2 Use 'gunzip' to unzip the archive Use 'tar xvf [product.tar] to extract the archive. Example: tar xvf libxml2-x.x.x.tar. A new sub directory will be created with the new version of the product. Example: /opt/builds/libxml2/libxml2-x.x.x This way several versions of the same products can be installed next to each other. For product specific installation instructions see following sections. Berkeley DBXML: To be able to build Berkeley DBXML, several other products need to be installed as well. These are: Berkeley DB. This product provides a general purpose database. Xerces. This product provides XML libraries. XQilla. XQilla is an implementation of XQuery 1.0 and XPath 2.0 using Xerces DOM. Berkeley DB XML requires a version of XQilla that is not publicly available and is distributed with Berkeley DB XML. It is recommended to first download all products before installing them because of dependencies.
Berkeley DB: Product root dir: /opt/builds/db Current version: 4.8.26 Download: http://download.oracle.com/berkeley-db/db-4.8.26.tar.gz Build Instructions: cd /opt/builds/db/db-4.8.26 ../dist/configure --enable-cxx make make install The --enable-cxx option is required. Xerces-c: Product root dir: /opt/builds/xerces Current version: 2.7.0 Download: http://www.apache.org/dist/xml/xerces-c/source/xerces-c-src_2_7_0.tar.gz Build Instructions: cd /opt/builds/xerces/xerces-c-src_2_7_0 export XERCESCROOT=`pwd` cd src/xercesc ./runConfigure -P $XERCESCROOT -plinux -cgcc -xg++ -minmem -nsocket -tnative -rnone [-b64] make make install XQilla: Product root dir: /opt/builds/xqilla Current version: 2.2.3 (customized release shipped with dbxml 2.5.16) Download: Ships with Berkeley DBXML 2.5.16. Build Instructions: cd /opt/builds/xqilla/xqilla-2.2.3 mkdir build cd build ../configure --with-xerces=/opt/builds/xerces/xerces-c-src_2_7_0 make make install
Product root dir: /opt/builds/dbxml Current version: 2.5.16 Download: http://download.oracle.com/berkeley-db/dbxml-2.5.16.tar.gz Build Instructions: cd /opt/builds/dbxml/dbxml-2.5.16/build_unix ../dist/configure make make install See file docs/ref/xml_unix/intro.html which ships with dbxml for more installation information. Check for and apply latest dbxml patches. Libxml2: Product root dir: /opt/builds/libxml2 Current version: 2.6.31 Download: ftp://xmlsoft.org/libxml2/libxml2-2.6.31.tar.gz Build Instructions: cd /opt/builds/libxml2/libxml2-2.6.31 ./configure make make install ProLinga-Soap: Product root dir: /opt/builds/prolinga Current version: 0.0.1 Download: http://download.prolinga.org Build Instructions: cd /opt/builds/prolinga/prolinga-soap-0.0.1 ./configure --prefix=/opt/builds/prolinga/prolinga-soap-0.0.1 make make install See the ProLinga-Soap Project documentation for more installation information. Build Instructions Extracting archive The product archive can be extracted to any desired directory using unzip (.zip file extension) or gunzip (.gz file extension)/ bunzip2 (.bz2 file extension) and tar. After extracting, the following directories will be created: Directory Structure. Directory Description &PRODUCT;-&version; Product version root directory. &PRODUCT;-&version;/config Config build information. &PRODUCT;-&version;/doc Documentation. &PRODUCT;-&version;/src All source and internal header files. &PRODUCT;-&version;/tests Self tests. &PRODUCT;-&version;/prolinga External (API) header files.
Compiling and Linking To build the product go to the product root directory and run the configure script. For default installation in /usr/local type: ./configure To install into another directory type: ./configure --prefix=/any/dirname/ For all other configure options, type: ./configure --help After running the configure script, the product can be build with: make To run the optional self-test type: make check After compilation, the binaries, libraries and header files can be installed with: make install You may need root access for this last option. By default, HTML documentation pages are available in the doc/html directory. These pages are generated from DocBook XML file format files in /doc. To re-generate the HTML pages from these files type: make html The command line XSLT processor xsltproc must be available from $PATH to be able to generate the HTML documentation. Dependencies Many of the ProLinga-Projects are dependent on 3rd party libraries. These (non system) dependencies of ProLinga-Repository are outlined below. If these 3rd party products are not installed in either /usr/bin, /usr/lib, /usr/include or /usr/local/bin, /usr/local/lib, /usr/local/include, the additional configure option needs to be provided when building. Dependencies Dependency Version Configure Option Berkeley DB >=4.8.26 --with-berkeleydb-prefix=/path/to/berkeleydb Berkeley DBXML >=2.5.16 --with-berkeleydbxml-prefix=/path/to/berkeleydbxml XQilla >=2.2.3 --with-xqilla-prefix=/path/to/xqilla Xerces >=2.7.0 --with-xerces-prefix=/path/to/xerces libxml2 >=2.6.5 --with-libxml-prefix=/path/to/libxml2 ProLinga-Soap >=0.0.1 --with-prolingasoap-prefix=/path/to/prolingasoap
Choosing the right type of build The default configuration options provide libraries which can be used both to develop/debug as to run the product. However better builds are possible for a dedicated development or production environment. In production environments, builds are needed which contain minimal (debug) overhead, so they are fast and small. To build such binaries/libraries, the --enable-final options can be used. Example: ./configure --enable-final The enable-final flag is configured to be used with GCC environments only. If access to more compilers become available over time, the enable-flag will be ported to those compilers as well. In development environments, builds are needed producing warnings, enabling maximum debugging info etc. Several options are available here. --enable-warnings : Set all compiler warning flags --enable-debug : Enable all debug messages --enable-gprof : Enables profiling with gprof (GCC only) Further development of ProLinga-Repository All ProLinga Projects are under constant active development. This includes issues related to fixing bugs, increasing performance etc, as well as expanding the various projects with new features. This chapter provides information of all these types of tasks as well as details of how to join the development team. To-do List This section contains a list of work that needs to be done on the product. A more detailed and up to date list is available on the ProLinga Bugzilla Server. Implement XML Schemas for validation. Libxml2 is finishing up XML schemas. If it is still too far away, start with DTD. ProLinga-Repository is written in C++. Libxml2 however is C only. There is a C++ wrapper for this XML parser library. Details at http://libxmlplusplus.sourceforge.net/. Libxml2 is used in ProLinga-Repository as the Xml Parser. Libxml2 is a high level parser used in the Gnome projects. DbXml (xml database) however is using Xerces. There may be performance improvements possible if Libxml2 is replaced by Xerces. XML documents returned from the database for instance can be passed on directly, rather then being parsed again by Libxml2. Some strings/paths are hardcoded in the product. This all should be reviewed and be removed or made all configurable using a config file. Utilities. prolingarepcreate at the moment can only be used to initialize the Repository. We should have options available as creating application dump, reading application dump etc. Code review and cleanup. Logging. A simple logging mechanism has been implemented. However, it appears that DbXml has a logging system of its own. This should be investigated and implemented (if useful). Self-tests need to be developed for quality control. Porting to other platforms. (MS-Windows/Solaris?), so we can make a distribution available on a specific port if we need to. Locking. At the moment, records are not being locked when updated. This mechanism is available, as well as technologies as Transactions and rollback. Need to be looked into. As enhancements, we should look into making ProLinga-Repository a more useful product by itself. This means not the hardcoded indexes Application, Type and Name as we have now. It is all we need for the ProLinga Development tools, but there maybe more benefits in this product by itself. At the moment it is "just" an Application Repository. Subversion A Subversion tree containing the latest project sources is available at http://svn.prolinga.org. For access please contact the ProLinga Development team. Bugzilla Bugzilla software development and bug tracking system to manage the ProLinga projects is available at http://bugzilla.prolinga.org. For access please contact the ProLinga Development team. Help to Develop Are you ready to join the ProLinga development team? There are several ways you can help: Use the product. Tell us what you think, ideas, bugs etc. Fixing bugs. Visit the development section of the ProLinga Web Site, choose a bug, fix it and send us the code changes. Enhancements. Have a look at the enhancement section of the to-do list and let us know which new feature you want to help to develop. Porting. To be successful it is essential that the ProLinga Projects are available on a wide range of platforms. We need help in porting and testing as well as providing binary releases for many different platforms. Let us know if and how you can help out. Real world examples. Are you using ProLinga products or do you know people who are happily using it. Tell us the story and provide some screenshots. ProLinga-Repository C++ API This section contains the API reference for ProLinga-Repository. All the public interfaces are documented here. &prolinga-repository-API; ProLinga-Repository User Guide This section contains the user guide for the ProLinga-Repository. All the binaries are documented here. &prolinga-repository-userguide; &fdl;