Indigo DQM is an advanced data management, query, processing and reporting system designed to maximise data assets, information and intelligence. This powerful application can be used for data analyses, data extraction, data migration and transformation, data queries and report generation.

The DQM Data Management System efficiently consolidates all your Data Assets into a central shared Data Store for the most effective management, access and reporting allowing critical information to be available on demand across networks. The data management query engine can run queries against any type of Data Source using standard relational SQL queries and / or XQuery / XPath.

Indigo DQM Data Management System uses the Indigo DRS Data Reporting and Document Generation System to create advanced reports, charts and documents. Indigo DRS is an advanced reusable Data Reporting and Document Generation platform that represents the ultimate in Data Reporting and is an information and Data Reporting powerhouse. Indigo DRS is totally unique in its ability to Query in both XQuery and Python and use Data from multiple Data Sources and Types simultaneously.

The optional Indigo DRD Report Designer Tools, not included, can quickly create advanced reports rich in content with many options for styling, formatting and customisation. Tables, Charts and Graphs can be added to the report effortlessly allowing complex data to be presented in a visual and meaningful format with the minimum of effort.

Indigo DQM includes all the functionality required for managing and processing data as a valuable resource. An essential data management and reporting system for advanced data analysis, reporting, management information and statistics.

Infrastructure Requirements

Indigo DQM is a Windows based system that requires any Win32 or Win64 operating system (server or workstation): Windows Vista, Windows 7, Windows 8, Windows 10 or Windows Server 2003, 2008, 2012 etc.

The system is compatible with Databases and Data Files on other operating systems such as Linux, Unix, Mac OS etc.

HR Requirements

Internal resources will require experienced DBA and/or SQL programmers for generating the necessary Data Command Queries to be programmed into the DQM Data Engine.

The Indigo DRS Report Designer will require someone with XQuery / XPath knowledge. Custom scripts will require somebody with VBA scripting experience.

Overall a competent IT department should have all the necessary skills to use Indigo DQM and Indigo DRS. Comprehensive support documentation, user guides and examples are included. Training can be arranged at an additional cost if required.

System Walkthrough

The following gives a very brief overview of the Indigo DQM Data Management System by creating a Data Source and a Data Query and showing the Result Outputs and various options for saving the results, reporting and migration.

Indigo DQM Data Management Console

Indigo DQM features the Data Management Console and Data Management Studio for managing Data Assets.

The Data Management Console contains shortcuts to the most commonly used areas of the system for user convenience. The available shortcuts shown are determined by the installed modules of the system.

Indigo DQM Data Management Studio

The Indigo DQM Data Management Studio allows multiple Data Queries, Data Sources, Reports and Execution Plans to be organised into Data Folders in one central location.

Featuring a powerful user interface rich in features and functionality all your data management requirements are handled by the Indigo DMS workplace.

The Data Management Studio is the central workplace for managing all Data, Queries, Data Sources and Reports. The Data Management Engine can run queries against any type of Data Source using standard relational SQL queries and / or XQuery / XPath.

Data Assets are available on demand allowing all Data to viewed, editing, executed and reported on in one central location maximising access and availability. Every conceivable view of Data is provided including Data Grid, XML, JSon, CSV, HTML and RDL Reports. Searching and filtering can also be performed on all Data Results. Indigo DQM includes all the functionality required for managing data as a valuable resource.

Data Results can be viewed in Data Grid, XML, JSon, CSV, HTML, RSS and RDL Report Types.

Creating a Data Source

You can create a private Data Source that a specific Data Command Query or Execution Plan uses or a shared Data Source which all can use. For this example we shall create a shared Data Source. Select the Data Sources Tab from the Data Management Studio.

Indigo DQM supports multiple Data Sources and Types including Microsoft SQL Server, Microsoft Access, ODBC, OLEDB, MySQL, Oracle, Postgre, Firebird, XML, JSon and CSV making it transparent to virtually all Data Sources.

Data Source Connection

Specify the Data Source Connection parameters by selecting the Data Source Type and Connection String from the Connection Dialog.

Data Source Files

Data Source Files can be XML, JSon, CSV or TXT. These Data files can be local, files on a network or files located on the Internet / Intranet.

The URI properties of the Data Source allow the parameters to be defined for Host, User Name and Password if required.

Encrypted Data files can also be downloaded using HTTPS therefore securing Data transmission.

Enter the Host, Port and Path for the Data and the Security Credentials. If the Data is Encrypted with AES 256 Encryption then set the Encrypted Property to True. The Advanced Encryption System (AES) uses a 256-bit encryption key. The secure Key must be setup in the Program Settings and Options.

CSV Data Source File

The name "CSV" indicates the use of the comma to separate data fields. Nevertheless, the term "CSV" is widely used to refer a large family of formats, which differ in many ways. Some implementations allow or require single or double quotation marks around some or all fields; and some reserve the very first record as a header containing a list of field names.

Connection String for the CSV Data Source File.

CSV Data can use different characters as the separator and text qualifiers. These characters can be changed when setting up the Data Source to read CSV Data. Additionally some CSV may or may not have the first line of text as the column header which represents the name of the field in the Data Table.

All these options can be setup for the CSV Data Source thus ensuring complete compatibility with all CSV and Text Data Sources.

Pulling Data down from the Internet is a useful and powerful feature of Indigo DQM.

XML Data Source File

Specify the Location to the XML Data and the Data Source Connection Properties.

The Connection Properties of the XML Data File.

Enter the Host, Port and Path for the Data and the Security Credentials. If the Data is Encrypted with AES 256 Encryption then set the Encrypted Property to True.

Web Service Data

Indigo DQM supports Web Service Data in the form of XML (Extensible Markup Language ) and JSon (Java Script object notation).

Web Service Data Sources can be used by all Data Command Queries and Execution Plans in the system.

Data Folders

Data Folders are containers for Data Commands, Data Sources, Data Reports and Executions Plans in the Data Store. Folders are a way or organising the Data Store so objects can be put in related folders similar to File folders.

Creating a Data Query

Enter the details of the Data Query along with a description and a reference. Create Data Folders to store and organise queries in the system

Using a Data Command Query with a shared Data Source and moving into a Data Folder.

Run Options

You can also run a custom script or program after the Data Command completes if further processing is required. The script can be VBS script that may do additional processing on the Data result outputs if saved to a file or directory.

Testing the Data Command

Data Command Queries can be tested before deployment using the Query Tab.

Click the Execute button to Run the Data Command. The output results are displayed below.

Data Command Files

Data Command Files are useful for sharing Queries with multiple Data Commands in the System. If you have a Query used by several Data Commands create a Command File by saving the Query to File and then pointing the Data Command Query to the Command File.

This is a more efficient way to share Command Text for support and maintenance should the Data Command need changing.

Encrypting the Data Command

The Data Command Query can be Encrypted for added security using the Advanced Encryption System (AES) with a 256-bit encryption key . Check the option and the Data Command will be Encrypted in the Data Store. If the option is Enabled for the Command File then the Command File must be Encrypted by the System. You can Save a Data Command Query from the Query Designer and use the Encryption Tool to Encrypt the Command File.

A secure Password must be setup in the Program Settings and Options or applied to an individual Data Asset Store. By default a global Encryption Password will be used but it is recommenced that you create your own unique Password.

Encryption Passwords must match when Encrypting / Decrypting Data. It is recommended that a copy of your Password be stored in a secure Location. If you lose the Password it will be impossible to Decrypt any Encrypted Data.

Enabling a Result Limit

The results can be limited to a maximum number of Data Rows by enabling the Result Limit option. Select the maximum number of Data Rows to be returned before the Data Command stops Executing.

Enabling Select and Sort Filters

Additional filters and sorting can be applied to the Data Command Query to return even more specific results. For example, in the above Test Data if a Select Filter is applied for Name = 'Cool Doc Fee' then only the Fees with this Name will be returned.

Test the Data Result Outputs

Testing the Data Source you can also apply XPath to the Data Source if CSV or XML. If the Data Source is a Database you can apply SQL and / or XPath to that Database using the SQL dialect for that particular Data Source.

Using the Data Command Query Designer

The Data Command Query designer gives a logical view of the Data Schema.

All Tables, Views and Procedures are shown in the Explorer. The Data Command Query designer allows complex queries to be created and tested before deployment.

The Explorer view of the Data Schema allows items to be inserted into the Query Command Text by drag dropping.

XQuery Designer

XQuery is a query and functional programming language that is designed to query and transform collections of structured and unstructured data, usually in the form of XML (Extensible Markup Language).

The XQuery Designer shows a logical view of the XML Data in a Treeview. Data can be queried using XQuery Commands and the Results displayed below.

XQuery provides the means to extract and manipulate data from a data source. XQuery contains a superset of XPath expression syntax to address specific parts of an XML document. The language is based on the XQuery and XPath Data Model (XDM) which uses a tree-structured model of the information content of an XML document.

Creating Data Reports

Indigo DQM Data Management system uses the Indigo DRS Data Reporting and Document Generation System to create advanced reports and documents for Data Queries and Execution Plans in the Data Asset Store(s). Various other reporting systems are also supported but for the best functionality and compatibility Indigo DRS Reports should always be used.

Select the Report options for the Data Command Query. Specify the Report Type as either an Indigo DRS Report, FYI RDL Report or a MS SQL RDL Report. If the Report is a Shared Report check the Radio Button and then select the Report from the dropdown.

Embedded Indigo DRS HTML Reports

Embedded Reports are stored with the Data Command and allow reports to be automatically output to a file. To Enable HTML reports check the 'Enable HTML Report' option and select either the Default report which is basically just a Data Table or the Indigo DRS Report which is the advanced Report generated with the Indigo DRS Report Designer.

Data Outputs

The Data Results can be output in various formats and saved automatically. Data outputs can be saved in HTML, PDF, RSS, XML and CSV formats for viewing and uploading to Web Servers for Internet / Intranet reporting content.

Select the Data Command Outputs and the location of where the Result Data should be automatically saved after Execution.

Data Migration

Results of the Data Command can also be migrated to another Data Source. Indigo DQM supports migration to multiple Data Sources and Types including Microsoft SQL Server, Microsoft Access, ODBC, OLEDB, MySQL, Oracle, Postgre, Firebird, XML and CSV.

Specify the parameters of the Destination Data Source to migrate the Data.

Data Command Logging

Indigo DQM System includes an audit trial allowing a Log to be kept of all executed Data Command Queries. The Log keeps detailed information about what queries have been run, by whom and when.

Additionally Snapshots of the Data can be taken at the time of execution allowing a Data state record to be kept over time for analysis and comparison.

The Log keeps a complete record of all execution parameters.

If Snapshots are enabled for the Data Command the exact Command Text that was executed at the time is captured along with the Result XML.

Data Execution Plans

Execution Plans can run multiple Data Command Queries against multiple Data Sources and Types in one operation. This powerful feature allows Data from different locations to be brought together into one result with the result outputs being available for snapshots, saving, migration and reporting.

Data Consolidation

Data consolidation using Execution Plans allows the collection and integration of data from multiple sources into a single destination. During this process, different data sources are put together, or consolidated, into the Indigo DQM Data Store.

Creating a new Data Execution Plan

Enter the details of the Data Execution Plan along with a description and a reference. Create Data Folders to store and organise Plans in the system.

Adding Data Commands to the Execution Plan

Select the Data Command Queries to run in the execution plan and select the order in which you want them to Execute. You can also choose to include or not include the result as part of the Execution Plan.

Adding Execution Data

Execution Data can be added to the Result XML which can be used for reporting information. Check the option and Execution parameters will be included in the XML Data such Execution Time, Username etc.

Testing the Data Output of the Execution Plan

The Execution Plan can be tested before deployment using the Results Tab.

Click the Execute button to view the XML Data Output for the Execution Plan.

Run Options

You can also run a custom script or program after the Execution Plan completes if further processing is required. The script can be VBS script that may do additional processing on the Data result outputs if saved to a file or directory.

Data Reports for the Execution Plan

Indigo DQM Data Management system uses the Indigo DRS Data Reporting and Document Generation System to create advanced reports and documents for Data Queries and Execution Plans in the Data Asset Store(s). Various other reporting systems are also supported but for the best functionality and compatibility Indigo DRS Reports should always be used.

Select the Reports Tab for the Data Execution Plan.

Indigo DRS Reports give the best functionality and compatibility. Because Indigo DRS uses standard HTML for reporting the reports can be displayed on any type of device and in any type of browser.

Data Outputs for the Execution Plan

The Data Results can be output in various formats and saved automatically. Data outputs can be saved in HTML, PDF, RSS, XML and CSV formats for viewing and uploading to Web Servers for Internet / Intranet reporting content.

Data Execution Log

In addition to the Data Command Log Indigo DQM also includes an audit trial for Data Execution Plans. The Log keeps detailed information about what Plans have been run, by whom and when.

Additionally Snapshots of the Data can be taken at the time of execution allowing a Data state record to be kept over time for analysis and comparison.

A complete log of all Executed Plans can be kept along with Data Snapshots.

The Log keeps a complete record of all execution parameters.

Data Results

If Snapshots are enabled for the Data Execution Plan the exact Command Text that was Executed at the time is captured along with the Result XML.

Shared Data Reports

Shared Data Reports are reports that can be used by all Data Command Queries and Execution Plans in the system.

If the report is common to multiple Data Command Queries or Execution Plans then create a shared Data Report.

By default the Indigo DRS report type is selected as this gives the best functionality and compatibility. Indigo Scape DRS is an advanced Data Reporting and Document Generation System using HTML, XML and XQuery to generate highly compatible and content rich business reports and documents with standard HTML.

Using a Shared Indigo DRS Report on the Internet / Cloud

Pulling Shared Data Reports down from the Internet / Cloud is a useful and powerful feature of Indigo DQM. This feature is only available with Indigo DRS Reports.

Report Save Output URI for the Result Data

Data Report Outputs can be saved to a local drive, network share or uploaded to the Internet / Cloud by configuring the Location of the Data Report Output URI. This feature is only available with Indigo DRS Reports.

Specifying the Host and Location for the Data Report

Enter the Host, Port, Path for the Data Output and the Security Credentials.

Indigo DRS Report Designer Tool*

The optional Indigo DRD Report Designer Tools, not included, can quickly create advanced reports rich in content with many options for styling, formatting and customisation. Tables, Charts and Graphs can be added to the report effortlessly allowing complex data to be presented in a visual and meaningful format with the minimum of effort.

Management Reports are only a click away with the HTML report designer which can rapidly create formatted reports against the Data Results with many options for customisation and styling. Indigo DRS advanced reporting and document generation platform gives the best functionality and compatibility with advanced reporting features and effortless integration of this technology into your projects you can be assured of having the best reporting capabilities.

Click here for more Information on Indigo Scape DRS Data Reporting and Document Generation System.

Report Properties

To define the Report Data Source open the Report Properties dialog or select the Report Node and edit the properties in the Property Grid.

The Report Data Source is a Dataset or Document in the form of XML Data. By default the Report Designer will automatically look for a Schema XSD File and apply it to the Data Source. An XSD File can also be specified for the Data Source in the report properties. If the Designer cannot find an XSD File the Report Data Schema will be inferred as a typed Schema from the Report Data XML.

The Data Source File and Schema defined in the Report Designer are only used in Design mode. When a Report is run against a Data Command or an Execution Plan the Result Data and Schema will be used from the Data Command or Execution Plan respectively.

Reports can be defined with a background Image so for example if a particular report is a draft then background image can be defined to indicate this.

Report CSS

Cascading Style Sheets (CSS) is a style sheet language used for describing the look and formatting of a document written in a markup language such as HTML. CSS is designed primarily to enable the separation of document content from document presentation, including elements such as the layout, colors, and fonts.

This separation can improve content accessibility, provide more flexibility and control in the specification of presentation characteristics, enable multiple HTML pages to share formatting by specifying the relevant CSS in a separate .css file, and reduce complexity and repetition in the structural content.

It is recommended that Cascading Style Sheets are used on all Indigo DRS Reports. Certain Report Items such as Tables and Paragraphs have a property to define the Style Sheet Class from the CSS thus determining the appearance of that particular Report Item.

Data Security and Encryption

Indigo DQM features AES 256 Encryption allowing Data Assets to be stored and transmitted across the Internet / Cloud in the most secure way possible.

Data Result Outputs, Data Source Files, Raw Data and Backup Data can be Encrypted and stored securely using the highest level of cryptography.

The Advanced Encryption System (AES) 256 is a symmetrical encryption algorithm that uses a 256-bit Encryption Key. It is accepted by the U.S. and Canadian governments as the most secure way to store sensitive Data.

With AES-256 bit encryption you can be assured that you will be the only one who can access your critical information.

Encryption Keys

The Encryption Key must be setup in the Program Settings and Options. A Default Key can be used but it is recommended that you generate your own secure Key.

Encryption Keys must match when Encrypting / Decrypting Data. It is recommended that a copy of your Key be stored in a secure Location. If you loose this Key it will be impossible to Decrypt any Encrypted Data.

Data Asset Stores

The Data Asset Store is a Database that holds all the Data Command Queries, Data Folders, Data Sources, Data Reports, Execution Plans and Execution Logs in a shared Data Repository. Its purpose is to consolidate Data Assets into a shared repository for the most efficient Data Management, Processing, Access and Reporting.

Data consolidation allows the collection and integration of data from multiple sources into a single destination. During this process, different data sources and types are put together, or consolidated, into the Indigo DQM Data Asset Store.

Indigo DQM allows multiple Data Assets Stores to be configured with Data Asset Stores Tool. Different Data Stores can be Added, Edited, Delete and Activated.

Configuration of multiple Data Asset Stores

Multiple Data Asset Stores can be setup and Activated 'Switched' using the Data Asset Stores configuration Tool.

To Activate the Data Asset Store click the Activate Button and the Store will become the Active Store.

Indigo DQM Data Asset Store supports multiple Data Sources Types including Microsoft SQL Server, Microsoft Access and MySQL. The Data Store can be migrated to any of these Data Source Types using the inbuilt features of the system or with the tools provided with your Database.

Default Data Asset Store

Initially the system comes with a default Data Assets Store as an MDB Database which is installed in the User Documents Folder. Additional MDB Data Asset Stores can be created on the fly using the MDB Data Store Wizard from the Tools Menu on the Data Management Studio.

Adding a Data Asset Store

Adding a new Data Asset Store using a Microsoft SQL Server Database. Enter a Name and Description for the Data Store.

Specifying the connection parameters to the Data Store using the Connection Dialog.

Data Encryption

Indigo DQM features AES 256 Encryption allowing Data Assets to be stored and transmitted across the Internet / Cloud in the most secure way possible.

Data Result Outputs, Data Source Files, Raw Data and Backup Data can be Encrypted and stored securely using the highest level of cryptography.

Encryption can be setup globally in Program and Settings or applied to an individual Data Asset Store where a Private Encryption Password will be used to En

To use a Private Encryption Password on the Data Store select the Radio button and Enter a Strong Password.

Click the Generate button to automatically create a strong Encryption Password. The strength of the Password is shown by the Quality Bar.

Great care must be taken to keep copies of Encryption Passwords in a secure location. If you lose your Password you will not be able to Decrypt your Data. You must also remember what Password you used to Encrypt Data if you decide to change the Password at a later stage. Encryption / Decryption will only work if the Encryption Passwords are the same.

Data Store Properties

To View the Advanced Properties of the Data Asset Store click the Properties button.

Exporting and Importing Data Stores

Data Asset Stores can be Exported and Imported using the Export and Import Wizard on the Data Asset Stores Tool.

Indigo DQM allows multiple Data Assets Stores to be configured with Data Asset Stores Tool. Different Data Stores can be Added, Edited, Deleted and Activated.

*NOTE: Indigo DQM is compatible with the Indigo DRS Data Reporting and Document Generation System. However, Indigo DQM does not include the Indigo DRS Report Designer Tools or the Reusable Report Engine Component SDK these are optional extra that require licensing separately!