Paper: Converting Adabas To Ibm Db2 For Z/Os With Consistads
Paper: Converting Adabas To Ibm Db2 For Z/Os With Consistads
Redpaper
                                                                                               Paolo Bruni
                                                                                           Clauder Balzano
                This IBM Redpaper publication explains the Natural and Adabas1 conversion to DB2 for z/OS
                by using ConsistADS. It includes prerequisite technical assessment requirements and
                conversion challenges. It also describes a real customer conversion scenario that was
                provided by the IBM Business Partners that facilitated these conversions for customers.
                1   Also written as ADABAS. It stands for advanced database system. Adabas is often used in
                    conjunction with Software AG’s programming language, Natural.
              When customers start a conversion process from Natural and Adabas to DB2 for z/OS, they
              want to use the relational database capabilities and features that are offered by using
              relational database architecture. These capabilities and features provide a flexible database
              platform that meets the following business requirements:
                 Web-enabled applications
                 Service-oriented architecture and business process management (SOA and BPM)
                 Data warehouse
                 Business analytics
                 Easy recoverability
                 Two-phase commit
                 Increased synergy with security
                 Utilities performance
                 SQL flexibility
                 Uses tables larger than 4 billion rows
                 Use of DB2 for z/OS features, such as partitioning and parallelism
                 Data sharing
                 Large objects
                 Stored procedures
                 Triggers
                 Referential constraints
Adabas requirements
              When we think about an Adabas conversion, it is possible to treat each Adabas file as a
              relational DB2 table. However, this simplistic assumption must not be made. Many Adabas
              database designs create multiple data type records within a single Adabas record, which is
              counter-intuitive to the relational table normalization philosophy. Because of the major
              differences in the data file system design models, a database administrator often is required
              to split Adabas files into several tables to reap the full benefit of a relational model design. A
              relational database system also allows the users of a client to query any table column by the
              name of the column based on a value, which is a capability that is not always allowed for in an
              Adabas file design.
              The following steps are used to convert Adabas data to DB2 for z/OS:
              1. Verify that each Adabas file is in third normal form. The file must be normalized if it is not
                 already.
              2. Create a DB2 table for each resulting file.
              3. Translate each Adabas field to a DB2 column.
              4. Identify a primary key for all defined tables.
              5. If referential integrity is implemented, a foreign key must be defined for each dependent
                 table.
Conversion challenges
          The following challenges must be considered when a customer chooses an Adabas
          conversion method:
               Adabas inventory
               Disk storage estimates
               Data relationship
               Performance
               Applications
          Adabas inventory
          Estimating the workloads, costs, and schedules for the project is important. Therefore, it is
          necessary to have a thorough inventory of Adabas and application data volumes. Customers
          that have the PREDICT data dictionary generate reports that provide all of the requisite
          information.
              Data relationship
              The goal of the conversion is to avoid redundant data by normalizing the data model. The
              chosen conversion tool must provide flexibility to help the customer decide whether to
              denormalize the multiple occurring field (MU) and the periodic group (PE) field. However,
              normalization is often used to create new tables.
              The tool must conduct the data type conversions and the character conversion according to
              the UNICODE, ASCII, and EBCDIC formats.
              Adabas allows field names to include hyphens, as shown in Example 1. The tool must convert
              these field names to the accepted DB2 for z/OS standards.
              The NULL value is treated differently between the databases. Adabas defines blank
              alphabetic fields and zero numeric fields as “null.” In DB2, null values are neither zero nor
              blank; they are unknown. A way to address this difference is to define the new columns by
              using NOT NULL WITH DEFAULT.
              With alphanumeric fields, Adabas allows for storing a field that is longer than its definition;
              therefore, you use the VARCHAR data type to manage the field.
              Performance
              Before you start the process to convert to DB2 for z/OS, you must consider the following
              issues:
               Avoid too much normalization of the physical model, because that might cause DB2 for
                z/OS performance degradation.
               Apply filter factors to avoid reading unnecessary data.
               Keep the same indexes used in Adabas and improve the indexes whenever possible.
               Understand the DB2 sort process.
               Avoid the use of cursors when only one row is retrieved.
               Avoid the use of numerous joins because of the creation of Natural data definition modules
                (DDMs).
               Use CURSOR for UPDATE only in the necessary columns.
              Applications
              The conversion of the Natural procedural language is one of the most challenging steps for
              the migration. This conversion is challenging because of the manner in which Natural
              compiles all of the objects (MAP, PROGRAMS, SUBPROGRAM, DDM, and so on), and
              commands when accessing the Adabas database.
        In addition to the changes required to convert Adabas calls to DB2 SQL, there might be
        application logic changes required as a result of the DB2 design. The following application
        changes result from data changes made to resolve differences between an inverted list,
        nonrelational database, and a relational database:
           Normalization effect
           Record splitting
           Repeating groups
           Record to multiple tables
           Translating group level fields
           Perform READ LOGICAL by using DB2 cursors
        Consist Software Solutions Inc. (Consist) developed a solution that combines the benefits of
        the conversion and transparency methodologies. Consist examined numerous approaches to
        the conversion and designed a solution based on the following demands of the client:
           Faster conversion
           Lower risk
           Increased cost savings
           Automatic functionality enhancement of the applications
        Conversions that are conducted by Consist are highly successful and the company is
        positioned as specialists in IBM DB2 for z/OS when a low-cost and short conversion period is
        required. Consist allows customers to convert multiple applications simultaneously.
        The important decision for any customer is to choose a solution that replaces Natural and
        Adabas from a vendor capable of providing an end-to-end automated conversion solution. A
        vendor also must have extensive experience in working with Natural and Adabas, converting
        applications to a DB2 environment, and a working knowledge of application development in
        the IBM System z environment.
        Consist offers ConsistADS to convert Natural and Adabas to DB2 for z/OS. Consist has
        decades of experience in delivering solutions that are written in Natural and Adabas in the
        System z environment.
        ConsistADS is an end-to-end conversion solution that includes DB2 for z/OS and several DB2
        Tools as part of a packaged solution that is delivered to the customer.
ConsistADS overview
              ConsistADS delivers complete automatic conversion and execution of business applications
              that are written in Natural and Adabas. ConsistADS with DB2 for z/OS offers a set of tools
              that automatically runs the conversion from Adabas database to DB2 and converts NATURAL
              programs, including mapping the original Adabas access calls to SQL commands.
              ConsistADS monitor allows high availability, transparency, and SNA integration because it
              behaves as a transaction coordinator through XCF protocol in z/OS. Multiple instances of the
              ConsistADS server in z/OS or in DB2 for Linux, UNIX, and Microsoft Windows can be
              reached by its IBM VTAM interface, which is used to communicate with the SNA network to
              provide transparent operation with 3270 terminals and to participate in an IBM CICS
              transaction environment.
              ConsistADS with the ConsistDMS component integrates DB2 for z/OS database and converts
              the Adabas hierarchical data file structures into DB2 SQL Data Manipulation Language
              (DML) objects.
              ConsistADS IDE combines functions that are used by any organization. ConsistADS IDE also
              allows the organization to develop business applications quickly and efficiently by taking
              advantage of the benefits of the latest information technology.
              Editing, compiling, debugging, installing, and distributing the functions of the ConsistADS
              objects allows customers to develop and test an application in a z/OS controlled environment.
              This flexibility results in IT cost reductions and increased productivity by optimizing
              IBM System z resources.
ConsistADS solution
           With ConsistADS suite, Natural and Adabas-based application systems are fully migrated and
           modernized. In addition to saving significant licensing and maintenance costs, customers
           benefit from DB2 for z/OS database and modern web User Interface technology.
           ConsistADS is a complete IDE suite that features a compiler, runtime system, and migration
           toolset. The special feature of ConsistADS is the capability to automatically migrate and run
           multiple application systems that are coded in Natural in a single conversion step.
           ConsistADS provides all of the tools necessary to preserve the original investment in mature
           applications that are written in Natural. The source code automatically is converted to Consist
           Advance, the 4GL language of Consist that is compatible with Natural. The full scope of the
           functionality of the language is run in ConsistADS with DB2 for z/OS.
           ConsistADS is an IDE platform that includes the Advance language, the RDBMS, and the
           ConsistDMS components for the conversion of the Adabas data to an RDBMS, such as DB2
           for z/OS. Ancillary DB2 tools are offered and packaged directly by Consist under a global
           OEM software partnership with IBM for the integration of IBM products into ConsistADS.
Customers experience the following benefits of the ConsistADS solution with IBM software:
              For more information, see the IBM DB2 tools for z/OS web page:
              http://www.ibm.com/software/data/db2/zos/tools/
              The following set of DB2 for z/OS tools is provided by Consist at the request of the customer:
               DB2 Administration Tool for z/OS
                    This tool simplifies the complex tasks that are associated with safely managing DB2
                    objects and schemas throughout the application lifecycle with the least possible effect on
                    availability.
               IBM Tivoli OMEGAMON XE for messaging for DB2 Performance Expert on z/OS
                    This tool evaluates the efficiency and optimizes the performance of the DB2 for z/OS
                    environment.
               DB2 Utilities Suite for z/OS
                    The Utilities Suite minimizes the downtime that is associated with routine DB2 data
                    maintenance and ensures the highest degree of data integrity.
               DB2 SQL Performance Analyzer for z/OS
                    This tool helps improve DB2 application design to achieve maximum productivity by
                    testing different “what if” scenarios.
               DB2 SQL Tuning for z/OS
                    The DB2 SQL Tuning for z/OS tool enables efficient customization and tuning of SQL
                    workloads and DB2 objects.
               DB2 Query Monitor for z/OS
                    This tool enables the customer to efficiently customize and tune the SQL workload and
                    DB2 objects.
              ConsistADS is packaged with IBM DB2 for z/OS and related DB2 tool products in the
              ConsistADS DB2 for z/OS solution set.
              The Consist solution must be installed in an IBM System z New Application License Charges
              (zNALC2) LPAR. Usage charges for DB2 and any DB2 utilities are based on the size of the
              zNALC LPAR.
              2
                  zNALC extends the IBM commitment to subcapacity pricing, allowing customers with a Qualified Application to
                  obtain a reduced price for z/OS. Charges are based on the size of the LPARs that are executing a Qualified
                  Application, assuming that all applicable terms and conditions are met.
                 The mainframe monitor also makes available to VTAM the entire set of ConsistADS
                 transactions in effect in every node of the network. This provides integration with CICS.
                 The virtual machine performs all of the standard functionality of the languages, such as
                 calculations, file access, device access, database access, and data movement.
                 ConsistADS virtual machine configuration options are provided through configuration files,
                 which are maintained by the ADSCONF utility.
                Integration infrastructure
                ConsistADS subprograms are integrated through web services. Each subprogram is
                converted to a Java class, and those Java classes are packaged and accessible through
                Apache Axis2.
                ConsistADS IDE
                The ConsistADS IDE is a graphical IDE used for software application development and
                maintenance of source code through a web interface. As Figure 2 shows, the tool provides
                comprehensive facilities, such as authoring, modifying, compiling, debugging source code
                software, and deployment of compiled software.
DMS overview
The DMS is a core component of the ADS solution that is used to migrate data from Adabas
to a DB2 database. DMS is implemented by using the Common Web Architecture (CWA)
environment. DMS contains its own database in which the migration control information is
stored and ensures a complete and accurate migration from the original Adabas hierarchical
files to the DB2 target database. DMS also provides the specific information that is needed to
allow the original application to run and access the new DB2 database.
DMS environment
DMS delivers the components that automatically convert Adabas hierarchical files into a DB2
target database in a single conversion process. The following components are included in
DMS:
         Adabas concepts: The specific concepts of Adabas are not covered in this book. The
         focus of the book is on the overall conversion process.
        Data design
        In this section, we examine data constructs mapping from Adabas to DB2.
For the DDMs, the conversion tool creates the VIEWs on DB2.
              Several approaches are used to resolve these repeating groups by using a relational model.
              Figure 5 shows the relational model that was normalized for these groups of repeating data,
              which results in new DB2 tables. The tables require foreign keys to establish all of the
              relationships that are declared in DB2 by using the referential integrity rules.
              The primary and foreign keys are defined in DB2 to maintain those relationships that are
              created in Adabas. If the keys exist, the keys are left in the code for the conversion.
                  Referential integrity: For more information about referential integrity in DB2 for z/OS, see
                  Data Integrity with DB2 for z/OS, SG24-7111.
              Other options are available to use according to performance needs and data modeling
              denormalization rules.
Another option is to use a fully denormalized data model. However, this option causes
performance issues, as shown in Figure 7.
Super-descriptors
Super-descriptors are the equivalent of indexes in DB2 for z/OS. A super-descriptor is a new
index that is created and used in programs as search criteria.
Application design
Adabas supports several procedural languages, such as COBOL, by using ADASQL and
direct calls. Natural is the most commonly used language, so we focus more on this language
in this section.
Example 2 shows the conversion for GET ISN, in which the accesses are made physically to
the record location.
              Table 2 shows how Natural commands in the application are matched to SQL statements in
              DB2.
FIND SELECT
STORE INSERT
              Many SELECT calls require a CURSOR on DB2 for z/O. Cursors must be disabled when only
              one row is retrieved.
              The ORDER BY clause requires sort use, unless indexes are used in the columns that are
              referenced by the ORDER BY clause.
An SQL tuning team must check the DB2 access path, index use, DB2 resources, and so on.
               DB2 for z/OS programming: For more information about DB2 for z/OS programming, see
               DB2 10 for z/OS Application Programming and SQL Guide, SC19-2969.
Customer’s challenge
          The customer wanted to upgrade their System z servers, modernize their IT environment, and
          reduce costs.
          The customer required the conversion of the Natural applications that are listed in Table 3 on
          page 17 and the Adabas files that are listed in Table 4 on page 17.
APP4 75 13100
30 5 123 75 869.0
              Figure 8 shows the general overview of the multi-server solution using ConsistADS. The
              monitor component is responsible for the License Control and serves as a central repository
              for licenses over the entire installation. This component also includes the Transaction Server
              which can be used to interface with VTAM terminals and other z/OS components such as
              CICS.
              The monitor accesses the ConsistADS application servers located in z/OS, z/Linux or other
              LUW platforms. This access is possible because the ConsistADS architecture uses TCP/IP
              as the standard for interfacing with the end-user as well as any other process that
              communicates with it. Due to this characteristic, ConsistADS can communicate with Natural
              RPC, and standard Web-services. AXIS2 objects can directly access the objects that run
              inside those VM Containers.
              Further, the monitor can be installed in any LPAR not necessarily related to the Application
              Server LPARs. The XCF protocol implements the support for sysplex, allowing multiple
              distinct and independent servers to be active at the same time.
              With details of one of the implemented logical partitions (LPAR) in z/OS, a typical production
              environment is shown in Figure 9.
          Figure 9 illustrates the ADS functionality of each application server environment in a z/OS
          LPAR. As mentioned before, the ConsistADS z/OS Monitor does not need to be located in the
          same LPAR as the ConsistADS application servers. The figure just exemplifies this possibility
          to illustrate the support for 3270 user interface under control of it (what is “it”?).
          The virtual machines can run independently in the case of batches, and under the control of
          the application server which implements the TCP/IP layer responsible for the all types of
          communication.
          The ADS Web Protocol is used to distribute the online and integration features over the entire
          network. So the figure shows not only users for online interface, but also, developers that are
          testing their applications and the Web-services that are consuming the functions provided by
          the application in z/OS.
          The converted Natural language, Advance, runs over the runtime environment in batch mode
          and online. The interface uses the web server to show web windows instead of showing the
          Natural maps.
Project phases
          The entire project took approximately six months to complete. Table 5 shows the amount of
          time spent on each phase of the project.
               Define JCL streams and batch jobs integration    Two months (conducted in parallel with the two
                                                                previous phases)
              The customer decided to implement each application as described in the methods for the
              conversions.
               Unique challenges: Each customer environment could pose unique challenges that might
               affect timing for the project.
Data conversion
          The ConsistADS solution uses the DMS to convert Adabas files to DB2 tables. DMS maps the
          Adabas calls to SQL commands to access DB2 for z/OS.
          DMS contains its own database in which the control information is stored. The contents of
          DMS ensures a complete and accurate migration from the original Adabas to the target DB2,
          and allow the original applications to access DB2.
Figure 10 shows the flow of the steps to conduct the data conversion.
Importing DDMs
DDMs are the equivalent of the table views in DB2 for z/OS.
The DDMs must be converted to text format through the DMS Source File Migration function
before the DDMs are used by the IMPORT utility. The DDMs must be extracted from the
Natural production environment by using the SYSOBJH utility (Transfer Mode). The DDMs
are then transferred to the source objects in the development environment by using FTP.
              Modeling
              After the FDTs and DDMs are imported into the DMS system, the modeling process is started
              for each imported FDT.
              Each modeling process that is run generates a logical file that represents the correspondence
              between the original Adabas file and the new relational data mode. Adabas allows for a single
              FDT to support several DDMs, and a single field in the FDT is reused within several DDMs
              that serve as the logical record criteria to extract data into DB2.
              A logical file defines the relationship between the original Adabas file and the target relational
              data model in terms of tables and columns. Through the logical file abstraction, we model
              Adabas multi-files into several logical views in a relational database paradigm.
Figure 15 shows the relationship of DDMs fields to each column in the target table.
The user generates the DB2 DDL for each logical file, as shown in Figure 17.
              Mapping
              The Mapping function generates the DDMMAPPING file, which is used by the ADS compiler to
              replace all of the Adabas call statements with SQL. This replacement is an important
              component of the application conversion process.
This function generates the JCL necessary for the execution of the Conversion and Validation
programs as part of the migration process on the source or the target environment.
After the generation process is complete, the JCL statements must be transferred to the
source and target environments for execution.
              After these steps are completed, it is possible to start the conversion of the Natural language
              as described in the next section.
           As shown in Example 4 on page 29, Example 5 on page 29, and Example 6 on page 30, you
           use the programs to access DB2 for z/OS dynamically or in static mode.
           Once the application is compiled, both batch and online processes are run in the ADS
           runtime environment, as shown in Example 5 on page 29 and Example 6 on page 30.
              Table 6 Results
               Before conversion                                After conversion
3 CPUs 3 CPUs
12 GB memory 5 GB memory
20 MSUs 20 MSUs
Author
              This paper was produced by a team of specialists from around the world working with the IBM
              Redbooks team.
              Paolo Bruni is an IBM Redbooks Project Leader who is based at the Silicon Valley Lab in
              San Jose, California, in the US. Since 1998, Paolo has authored IBM Redbooks publications
              about IMS, DB2 for z/OS, and related tools and has conducted workshops worldwide. During
              his many years with IBM in development and in the field, Paolo’s work has been related
              mostly to database systems.
        Find out more about the residency program, browse the residency index, and apply online at:
        ibm.com/redbooks/residencies.html
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
  AIX®                                OMEGAMON®                           System z®
  CICS®                               RACF®                               Tivoli®
  DB2®                                Redbooks®                           VTAM®
  IBM®                                Redbooks (logo)    ®                z/OS®
The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.