Basics of Data Data Migration Using IS Migratio Migratio n Workb ench
-Gagan Agrawal
1
INTRODUCTION
1.1
Pu rp os e o f t he Do cum ent
The purpose of this document is to explain
Basics of Data Migration
What Is Data Migration Steps Required For Data Migration Data Migration Process Flow IS-U/CCS Data Migration Scenario
What Tools/Technologies Is Being Used And Why
IS Migration Workbench Concepts
The IS-U Migration Workbench Principle Migration Object Definition Automation Data Definition Field Rule Definition Fixed Value Definition Conversion Object Definition Other Important Terms Special Technologies List of Standard Migration objects Hierarchy display of Migration Objects
2
BASICS OF DATA MIGRATION
2.1
What is Dat a Mi gr ati on
When implementing a SAP system in any organization the data stored in existing IT applications has to be transferred to the SAP database. Normally it will require manual entry of data through keyboard using user transactions. If the amount of data is too large this process becomes difficult, costly, and time consuming. Data Migration is the transfer of data from Non-SAP systems (Legacy applications) to SAP system through the use of data transfer technologies and tools provided by SAP. Relevance of Data Migration :
Accounts for 15% - 20% of the total SAP implementation costs Smaller implementation projects: up to 40%
Why is data migration so expensive?
In many cases you need to develop conversion programs (which can not be used later on) you need to deal with lots of technical details and different technologies data volume depends on the project volume the effort for programming does not depend on the data volume
Basics of Data Migration
Page: 1/13
2.2
Steps Required For Data Migration
To successfully carry out any data migration project following steps should be followed:
Determine which business objects are to be transferred. Unload data from legacy system into files. Determining required Fields of source and target systems. Mapping source and target fields. Analyzing and cleaning legacy data. Determining what data transfer method/tools should be used. Configuring Data Transfer Tools – Creating Projects, Setting field rules and generating data transfer programs. Testing – Testing data transfer with test data. Convert extracted and cleaned data into the structures of the transfer file (file in SAP format). Final Data Migration – Load converted data into SAP system. Checking the results with migration statistics
Determin ing Bus iness Objects To Be Transferr ed
The objects to be transferred depend on the business application that you are using. Determine which business objects needs data to be transferred and which don’t. Unloading Data from L egacy system
Data stored in the database of legacy system needs to be unloaded into sequential file or spreadsheet. If the legacy system is R/2 then it can be done by SAP tools, an otherwise, external program needs to be written to perform this task. Determin ing Required Fields of Source and Target systems
You need to determine what data is needed by finally configured SAP systems i.e. prepare a list of fields that must be filled by legacy data. Also, make a list of data fields provided by legacy system. Mappin g Sourc e and Target Fields
The one to one mapping between source field and target field needs to be done i.e. from where data is coming and where it is going. Also, it should be determined that what modification or conversion needs to be done while processing that field. Determine how the data is structured (length and sequence). Data type of source field and target field should be given a special care. Analyzi ng and Cleani ng Legacy Dat a
Data in the legacy system must be cleaned to get rid of incorrect or incomplete data. Determine which data can be transferred unmodified, which must be converted and which cannot be transferred at all. Perform all necessary corrections on incorrect data. Determi ning What Data Transfer Method/Tools Shou ld B e Used.
SAP provides a large number of tools and techniques for migrating data from legacy system to SAP system. Each of these has its own benefits and limitations. You must determine which tool / technique best suits your migration needs.
Basics of Data Migration
Page: 2/13
You must take following facts into consideration: How much time you have. Time taken for developing data transfer programs. Time taken to transfer data (Speed of Data Transfer). Level of Data Integrity Offered. Error handling/ Analysis tools offered. Periodic or One time Data Transfer. Internal or External Data Transfer. Foreground or Background Data Migration
SAP provides following techniques for Data Migration:
BDC (Batch Input Session and Call Transaction) SAP supplied Batch Input and Direct Input programs BAPI/RFC ALE/IDoc SAP provides following tools for Data Migration:
Data Transfer Workbench LSMW (Legacy System Migration Workbench) CATT (Computer Aided Test Tool) IS-U Migration Workbench
Configuring Data Transfer Tools
If using any Data Transfer Tool, you must configure it to carry out data migration according to your requirement. It involves creating data transfer projects, setting various attrib utes, setting field rules (rules that determine how data is processed during migration), and generating data transfer programs. Testi ng – Testing Data Transfer With Test Data
After configuration of data transfer tools and generation of data transfer program all settings and developments must be tested with test data. For this, a small test file should be created with test data and data transfer testing should be done using this test file. Error logs if any should be analyzed and necessary corrections should be done. Almost all of data transfer tools provides the option of creating test data file and error log generation. Convert Data File to SAP Form at
SAP requires data to be arranged according to its own format. Extracted and cleaned data must be converted to the format required by SAP. Some Data Migration Tools provides this facility. If this facility is not provided by data migration tool you are using, customer programs needs to be written to carry out this conversion. Final Data Migrati on
After all configuration, development and testing has been done and input file has been prepared according to format required by SAP, final Data Migration can be carried out using this file. After this data migration, all data from the input file will be transferred to SAP system’s database. Checking the Results
If in any case error has occurred, analyzing error logs and migration statistics can check it. Necessary action must be taken to migrate the data correctly.
Basics of Data Migration
Page: 3/13
2.3
Dat a Mi gr at io n Pr oc es s Fl ow
One or several files
How Data Migration works
Structure relations
Legacy format
Legacy data on PC
Read data
Legacy data on application
Field mapping Convert data DATA TRANSFER TOOLS
Conversion rules
Batch Input processing (BDC) SAP for mat
Direct Input processing
SAP System
BAPI/IDoc
2.4
Data to be migrated is stored in database situated either on PC or on Legacy Application Server. The first step is to extract this data from legacy application’s database and put it in either a tab delimited text file or any spreadsheet. This data is then cleaned to remove any existing inconsistencies. The data stored in these files are in legacy format. This data must be arranged according to SAP format. So, next step will be to convert this data to SAP format. After the data has been converted to SAP format, using any one of SAP provided technologies i.e. BDC, Direct Input or BAPI/IDoc does actual data migr ation. This can be done by using any one of SAP provided data transfer tools or by developing custom data migration programs, which use above m entioned technologies. The data is now migrated to SAP system. IS-U/CCS Data Migration Scenario
During the implementation of IS-U-CCS component of SAP, an unusually large data volume has to be extracted from one or more legacy systems, and moved to the database of the R/3 system. Since the production system cannot tolerate an extended downtime (also unacceptable for customer care and contract accounting reasons), this data transfer is subject to high performance demands. It is therefore necessary to optimize all the involved resources.
Basics of Data Migration
Page: 4/13
In case of data migration for IS-U-CCS there are a large number of issues which must be addressed for successful migration of data to SAP system. They are listed below :
2.5
Data to be transferred is usually very large in volume. So, special care should be given to the speed of data transfer.
Various business objects of IS-U-CCS are interlinked and dependent to each other. So, a referential integrity check must be implemented while transferring data of these business objects.
Since IS-U-CCS is a billing and customer care system, it cannot tolerate extended downtime. It requires data to be migrated in the fastest possible way.
Migration of data for several business objects is time-dependent. For example for MOVE-IN process. So, there should be a mechanism which can handle this time dependency during data migration.
The data of IS-U-CCS system is of very high magnitude of complexity. So, a mechanism is needed which can effectively process data on field by field basis.
In case of errors, a restart mechanism is needed so that data transfer can be continued from where the error occurred.
Some business objects of IS-U-CCS needs to be transferred in a single run to cover interdependency. This facility must be provided by data transfer tool/technology being used. What Tools/Technologies Is Being Used and Why
For data migration of HT and LT consumers w e are using IS-U Migr ation Work bench which transfers data from legacy system to SAP system through use of Direct Input method. Direct Input is the fastest available technique for transferring data. It doesn’t process screens for transferring data but perform all necessary checks through function modules. IS-U Migration Workbench is a special tool provided by SAP to handle complexity involved in migration of IS-U data. SAP IS-U Migrati on Wor kbench – General Facts
SAP IS-U Migration Workbench A tool that complements the known migration techniques Used for migrating SAP IS-U/CCS and SAP IS-Waste business objects Comprehensive cover of all data transfer actions Designed for data migration from R2-RIVA system with SAP unload programs Any source system with own unload program High Performance Migration Import programs based on fast Direct Input technique Parallel import possible Interface structure width can be reduced Basics of Data Migration
Page: 5/13
SAP IS-U Migr ation Wo rkbench - Techn ical Facts
Open Interface Structures Independent data model from the source system Transfer in object-oriented form (business objects) Service function modules save data Optimum consistency check (based on dialog check) High throughput using Direct input Technology Possibility of adjusting migration structures Reducing data flow from source system Migration control due to migration customizing Easy to create and change conversion rules for each field Offers high degree of flexibility for each single field mapping Key and Status Management in SAP R/3 Easy to reference data objects using their legacy system key Test run feature Import without database updates Migration statistics continually updated Overview of processed file and current status of import runs Counter for successfully imported objects and errors Performance overview Error Log Collects all reported error messages Identifies corrupt data records Error Files Reusable Import directly once error is solved SAP IS-U Migr ation Wor kbench – Overall Funct ional Scope
Migration of All master data
All billing relevant master and transaction data needed for the next billing execution Device Installations Meter Readings Billing Parameters Consumption history Billing Period
All necessary transactional data Open items Dunning Items Budget Billing Plans Instalment plans Deposit
Basics of Data Migration
Page: 6/13
SAP IS-U Migr ation Wor kbench – How it so lve our pr obl ems
It solves our migration speed problem by using Direct Input method – the fastest data transfer method.
For our problem concerning interlinking of various business objects and referential integrity check it provides a mechanism called Key and Status Management which interlinks inter-dependent business objects with a key.
It perfectly handles date dependency of various transaction data by keeping proper context of historical data. For example it handles the MOVE-IN process as described below
It provides a facility to set processing rules for individual fields which handles data at field level.
It provides a restart mechanism which enables us to continue data migration from the point at which error occurred.
It provides a special migration object called Hyper-Object which can incorporate all interdependent migration objects and enable us to perform data migration of all these objects in a single go.
Basics of Data Migration
Page: 7/13
3
THE IS-U MIGRATION WORKBENCH CONCEPTS
3.1
The IS-U Migration Workbench Principle
IS-U migration is an open interface for transferring master data and transaction data to the R/3 application IS-U. The outflow is orientated around the target system R/3-IS-U. This means that the data model or the function of the legacy system for migration is neither required nor used. The migration procedure cannot run based on tables, because of the relational IS-U data model and the considerable complexities it contains. Instead it has to compile related data in such a way as to guarantee consistence in the data after the transfer. The data is compiled into so-called migration objects for the migration into the R/3 application. In these units the data is transferred into R/3 System. At the same time, special service function modules have been created in IS-U for the business objects of this application. They make an object orientated transfer possible. These modules operate in direct input mode and can thus avoid the performance disadvantages of batch input processes. The structure of a migration object is defined in controlling tables delivered by SAP. It can consist of several R/3 structures. You can find all the structures connected with a migration object in the dictionary. User-defined enhancements (fields) are taken into consideration at the same time. Data can be extracted from the legacy system using user-defined extraction programs. The exact record structure is determined per table settings in the IS-U migration workbench (transaction EMIGALL). Data sets that have been generated in this way, and transferred to your target computer by file transfer, are read and processed in the R/3 System by the import program, which is generated by the customer system. The code conversion (EBCDID to ASCII) is carried out at the same time. Following that, the data is formatted and transferred to the direct input module. To view the complete documentation of IS-U Migration Workbench use transaction EQ81 3.2
Mi gr at io n Ob jec t Def in it io n
A migration object is a logical business unit containing the data that is transferred to the R/3 System during IS-U migration A migration object is described by one or more structures. As a whole they are called Aut o data. The service function modules being used predetermine them; in the R/3 environment they are defined via the dictionary. For example, PARTNER object 3.3
Au tom at io n Dat a Def in it io n
Automatio n data specifies the quantity of all fields of a migration object, as well as its
structural order. The automation data is clearly predetermined by the interface of the function module being used. 3.4
Fi el d Rul e Def in it ion
In IS-U Migration, field rule denotes the unique rule basis, with which an individual automation data field is processed during import. Examples of field rules are:
Initial value (allocating the typical initial value) Supplying a constant value (Fixed value) Allocating a value transferred from the legacy system
Basics of Data Migration
Page: 8/13
3.5
Executing a series of ABAP statements (complex field rules) Mapping through a conversion table
Fi xed Val ue Def ini ti on
In IS-U migration, a fixed value is a definition of a generally used assignment of a constant value or table content. Examples of fixed values are:
The constant for the field company code (for example '0001') The constant for the application of contract accounts receivable and payable The current date (field content sy-date) The logon language (field content sy-langu)
A fixed value can be used in several Field Rules and is suitable for the uniform supply of fields with the same meaning. 3.6
Co nv er si on Ob jec t Def in it io n
In IS-U migration, a conversion object refers to the definition of a generally applicable modelling rule for a set of discrete input values to output values. The data category and field length of the input field can be set independently of the target field. Conversion objects can be used for: Mapping the industry key in the legacy system to the industry in IS-U
Converting the company code from RIVA (e.g. from "01" to "0010")
Mapping the values from a check table in the legacy system to the values in an IS-U check table
A conversion should be used w hen no clear fixed value can be set, and when creating an ABAP rule as a CASE distributor would take too long. A conversion object can be used in several field rules, and is therefore suitable for supplying fields with the same meaning. 3.7
Ot her Im po rt an t Term s Company
Migration customizing is a client independent application. The Company represents the project to which migration objects are allocated. In order to work in various clients with their own migration configurations, different companies have to be created. The required migration objects can then be copied for each company. The company SAP serves as a reference configuration. SAP delivers the i ndividual migration objects; they are overwritten when an IS-U upgrade takes place.
Migration class
The migration class defines the business object. A migration object is based on the business object. In this way you can, for example, give the SAP migration object ACCOUNT its own name when you copy it. Now, when you transport settings, you can still recognize that the operation involves the migration object ACCOUNT. Basics of Data Migration
Page: 9/13
Customer structure
Using migration customizing you can allocate certain processing rules to parts of the auto data. You can also allocate the rules to the auto data fields. In this way you can determine which fields must be available in the data records of the import file. These fields form the customer structure, which you can display via the menu Utilities - Display customer structure. You can display the customer structure in either list or layout form.
Key and st atus m anagement (KSM)
In order to structure the database-key relationship between individual objects, the IS-U key for the data object must be transferred within the data records. The KSM administrates the relationship between the legacy system and the IS-U key, so that the extraction program in the legacy system can work with the available keys. KSM is management of information about created data objects and their import status in ISU migration. This management is implemented using the TEMKSV database table. The following functions are implemented using the KSM:
Check whether a data record on the import file was already transferred in IS-U migration. Linking external keys (that is, keys from the legacy system) with keys created during import into IS-U.
These checks are carried out for every processed data object during the runtime of the import program. The newly imported data records are also recorded in the TEMKSV table. You can evaluate via table TEMKSV and make changes to external keys using the KSM maintenance tool in the IS-U migration workbench.
Event
The load report for a migration object is not delivered, but generated in the customer system. Here you have the opportunity to include your own coding at certain points in the report. These points, and the exits defined here, are called events.
Basics of Data Migration
Page: 10/13
List of Standard Migration obj ects(as of SAP ERP 2005 wit h IS-Utlit ies 600)
Migration Object ACCOUNT ACCOUNTCHA ACC_NOTE ADRCITY ADRCITYISU ADRPSTCODE ADRSTREET ADRSTRTISU AREA BANK BBP_CB_TAB BBP_EXT BBP_INT BBP_MULT BBP_PAYSC BCONTACT BCONT_NOTE BILLDOC BILLDOCEBF BILLTRIGCR BILLTRIGG CLEANOBJ CONNCTINST CONNECTION CONNOBJ CONSHIST CONSUMPT CONTAINER CONTGRP DEVGRP DEVICE DEVICEMOD DEVICERATE DEVICEREL DEVINFOREC DEVLOC DISC_CLOSE DISC_DOC DISC_ENTER DISC_EVENT DISC_ORDER DISC_RCENT DISC_RCORD Basics of Data Migration
Description Contract Account Contract Account Change Adding notes to contract account Create City Add IS-U data to City Create Postal Code Create Street Add IS-U data to Stree Area of a cleaning object - Waste Management Bank Master Collective document numbers for budget billing requests Transfer Payment Plans Create budget billing plans according to IS-U schedulling Create budget billing plans according to external schedulling Budget Billing Payment Scheme Transfer data to Customer Contacts Adding notes to Customer Contacts Transfer Billing documents with all billing line items Transfer EBF(easy bill correction framework) Billing documents Create billing orders for billing which has already taken place in legacy system (in case SAP billing schemas are very close to schemas in legacy system Execute billing for billing orders Cleaning Object - Waste Management Installing Connection(equipment) in a Connection Object(Functional location) Creating a Connection(Equipment) Create Connection Object Create partial billing documents to show consumption history Transfer data of Period Consumption in Installation structure Create Container(technical object equipment) - Waste Management Create Container Groups along with Containers Create Device Groups for devices Create Device (equipment) Modications of both Installed & Non-Installed devices Carry out Rate changes in the installation structure Carry out Device Allocations Create Device Info Record Create Device Location Closing Disconnection documents Create Disconnection document Enter Disconnection entries Create Disconnection workflows Create Disconnection order Create Reconnection entries Creating Reconnection order Page: 11/13
DISC_WF DOCUMENT DOC_DOUBT DOC_INT DUNNING FACTS FUNCLOC FUNCLOCCHA GOODSMVT HYPER INSTLN INSTLNCHA INSTPLAN INST_MGMT INTCASHDEP LOADPROF LOT LOTFINAL METERREAD MOVE_IN MOVE_IN_H MOVE_OUT NOTE_CON NOTE_DLC OBJCLASS OBJSTATUS OFFICEDOC PARTNER PARTNERCHA PART_REL PAYMENT POD PODCHANGE PODSERVICE PREMISE PREMISEWA PROFASSIGN PROFHEAD PROPERTY REFVALUES REGPOLIT REGRELSHIP REQUEST ROUTE SALES_STAT SECURITY Basics of Data Migration
Creating Disconnection workflows Transfer open items - receivables/credit memo Transfer Doubtful entries - Write Off for open items already migrated Transfer date of last interest calculation for open items already migrated Transfer dunning data & creditworthiness of business partners Transfer facts for installation Create Functional Location in PM Change data for Functional Locations CONNOBJ, DEVLOC, FUNCLOC Create material document for goods movement Template to create new hyper objects Create Installation Change Installation data Create Installment Plans for migrated open items Device Installation/Removal/Replacement Transfer Interest documents for payment of Cash Security deposits Create load profiles and allocating to Installation Create Data object sample lot for devices Transfer Completion data for creating sample lots Transfer Meter Readings Create Move-In Move-In and Move-out of historical contracts of legacy Create Move-Out Transfer Static note to field service in Connection object(For meter reader) Transfer Static note to field service in device location (For meter reader) Creating classification of technical objects Changing user status of technical objects and profiles Create SAPOffice documents linked to objects Create Business Partner Changes to Business Partner Create Business Partner relationships Transfer payment documents for open items Create Point of Delivery for Installation or Device Changes to Point of Delivery Creating object Point of Delivery Service Create Premise Extend IS-U-WA specific data for device Allocate profiles to registers Create Profile header data Property allocation like connection object, premise, installation to Business Partner Transferring information on reference values (assigned to installation) Transfer political regional structure data Transfer register relationships Create requests in FICA Create route master data object Transfer data on Sales statistics Create security deposit request Page: 12/13
SERVFREQ SM_NOTIF SM_NOTIFOK STRT_ROUTE TECHINST TECHOBINSP
Create Service frequency Create service notification Change status of service notification to "done" Create street routes Create technical installation Allocate last inspection date to device or technical installation
Hierarchy display of Migration Objects
The following diagram represents the individual m igration objects with their dependencies. Read from above to below, the diagram displays the chronological order of migration. Note: The migration objects to be used and their corresponding hierarchy differs from one project to another project depending upon the business scenarios involved.
| (1) ADRPSTCODE REGPOLIT | (2) BANK | | | |----------| | | ADRCITY-------| | | | | ADRCITYISU | | | ADRSTREET------| | | ADRSTRTISU | | ----------------------------------------------------------------------DEVICE CONNOBJ ------PARTNER -------| | | | | | | NOTE_CON | PART_REL | | | | -------- --------------------- -| | | | | | (5) | | | | | BCONTACT | PREMISE --| | ACCOUNT-- | | | | | | | | | BCONT_NOTE | | | | | | | DEVGRP | (3)--- DEVLOC -| | ---| | | | | | | | -------- ------ NOTE_DLC | ACC_NOTE | | | | | | | | | | INSTLN -------------- | -----------(7) | | | | | | | -------- --------- FACTS | | | PROPERTY | | REFVALUES | | | ---- INST_MGMT | | | | | (INSTALL) | LOADPROF | POD | | | | | METERREAD ----------- ---------------- | CONSUMPT (6) | | STRT_ROUTE REGRELSHIP
MOVE_IN |
PODSERVICE
| ------------------------------------------(4) | | | | BBP* DOCUMENT BILLDOC SECURITY | | | - - - - - ------------PAYMENT | | | PAYMENT INSTPLAN INTCASHDEP
Basics of Data Migration
Page: 13/13