2011 DATA STAGE
2/12/2011
DATA STAGE
Page 2
DATA STAGE
Page 2
DATA STAGE 1. Explain about Data stage Architecture? Datstage is an ETL Tool and it is client-server technology and integrated toolset used for designing, running, monitoring and administrating the “data acquisition” application is known as “job”. A job is graphical representation of dataflow from source to target and it is designed with source definitions and target definition and transformation Rules. The data stage software consists of client and server components
Data stage Designer Data stage Server Data stage Director
Data stage Manager
TCP/IP
Data stage Repository
Data stage Administrator Administrator
When I was installed Data stage software in our personal PC it’s automatically aut omatically comes in our PC is having 4 components in blue color like DATASTAGE ADMINISTRATOR, DATASTAGE DESIGNER, DATASTAGE DIRECTOR, DATASTAGE MANAGER. These are the client components. DS Client components:1) Data Stage Administrator :This components will be used for to perform create or delete the projects. , cleaning metadata stored in repository and install NLS.
Page 3
DATA STAGE 2) Data stage Manager:it will be used for to perform the following task like.. a) Create the table definitions. b) Metadata back-up and recovery can be performed. c) Create the customized components. 3) Data stage Director :-
It is used to validate, schedule, run and monitor the Data stage jobs. 4) Data stage Designer :It is used to create the Datstage application known as job. The following activities can be performed with designer window. a) Create the source definition. b) Create the target definition. c) Develop Transformation Rules d) Design Jobs. Data Stage Repository:It is one of the server side components which is defined to store the information about to build out Data Ware House. Data Stage Server:This is defined to execute the job while we are creating Data stage jobs.
2. What is a job? And Types of the Job? Ans:- Job is nothing nothing but it is ordered ordered series of individual individual stages which are linked together to describe the flow of data dat a from source and target. There are three types of jobs can be designed. a) Server jobs b) Parallel Jobs c) Mainframe Jobs 3. Have you work either parallel jobs or server jobs? Ans:- I had been working parallel job since 3+ years onwards… 4. What is difference between server jobs and parallel jobs? Ans:Server jobs:a) In server jobs it handles less volume of data with more performance. b) It is having less number of components. c) Data processing will be slow.
Page 4
DATA STAGE d) It‟s purely work on SMP (Symmetric Multi Processing). e) It is highly impact usage of transformer stage.
Parallel jobs:a) It handles high volume of data. b) It‟s work on parallel processing concepts. c) It applies parallism techniques. d) It follows MPP (Massively parallel Processing). e) It is having more number of components compared to server jobs. f) It‟s work on orchestrate framework 5. What is stage? Explain different various stages in data stage? Ans:- A stage defines a database, file and processing There are two types of stages. a) Built-in stages b) Plug-in stages. Built-in stages:- These stages defines the extraction, Transformation, and loading These are also divided into two types I. Passive stages:- which defines read and write access are known as passive stages. EX:- All Database stages in palette window by designer. II.
Active stages:- which defines the data transformation and filtering the data known as active stage. Ex:- All Processing Stages. 6. Explain parallism techniques? Ans:- it is a process to perform ETL task in parallel approach need to build the d ata warehouse. The parallel jobs support the following hardware system like SMP, MPP to achieve the parallism. There are two types of parallel parallism techniques. A. Pipeline parallism. B. Partition parallism.
Pipeline parallism.: - the data flow continuously throughout it pipeline . All stages in the job are Operating simultaneously. For example, my source is having 4 records as soon as first record starts processing, then all remaining records processing simultaneously.
Partition parallism: - in this parallism, the same job would effectively be run simultaneously by several processors. Each processors handles separate subset of total records. For example, my source is having 100 records and 4 partitions. The data will be equally partition across 4 partitions that mean the partitions will get 25 records. Whenever the first partition starts, the remaining three partitions start processing simultaneously and parallel.
7. What is configuration file? What is the use of this in data stage? It is normal text file. it is having the information about the processing and storage resources that are available for usage during parallel job execution. The default configuration file is having like a) Node:- it is logical processing unit which performs all ETL operations. b) Pools:- it is a collections of nodes. c) Fast Name: it is server name. by using this name it was executed our ETL jobs.
Page 5
DATA STAGE d) Resource disk:- it is permanent memory area which stores all Repository components. e) Resource Scratch disk:-it is temporary memory area where the staging operation will be performed. 8. What are the data partitioning techniques in data stage? In this data partitioning method the data splits into various partitions distribute across the processors. The data partitioning techniques are a) Auto b) Hash c) Modulus d) Random e) Range f) Round Robin g) Same The default partition technique is Auto.
Round Robin:- the first record goes to first processing node, second record goes to the second processing node and so on….. This method is useful for creating equal size of partition. Hash:- The records with the same values for the hash-key field given to the same processing node. Modulus:- This partition is based on k ey column module. This partition is similar to hash partition. Random:- The records are randomly distributed across all processing nodes. Range:- The related records are distributed across the one node . The range is specified based on key column. Auto:- This is most common method. The data stage determines the best partition method to use depending upon the type of stage.
9) Explain each and every File stages?
File stages Note:- All file stage are passive stages means which defines just to read or write access only.
Sequential File stage: it is one of the file stages which it can be used to reading the data from file or writing the data to file. It can support single input link or single output link and as well as reject link.
When I was go for properties of sequential file stage….
Page 6
DATA STAGE
Dataset:It is also one of the file stages which it can be used to store the data on internal format, it is related operating system. So, it will take less time to read or write the data.
When I was go for properties of dataset stage….
Page 7
DATA STAGE
File Set:It is also one of the file stage which it can be used to read or write the data on file set. The file it can be saved with the extension of “.fs”. it operating p arallel
When I was go for properties of file set…..
Page 8
DATA STAGE
10) What is exact difference between Dataset and File set? Dataset is an internal format of Data Stage the main points to be considered about dataset before using are: 1) It stores data in binary in the internal format of Data Stage so, it takes less time to read/write from dataset than any other source/target. 2) It preserves the partioning schemes so that you don't have to partition it a gain. 3) You cannot view data without data stage
Now, About File set 1)It stores data in the format similar to a sequential file. 2) Only advantage of using file set over a sequential file is "it preserves partioning scheme". 3) You can view the data but in the order def ined in partitioning scheme.
Page 9
DATA STAGE Q) Why we need datasets rather than sequential files? When you use sequential file as Source, at the time of Compilation it will convert to native format from ASCII.where as, when you go for using datasets... conversion is not required. Also, by default sequential files we be Processed in sequence only. Sequential files can accommodate up to 2GB only.sequentilal files does not support NULL values....all the above can me overcome using dataset Stage....but selection is depends on the Re quirement....suppose if you want to capture rejected data....in the case you need to use sequential file or file set stage.... Seq file is used Extract the from flat files and load into flat files and limit is 2GB Dataset is a intermediate stage and it has parallism when load data into dataset and it improve the performance.
Complex Flat File:This file is used to read the data form Mainframe file. By using CFF we can read ASCII or EBCDIC (Extended Binary coded Decimal Interchange Code) data. We can select the required columns and can omit the remaining. We can collect the rejects (bad formatted records) by setting the property reject to save (other options: continue fail). We can flatten the arrays (COBOL files).
When I was go for properties of CFF stage…
Page 10
DATA STAGE 11) Explain about various types of processing stages?
Processing Stages Aggregator stage:It is one the processing stage which it can be used to perform the summaries for the group of input data. It can support single input link which carries the input data and it can support single out put link which carries aggregated data to output link.
When I was go for properties for aggregator stage..Double click on aggregate stage then it will show…
Page 11
DATA STAGE
Copy stage:It is also one of the processing stages which it can be used just to copy the input data to number of output links. It can support single input link and number of output links.
When I was go for the properties of the copy stage. Double click on copy stage It will show like…
Page 12
DATA STAGE
Filter Stgae:it is also one of the processing stage which it can be used tooo perform the filter the data based on given condition. It can support single input l ink and „n‟ no of output links and optinally it support one reject link.
When I was go for propertief of filter stage. Double click on f ilter stage it will show…
Page 13
DATA STAGE
Switch stage:- it is also one of the processing stage which it can be used to filter the input data based on given conditions. It can support single input link and 128 output links .
When I was go for properties of switch stage.. double click on switch stage..
Page 14
DATA STAGE Q) What exact difference filter stage and switch stage? Both stages functionality and responsibilities is same. But the difference way of execution like.. In filter stage, we have to give the multiple conditions, on multiple columns. But every time data come form source system and filter the data and loads into target. In switch stage, we have to give the multiple conditions on single column, but all data come form source only once and check all the condition in the switch stage and loads into target.
Join stage:It is also one of the processing stages which can be used to combine two or more input datasets based on key field. It can support two or more input datasets and one output dataset, and doesn‟t support reject link. Join can be performing inner join, left outer, right outer, full-outer joins.
Inner join means to display the matched re cords from both the side tables. Left-outer join means to show the matched records from both sides as well as unmatched records from left side table. Right-outer join means to show the matched records from both sides as well as unmatched records from Right side table. Full-outer join means to show the matched as well as unmatched records from both sides. When I was go to the properties of join stage. Double click on join stage…
Page 15
DATA STAGE
Merge stage:It is also one of the processing stages which it can be used to merge the multiple input data. It can support multiple input links, the first input link is called “master input link” and remaining links are called “Updated links”. It can be perform inner join and left-outer join only.
Page 16
DATA STAGE
When I was go to the properties of the merge stage..Double click on merge stage…
Page 17
DATA STAGE Q ) On which case inner join perform and on which case left -outer join perform? In merge stage it is having one property is here. To see the above picture.. If Unmatched Master Mode= Drop then it will be perform inner join. If Unmatched Master Mode= keep then it will be perform left -outer join.
Look-up stage:This is also one of the processing stages which can be used to look -up on relational tables. It can support multiple input links and single output link and support single reject link. This is simple job for regarding on explanation of look-up stage…
This stage will be performing inner join and left-outer join. When I was go for properties of look-up stage. Double click on look-up stage, it will show like… Please look into below diagram
Page 18
DATA STAGE
Q) On which case it will be perform either inner or left-outer join? To see observe picture it is having one icon is – constraints. Double click on that icon it will show one window is look like…
Page 19
DATA STAGE It‟s having two options like: Condition not met. Lookup Failure. Default it is having fail. If the condition not met =“drop”, it will b e perform inner join. If the condition not met option = continue, it will be perform left-outer join. Default it having lookup Failure=Fail. If it is having continue option will be support Reject link. Q) What is main difference between join, merge and lookup? It‟s mainly differ from 1. Input requirements 2. Treatment of unmatched records 3. Memory usage 1. Input requirements :Join will be support two or more input links and single output link doesn‟t support reject link. In case of merge, it can support multiple input links and multiple output links and also support reject links same as updated links. in case of lookup, it supports multiple input links and single output link as well as one reject link. Join support 4 types of joins like inner, left-outer, right-outer, and full-outer. In case of merge it will support inner as well as left-outer only. In case of lookup, it will be support inner as well as left-outer. 2. Treatment of unmatched records:Join doesn‟t get any unmatched records because of doesn‟t support reject link. In case of merge, it doesn‟t catch the unmatched master records on mater link. Each and every update unmatched records go to corresponding update rejects links. In case of lookup, it catches the unmatched primary records only.
4. Memory usage:If the reference dataset is larger than physical memory then we can go for join stage for better performance. If the reference dataset is smaller than physical memory then it recommended to use lookup.
Page 20
DATA STAGE
Funnel stage:- it is also one of Active processing stage which can be used to combined the multiple input datasets into single output datasets. Note:- all the input datasets is having same structure.
When I was go for properties of funnel stage…
Remove Duplicate Stage:- it is also one of processing stage which it can be used to remove the duplicates data based on key field.
Page 21
DATA STAGE It can support single input link and single output link.
When I was double click on remove duplicate stage, it will show…
Sort stage:It is also one of the processing stage which can be used to sort data based on key field, either ascending order or descending order. It can be support single input li nk and single output link.
When I was go for properties of sort stage…
Page 22
DATA STAGE
Modify Stage:It is also one of the processing stages which it can be used to when you are able to handle Null handling and Data type changes. It is used to change the data types if the source contains the varchar and the target contains integer then we have to use this Modify Stage and we have to change according to the requirement. And we can do some modification in length also.
When I was go for properties of modify stage…
Page 23
DATA STAGE
Pivot stage:it is also one of the processing stage. Which can be used to make many people have the following misconceptions about Pivot stage? 1) It converts rows into columns 2) By using a pivot stage; we can convert 10 rows into 100 columns and 100 columns into 10 rows 3) You can add more points here!! Let me first tell you that a Pivot stage only CONVERTS COLUMNS INTO ROWS and nothing else. Some DS Professionals refer to this as NORMALIZATION. Another fact about the Pivot stage is that it's irreplaceable i.e. no other stage has this functionality of converting columns into rows!!! So, that makes it unique, doesn't!!! Let's cover how exactly it does it.... For example, lets take a file with the following fields: sno, sname, m1, m2, m3
Basically you would use a pivot stage when u need to convert those 3 fields like m1,m2,m3 into a single field marks which contains a unique value per row...i.e. You would need the following
Page 24
DATA STAGE output
When I was go for properties of pivot stage…..
Page 25
DATA STAGE
Surrogate key Generator: It is also one important stage on processing stage which it can be used to generate the sequence numbers while implementing slowly changing dimension. It is a system generated key on dimensional tables.
When I was go for properties of surrogate key generator stage…
Q) What is difference between primary key and surrogate key? Surrogate key is an artificial identifier for an entity. in Surrogate key are generated by the system sequentially. Primary key is a natural identifier for an entity. In primary key are all the values are entered manually by the are uniquely identifier there will be no repletion of data.
Transformer Stage:Page 26
DATA STAGE It is an active processing stage which allows filtering the data based on given condition and can derive new data definitions by dev eloping an expression. This stage uses Microsoft .net framework environment for it‟s compilation. The transformer stage can be performing data cleaning and d ata scrubbing operation. It can have single input link and number of output links and also reject link.
When I was go for properties of Transformer stage… v
In this Editor it is having stage variables, Derivations, and Constraints. Stage Variable - An intermediate processing variable that retains value during read and doesn‟t pass the value into target column.
Page 27
DATA STAGE Derivation - Expression that specifies value to be passed on to the target column. Constraints - Conditions that are either true or false that specifies flow of data with a link. Q) How can I define stage variables on Transformer Stage? When I was click stage properties on Transformer Stage, it will show one window look like…
When I was click on constraints icon it will show one window…
Q) What is order of the Execution in Transformer stage? Order of Execution is… 1) Stage Variables. 2) Derivations. 3) Constraints.
Change Capture Stage:This is also one of the active processing stage which it can be used to capture the changes between two sources like After and Before. The source which is used as reference to capture the change is called after dataset. The source which we are looking for the change is called before dataset. The change code will be added in output dataset. So, by this change code will be recognizing delete, insert or update.
Page 28
DATA STAGE
When I was go for properties of change capture stage..
Page 29
DATA STAGE
15) Explain Development and Debug stages?
Development and Debug Stages Row generator Stage:it produces set of data fitting specified meta data. It is useful where you want to test your job but have no real data available to process. It is having no input links and a single output link.
When I was go for properties of row generator stage…
Page 30
DATA STAGE
Column generator stage:This stage adds the columns to incoming data and generates mock data for these columns for each data row processed. It can have single input link and single output li nk.
When I was go for properties of column generator stage….
Page 31
DATA STAGE Sample job for column generator stage:-
Input data:-
Output data:-
Page 32
DATA STAGE
Head Stage:This stage helpful for testing .and debug the application with large datasets. T his stage selects TOP N rows from the input dataset and copies the selected rows to an output datasets. It can have a single input link and single output link.
When I was go for properties of head stage….
Tail Stage:This stage helpful for testing and debug the application with large datasets. This stage selects BOTTOM N rows from the input dataset and copies the selected rows to an output datasets. It can have a single input link and single output link.
Page 33
DATA STAGE When I was go for properties of tail stage…
Sample Stage:This stage will be having single input link and any number of output link when operating percent or period mode.
When I was go for properties of sample stage….
Page 34
DATA STAGE
Peek Stage:it can have a single input link and any number output link. It can be used to print the record column values to the job log view.
When I was go for properties of peek stage…
Page 35
DATA STAGE
Mock data:-
Page 36
DATA STAGE
21) Can you explain Type-2 implementation?
SCD type-2 is the common problems in DWH. It is to maintain the history information for particular organization in target. So, for every update in the source, it insert new record in target. In this implementation, it is having two input datasets like before and after datasets, these two are connected to change capture stage which is connected to transformer stage which is having two output links like insert link and update link. The insert link is connected to stored procedure stage which is connected to transformer which is connected to target stage. And also other output link (update link) of the transformer stage which is joined with the target stage while removing records by using remove duplicate stage. The output link of the join stage which is connected to transformer stage which is connected to target update stage. For example, the source is having EMP table with 100 records. When I was run the job, the records was initially loaded into target insert stage , how it means, First compare two input datasets, in first time there is no change in the records. So, the change capture stage gives the change code=1. The transformer stage transforms the records from source to target by generating sequence to the records by using stored procedure stage. If any updation is occurred at source, that updation records will be stored to target side (TGT_UPDATE). How it will be store means, first two compare two input datasets, changes is occurred at source level then change capture stage gives the change code=3 . by using this change code, the transformer stage transform the records to join stage through the update link. Join stage joins the updated records and target records by removing duplicate records using remove duplicate stage. The output of the join stage to connected to transform stage which was transforming update records to target update stage.
17) What is Environment Variables? Basically Environment variable is predefined variable those we can use while creating DS job. We create/declare these variables in DS A dministrator. While designing the job we set the properties for these variables. Environmental variables are also called as Global variables. There are two types variables are there. 1. Local Variables 2.Environmental variables/Global Variables
Page 37
DATA STAGE
Local Variables:- only for particular job only Env Variables:- In any job through out ur project in this some default variables r there and also we can define some user defined variables also. How means, Creating project specific environment variables- Start up Data Stage Administrator.Choose the project and click the "Properties" button.- On the General ta b click the "Environment..." button.- Click on the "User Defined" folder to see the list of job specific environment variables. Give me to you some example for environment variable. So that it will be more clear for us. Example is you want to connect to database you need use id , password and schema. These are constant through out the project so they w ill be created as environment variables. Use them where ever you are want with #Variable#. By using this if there is any change in password or schema no need to worry about all the jobs. Change it at the level of environment variable that will take care of all the jobs.
18) Explain Job parameters? There is an icon to go to Job parameters in the tool bar. Or you can press Ctrl+J to enter into Job Parameters dialog box. Once you enter give a parameter name and corresponding default value for it. This helps to enter the value when you run the job. Its not necessary always to open the job to change the parameter value. Also when the job runs through script its just enough to give t he parameter value in the command line of script. Else you have to change the value in the job compile and then run in the script. So its easy for the users to handle the jobs using parameters.
20) What is difference between version Data stage 7.5 and 8.0.1? Main differences b/w data stage 7.5.2 to 8.0.1 1. In data stage 7.5.2 we have manager as client. in 8.0.1 we don‟t have any manager client. the manager client is embedded in d esigner client. 2. In 7.5.2 quality stage has separate designer .in 8.0.1 quality stage is integrated in designer. 3. In 7.5.2 code and metadata is stored in file based system. in 8.0.1 code is a file based system where as metadata is stored in database. 4. In 7.5.2 we required operating system authentications. in 8.0.1 we require operating system authentications and data stage authentications. 5. In 7.5.2 we don‟t have range lookup. In8.0.1 we have range lookup. 6. In 7.5.2 a single join stage can't support multiple references. In 8.0.1 a single join stage can support Multiple references. 7. In 7.5.2, when a developer opens a particular job, and another developer wants to open the same job, that job can‟t be opened. in 8.0.1 it can be possible when a developer opens a particular job and another developer wants to open the same job then it can be opened as read only job.
Page 38
DATA STAGE 8. In 8.0.1 a compare utility is available to compare 2 jobs, one in development another is in production. in 7.5.2 it is not possible
9. In 8.0.1 quick find and advance find features are available, in 7.5.2 not available 10. In 7.5.2 first time one job is run and surrogate key s Generated from initial to n value. Next time the same job is compiled and run again surrogate key is generated from Initial to n. Automatic increment of surrogate key is not in 7.5.2. But in 8.0.1 surrogate key is incremented automatically. a state fiyle is used to store the maximum value of surrogate key.
Q) How did you handle Rejected Data? Reject-link is defined and reject data is loaded back into DWH. So reject link has to be defined every output link you wish to collect rejected data. Reject data typically a bad data like duplicates of primary keys or null-rows where data is expected.
Q) What are Routines and where/how are they written and have written any routine before? It didn‟t use Routines at any time in my project. But I know the routines. Routines are stored in the routine Branch of the DS Repository. Where you can create, view or Edit. The following different types of routines. 1) Transform Function 2) Before-after Sub routines 3) Job control routines.
Q) Explain METASTAGE in DS 8.0.1? It is used to handle the metadata which will be very useful for data lineage and data analysis later on. Meta data is type of data we are handling. These data definitions are stored in repository and can be accessed with the use of Meta stage.
Q) Do you know about INTEGRITY/QUALITY stage? Quality stage can be integrated with data stage. In quality stage we have many stages like investigate, match, survivorship, like that we can do the quality related works and we can integrate with the data stage we need quality stage plug-in to achieve the task.
Q) How can I call the Store procedure in data stage? There is a stage named Stored Procedure available in Data stage palette under Database category. You can use that stage to call your procedure in Data stage jobs. Q) What is job control? How it is developed? Explain with steps?
Controlling Data stage jobs through some other Data stage jobs. Ex: Consider two Jobs XXX and YYY. The Job YYY can be executed from Job XXX by using Data stage macros in Routines. To Execute one job from other job, following steps needs to be followed in Routines. 1. Attach job using DSAttachjob function. 2. Run the other job using DSRunjob function 3. Stop the job using DSStopJob function
Page 39
