Ab Initio Interview Questions 1. A.
What is Override Key? If you set the override-key parameter for a particular port, then the input on that port must be sorted according to the override-key parameter for that port. In the join component if we are joining two I/P tables has different names as a primary key for that we will specify one key as a main key from one table and other key in other table we will specify as an override key.
1A. A.
What is surrogate key? Alternative name for the natural key. We can create this surrogate key in slowly changing dimensions. This is Type 2 functionality. Ex: - To maintain historical information and current information.
2. A.
What is Maxcore? Maxcore is Maximum amount of memory used by the component at the run time. Default values Sort-10MB and Join - 64MB
2A A
Explain what max-core's are used for and what you need to look out for, when setting them? maximum amount of memory in bytes a component will use, Too Low: will swap to disk, slow down application, Too high: consume too many resources, slow down.
2B. A.
What are the components contain Max Core values? Join, Scan, Sort, Rollup
2C. A
When it spills to disk, where the data is stored by default? \Temp directory.
2D. A
With an in memory join, what needs to fit into max-core? all non-driving inputs + overhead for hash table.
2E. A
With an in memory sort, what needs to fit into max-core? 3 times the size of the input data, recommended max. 100MB.
3. A.
What is Outer join? Outer join - sets the record-required parameters for all ports to false.
4. A.
What is not key based partition component? Broadcast, Partition By Load Balance, Partition By Round Robin, Partition By Percentage, Partition By Expression.
5. A.
What is not key based departition component? Concatenate, Gather.
5A. A.
What is the use of the partition components in graphs? For Data Parallelism.
6. A.
By using concatenate component, Deadlock occurs T/F? True. To avoid dead lock causes Auto Buffer.
7. A.
How to change the field name without changing value? Redefine Component.
8. What is Condition component? A. Reusable code 9. A.
What is check order component? To check the order of the data
10A. How can u copy the multi file? A. m_cp 10B. A.
How many multi file commands u know? m_mkdir, m_mkfs, m_cp, m_expand, m_mv, m_eval
10C. How do you kill a Running Ab Initio process? A. m_kill process_id 11. A.
What command is used to evaluate DML? m_eval, m_dump
12. A.
What command is used to create a multifile system? m_mkfs. Syntax: m_mkfs ctl file1 file2 file3 file4.
13. A.
Which one do you use first? Either a Sort or Partition component? Sort is used after partition is done
14. A.
What information exists in Control file? Partition file reference. Control file has the paths where the multifiles are located.
15.
Difference between phases and checkpoints? Phase is a stage of graph that runs to completion before that start of the next stage. By dividing a graph into phases, u can save resources avoid deadlocks and safeguard against failures. Phase:- We can divide the graph into number of phases. After successful of the Phase1, Phase2 will run. A checkpoint is a phase that acts as an intermediate stopping point in a graph to safeguard against failures. By assigning phases with checkpoints to a graph can recover completed stages of the graph if failure occurs.
Check point: - If we keep Checkpoint in the graph, we can restart the process whenever any system failures occur. 15. A
Explain the concept of phases in Ab Initio and what they can be used for? First phase must complete before second runs; can save concurrent resource usage; saves status after each phase if checkpoint.
16. A.
What directory is used at the time of running a graph? Ab Initio work directory
17. A.
Difference between Scan and Rollup? Scan: Multiple I/P flows and gives multiple O/P’s flows and gives the Cumulative Running totals depends upon key specifier. Roll Up: Multiple I/P flows and gives the single output for each group depends Upon key specifier.
18. A.
Compound Data types? Vector, Record Types, Union
19.
What is Partition by key and sort components? Repartition.
20.
What is the difference between Partition by Round-robin and Partition with Load Balance? Partition by Round-robin: Distributes data records evenly to each output flow in round-robin fashion. Partition with Load Balance: Distributes data records to output flow partitions, Writing more records to the flow partitions that consume records faster.
A.
21 A
What component is generally used for expanding processing from serial to parallel? partition
22 A
If you want to do a join in parallel, which partition component would you use? partition by key
23 A
What is the layout tab of a component used for in Ab Initio? where component runs AND number of ways parallel
24 A 25 A
If you had an ad-hoc multifile of 100 files, and you wanted to run only 4 ways parallel, what would you do? concatenate or custom component. With in a graph, how would you take a 4 way parallel stream to 8 ways, what component would you use? Repartition using partition ->> gather, or partition -> fan-in component.
25A. A.
If we are creating multifile, what files are created? 1. Control files. 2. Data Files (Serial files) (Partition files).
26 A
What are include files used for? It allows sharing of functions and named types across multiple transforms.
27 A
Some things to look for when tuning a graph for performance? skew, unnecessary sorts, max-cores, ways parallel etc.
28 A
Explain skew, and how it affects an applications performance? If skew is bad, then bottleneck on certain partitions, as spread of data among partitions uneven.
29. A.
What is look up file? Look up is a Single file. It contains small information. Look up file is not used for Sorting. Look up file is looking for the data.
29A. A.
What is lookup_local? It is multi file. By using this we can check for the multi files in the local machine.
29B. A
Explain difference between lookup and lookup_local? when lookup file is multifile, only local partition is examined. The input data must be partitioned on same key as lookup.
30 A
What files are the database configuration parameters stored in? dbconfig(.dbc)
31 A
If a job fails, how do you rollback to the last successful checkpoint manually? m_rollback
38. A
Explain the difference between local and formal parameters? local= static, formal = dynamic at runtime.
39. A.
Where we can store the Temporary files? Temp directory.
42. A.
Compress is not working in Windows Environment? (True/False) False.
43. A.
What version of Ab Initio did u used? GDE: - 1.13.4, 1.12.9, 1,12.5, 1.11, 1.10.11, Co-Operating System: - 2.12, 2.11, 2.10.11
44.
What is the first component did u use in your group?
A.
I/P Table component (or) Reformat is used depends on the graph.
45. A.
How Many graphs did u develop in your last project? 15
46. A.
Tell me the functionality of your graph?
47. A.
What are the sources of data? Oracle, Sybase, DB2, Flat files etc...
48. A.
What are the problems did u got while creating graphs?
49. A.
Did u mainly involve in Back-end or front-end? Back-end.
50. A.
What are layouts? List of host and directory files. Layouts contain URL’s of multifiles.
51. A.
Suppose if u have 50,000 records in I/P table and if u wants to test 10 records how can u test it? Using Filter By Expression. Next_in_sequence () < 10
52. A.
How to join two tables? Using Join component with common column name.
53. A.
How many tables we can join at the time? 18 (or) 20 tables.
54. A.
Tell me about parallelism? 1. Component parallelism 2. Data parallelism 3. Pipe parallelism.
55.
You handed a process written in Ab Initio, users are complaining that it runs Slowly. Out line strategy for improving the performance? A. 1. Parallelism 2. Sort in memory. 3. Spilling to disk. 4. Carrying around unnecessary columns. 56. A.
What r the folders available in Sand Box? (.DML),(.XFR),(.DB),(.MPC),(.MP),(.DAT),(.MDC),(.DBC),(.CFG),(.MFS)
56A.
Explain the concept of a sandbox and what it is used for?
A
group associated graphs and files in single directory, where user works (or) Share parameters
56B. A.
What is the difference between a sandbox and the graph parameters? Sand box parameters are global and can be accessed into any graph for particular user. Graph parameters are local to the graph and cannot be accessed into other graphs.
57. How many processors did u use in your last project? A. 6 58. A.
How can a rollup replace a sort and dedup, when can it does so? Rollup implicitly does a unique sort. If you care which of the duplicates are kept, You probably cannot use a rollup to replace a sort and dedup.
59. A.
When u runs an Ab Initio graph when does the .rec file get deleted? It is deleted after the graph runs but before the end script runs.
60. A.
Does a join of two sorted data streams preserve their respective sort order? Some times. If the flows are already sorted and are sorted on the same key the join retains the sort automatically. If the flows are not pre-sorted you have the choice to maintain the sort order or not.
61.
What are the difference between Sort & Sort With in Groups components, is the output the same? Is the performance the same? Sort: It simply sort (By default ascending). Sort With in groups: It will sort with in the groups using the minor key. But output of sort & sort with in groups is not same. The result is the same but the performance of the later is quicker because you are sub sorting the already sorted data.
A.
62. A.
When deciding upon a partitioning key what reflects a wise choice? Even or nearly even data distribution among partitions denotes a good partitioning key.
64. A.
Have you ever used the repository?
65. A.
Multifile Unix commands? M_ls -l: - lists all the multi files. M_dump: M_expand: - lists the location of the partitions.
66.
What is EME? Why is it used?
A.
EME: Enterprise Meta Environment. EME is a high-performance object-oriented Storage system that inventories and manages various kinds of information Associated with Ab Initio applications. It provides storage for all aspects of data Processing system from design information to operations data.
67. A.
We can run the graphs from GDE? How can I do it with out GDE? By deploying the graph as Korn shell script.
68. A.
Can you execute the graph more than once at the same time? How? Yes. By setting .rec file.
69. A.
How do you aggregate summary records? By using rollup component. Depending on the size of the data we use sorted input Or unsorted input.
70. A.
Which one does you use first sort or Rollup? We use sort first and then Rollup.
71. A.
Explain about your last project and its environment?
72. A.
Did you ever use multiway processing? Adv of Parellel mfs over Serial? Parallel Processing. The data is divided into patitions.
73. A.
Explain about Normalize component? Normalize generates multiple output records from each input record. Normalize Can separate a data records with a vector field into several individual data records, each containing one element of the vector.
74. A.
Co-Op is installed on two servers A and B, graph is running on A, How can I rename the graph on server B? By using Run program. Issue mv command.
75. A.
How do you communicate between two servers? By Node name and SSH keys.
76. A.
What is .rc in Ab Initio? What does it contain? Recovery. It contains the recovered data.
77. After running the graph in GDE, what file is created in sand box? A. .rec file. 78. A.
I n a Reformat component how do you set the parameters if I have 1 input and 2 output files?
79. A.
What are the databases you used most of the time? Oracle, Teradata.
80. A.
What component do you use to load data into Oracle database? Output table component.
81. A.
What is the main parameter in Output Table component? Commit table parameter.
82. A.
What is the advantage of using commit? For data recovery, you can rollback to the previous commit.
82A. What is an Inner join and a Outer join? A. Inner join is to get the matching records and the outer join is to get the matching Plus Non matching records by putting “null”, if the value does not exist. 83. A.
What type of loader you generally use? SQL * Loader.
84.
Which one do you prefer among Join and Lookup file if I have two inputs, 1 with 100000 and 2 with 5000 records? If it less records go for Lookup file.
A. 85. A.
How do you access your system remotely? ftp, rlogin, scp, ftp transfers files to and from a remote network site. rlogin is for Remote login with in the network.
86. A.
Difference between graph and component? A graph
87. A.
What is Abinitio standard environment?
88. A.
What are is_valid, is_error, is_digit, is_defined, string_* functions?
89. A.
Do you have understanding of multifile, can you have them in windows env?
90.
There are two datasets A with 100 million records and B with 50,000 records data is not sorted .you have to join them what component would you use? How can you modify?
A. 91.
In a data stream there is a field from 1-9 what component will you use?
A. 92. A.
How will you optimize SQL code in Abinitio?
93. A.
Abnitio vector to be?
94. A.
Diff. between Scan & Rollup?
95. A.
Explain some string function? lpad.
96. A.
How to Increase Performance?
97. A.
What is the difference between hash Partitioning and Time series partitioning?
98. A.
How do we execute the work scripts from abinito tool?
99. A.
What is mvs?
100. How do you design the objects using Ab Initio? A. 101. What are the type of data files we have loaded and how we loaded it? A. 102. Types of partitioning and how do we do partitioning? A. 103. How to schedule the job load and how much time it will take to load a 10gb data file and what type of parallelism? A. 104. What is start and end script? A. 105. What is run? A. 106. How do you Recover files manually?
A. 107. You can join two tables using Join key word in SQL? A. 108. Have you wrote any Packages? Where do you write? A. Yes, User defined functions in Package editor , We wrote error handling functions in package editor , and those are included using ~ (tilde) package name , error handling functions are used to handle the errors. 109. What is Conditional DML? A. 110. What are Multistage Components? A. Rollup, Scan, Normalize, Denormalize, Reformat. 111.
A.
What are Multistages? There are 5 Stages Input select Initialization Transformation Finalization Output selection
112. What is parallelism in Ab initio? A. 113. Advantage of Ab initio? A. 114. How to create a Multifile? A. M_touch. 115. How to crate a Multifile System? A. 116.How do you get environment variables in Ab initio? A. M_env 117. How do you executive SQL Statements in Ab initio? A. Run Execute component. 118. How do you execute UNIX commands? A. Run Program Component. 119. Explain about partition , departion ,normalization, denormalization
componets? A. 120. What is the use of lookup file? A. 121. Why we use intermediate files? A. 122. what is dedup ? A. 123. What is Transformation editor? A. Statement, Variable, Business Rules. 124. Which component discards records? A. Trash component. 125. Give some examples of a data flow? A. Fan-flow, parallel-flow, all-to-all flow, multiplex flow. 126. Which of these is an example of a partition component: Partition by field, broadcast, partition by division, replicate? A. Partition by field, broadcast. 127. What parameter specifies the memory size for the sort component? A. Maxcore. 128. Which component does not order records by flow or key? A. Gather, concatenate. 129. What is the method to create user-defined functions that the validate component can use to verify data? A. is_valid function prefix syntax. 130. With in an include statement, what does the ~ {tide} character do? Does it indicate that the given include file: is relative to the local sandbox xfr directory A. Yes. 131. In a component MPC file, what does the image line specify? a) The location of the script or program to execute. b) The label of the component when displayed in a GDE graph. c) The icon used when displayed in the GDE component library. d) The argument list passed to the unitool launcher. e) None of the above. A.
132. How do you describe the characteristic of the driving input for the join component? A. The records are stored in memory prior to executing the join transform. 133. Which action will cause the current partitioning keys to become invalid? a) Multiplying partition keys by a constant b) Joining using non grouped input with fewer keys than Partition by. c) Using rollup with grouped input with fewer keys than Partition by. d) Gathering(2) 8way multi-files into a single 8way multi-file. e) All the above. A. 134. What environment variable can be modified to alter the format of monitoring reports? a) XX_REPORT b) XX_DEBUG c) AB_CONFIG d) XX_MONITOR e) None of tht above. A. None of the above 135. What is the component that does not force a phase break? A. Intermediate file. 136. Which of the following is not a valid DECIMAL format? a) Left blank padded number b) Left zero padded number. c) Number with implicit decimal point. d) Right blank padded number. e) None of the above. A. Right blank padded number.
1) How do you identify duplicates? 2) There are two tables A and B, How can you get all rows in A but not in B? 3) How can you optimize your code in SQL? 1.
How can you run a process in the back ground?
A.
By appending “&” to the command or by using “bg” or “nohup”.
2. A.
How can you bring the job running in back ground to fore ground? By typing “fg”.
3. A
What is “awk”? Which one do you prefer “cut” or “awk”? awk options filename: Scan for patterns in a file and process the results. Cut options filename: Cut specified fields/characters from lines in a file. Awk can cut the fields from any place where as cut can do it once.
4.
2. What is is_valid(),is_digit..etc and is_defined() does 3. Recovery (Manual) 4. Diff. between Scan & Rollup 5. Explain some string functions ? lpad. 6. How to Increase Performance ? 7. What is the difference between hash Partitioning and Time series partitioning?. 8. How do we execute the work scripts from abinito tool?. 9. What is mvs? 10. How do you design the objects using Ab Initio? 11. What are the type of data files we have loaded and how we loaded it? 12. Types of partitioning and how do we do partitioning?. 13. How to schedule the job load and how much time it will take to load a 10gb data file and what type of parallelism? 14. What is start and end script? 15. What is run? gather – gather collects records from many sources, reads data from flow partition. It reduces data parallelism, reduces pipeline parallelism And it doesn’t support default record assignment LocalMerge – reads data from many sorted sources and maintains the sort order concat it takes multiple streams of data and append then one after another, it maintains the order….. Interleave
It collects the records from many sources in round robin fashion. It reads block size of records from first partition Partition by Load Balancing…..To balance uneven processing capability…. Partition by key and sort……all records with same key are in same partition………Local Lookup……. Partition by round robin ……………. Distributed data evenly across the out partitions..reads as chunks HashPartition…………….Reads records arbitary order from the input and distributes them to flow partition.. CheckOrder ----------- to verify that data sorted according to your specification…. Transformer Aggregate – generates summary records for group of input records……… Dedup – removes the duplicate records – suppress the duplicate records……… De-Normalize --- to group multiple records into single out (Praveen,kumar,thadakamalla – praveenkumarthadakamalla) having the same address with multiple people…… Normalize - use one record into multiple records………….house to people…………… Merge to co-relate the data – from different sources……….reads records from multiple input ports………..operates on the records……….with merging keys….. Merge-Join – to perform inner, outer and semi join in the form of relational database. Reformat…………used to change the record format of your data… Rollup – used to generates the summary records for groups of input records……….requires sorted input…. basically it reduces each group into a single out put record…………..finalize print one time…Year to Date Scan --- finalize prints every time………..multistage transform---for a series of summary records for groups of input records... produces intermediate summary records… Performance --- check points…. phases…. dead lock…memory release…Local Look up…before joiner put phase…so lots of memory gets released…
Layout ……….location of URL…………specifies how some part of an application is partitioned. Eg: component’s layout specifies the number and location of its partitions, giving a hostname and pathname for each partition. Every component of an application has a layout (even for one partition). mkfs………..multi file system…$mpjrect….0…for success…..for check the graph is successful m_mkfs --- for creating multi file M_ls m_cp or m_dump or m_rollback, are co>op system shell-level utilities. (For managing Parallel files, managing metadata, recovering a check pointed process) Mp commands – components run with mp commands – ab intio command interpreter eg: mp command-name argu1 argu2 mp job – establish the “frame work” mp ifile – defines data file components mp metadata define metadata that describes data in flows.. mp hash-partition – define program components… mp run – for running the application…… Skew – monitoring……at the user requests….the co-op system monitor ab-initio jobs and issue periodic reports…….monitor is control either two ways………. Shell --- set the confiigration variable xx_report….before running the job…. With in the script supply arguments to the report option the mp run command……the two interfaces accepts the same set of key word arguments…if both interfaces are use……the effect addidive….in summary the key words…are verbose error, expanded graph…….flows……..times………skew, skew = n, scroll = mode….file = filename Interval = n….table flows……… The two most used values are flows and times………. The value flows enables monitoring of all data flows………… The value time enables the monitoring of all process………….. Additional are for reporting……..characteristics…….the value xx_report are mp run report….. series of space separated keywords………… For export xx_report = flows time interval = 10 Basic mp run xx_report……...data bytes………bytes transmitted or received on the flow….records…….records..transmitted received on the flow… Un opened/ % by which the amounts of data in the flow partitions are skewed
0% - all partitions have equal data values 100% - means some partitions…has all the data……… charactertistics of ab-intio job……..report………………. Layout ……..the number and locations of the partition of the components are described as a layout you specify a layout for two reasons… create a multi file system a place where parallel files are stored… construct a layout object used to describe the parallelism of a component program in a graph application. A layout is a list of hostname/pathname pairs…each entry in the list represents one partition of multi file or a program component…..layouts used derived from layout of parallel files……..program layouts and remote connection……………..dataset/program as a layout……defines...location…(hostname+pathname) …depend…….on use…we go for single layout or multi………. Recovery Abnormal termination……….completed job means …job started….and successful.. Software error – c0>op will take care… Native problem ….don’t investigate…go to native… Jobname.rec………..host on every node ..contains…set of pointers to the log files on every node Log files……………..start/end………..hostsystem…variable/ab-initio/host/unique-id… sequence character. Automatic…………..software………………..temporary files……………….kill all the process.. Investigating…………recapsulation………. Restore the earlier system…………….shut down………performed in the……..intermediate…..get the job running M_rollback(-d,-I, -h)…………manually -d deletes job with recovery files/log files
-I display the state of the job and prompt the user whether the job should be deleted jobs at first point will rollback……. M_rollback my-job.rec Action of m_rollback for a job with no checkpoints……m_rollback myjob.rec
Xx_nice..xx_timeout……..xx_interval..ab_connection/_script Ab_home.ab_password..ab_nodes When running applications, please note the environment variables are passed downward only. /usr/local/abinitio export AB_HOME=/USR/LOCAL/ABINITIO export PATH = $AB_HOME/BIN:$PATH The above settings enable your shell to locate the installed Ab Initio Software.
A parallel file is called multi-file system. Multifiles are stored in parallel directories called multidirectories, which reside in a multifile system. URL: protocal://hostname/pathname Mfile: for multifile File: for serial file A multidirectory : mfile://pluto.us.com/usr/ed/mfs1/dat A multifile: mfile://pluto.us.com/usr/ed/mfs1/dat/s95/new.dat m_ls –1 mfs Drwxrwxr-x D owner group 512 may 29 17:39 dir -rw-rw-r-- M owner group 214 may 29 17:03 5% out.dat where D and M are multidirectory and multifile. 5% is the file skew, computed on the no. of bytes in each partition. The concept of skew refers to an unbalanced load among the partitions of a multifle or among the partitions of a dataflow. Eg: For a particular flow or file, they are k partitions. There are total bytes for all partitions. Then average = total/k. Average = 1000/20 = 50.
Then, the skew for a partition with n bytes is (n-average)/max ( -100 - 0 - 100%) By the way the sum of all the skews is 0%. Metadata – describes data formats and computations. m_dump produces a human-readable report that shows how input data is interpreted by Ab Initio metadata. Like m_dump foo.dml foo.dat (prints data in foo.dat as interpreted by the metadata in foo.dml) Mp job (takes care of configure and the execution environment, checkpointing, termination, recovery, monitoring, debugging) m_attach to facilitate remote startup on large parallel systems. m_env display the current settings of the ab initio. Mp_checkpoint insert in between the two transform components . In event of failure the application can restart from the most recent checkpoint instead of from the beginning Incase of software error or user control-c command…the co>op takes care of automatic rollback, thus restoring all files, flows, and processes to their initial state or to their state at the most recent checkpoint. When a job does not complete normally, it leaves a file in the working directory on the host system with the name jobname.rec……located at /var/abinitio/vnode/unique-id Analysing a database table Once a prototype configuration file is created for the database, each table must be analyzed with db_config (analyzes a table to determine ..column names and types, the applicable nodes, and the best scheme for loading or unloading it) It generates couple of files load, unload, config file Output_prefix.dml Output_prefix.dbml Output_prefix.cfg Output_prefix.unload Output_prefix.load Database Components
DB Unload DB Load DB Truncate SQLrun – run miscellaneous SQL against the database M_db_env – prints the database environment information Db_layout for layout of the table db_layout foo.cfg Unload.dml Record String(“,”) name; Decimal(“,”) age; End; Reformat.dml Record String (10) name=””; Decimal(3) age = “”; End;
