tag:blogger.com,1999:blog-66828611730338156322024-03-07T21:23:58.849-08:00Data Ingestion Framework for HadoopProden Technologieshttp://www.blogger.com/profile/08742796894235273079noreply@blogger.comBlogger1125tag:blogger.com,1999:blog-6682861173033815632.post-64088364471417391632017-06-15T00:36:00.001-07:002017-07-28T09:45:53.496-07:00Data Ingestion Framework for Hadoop<div dir="ltr" style="text-align: left;" trbidi="on">
<html>
<head><title>data ingestion tool for hadoop</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
</head>
<style>
.exceutescreen{
-webkit-column-count: 2; /* Chrome, Safari, Opera */
-moz-column-count: 2; /* Firefox */
column-count: 2;
-webkit-column-gap: 20px;
-moz-column-gap: 20px;
}
.scrollimg{
height: 30%;
width: 20%;
}
.scrollToTop{
width:10%;
height:10%;
padding:10px;
text-align:center;
font-weight: bold;
color: #444;
text-decoration: none;
position:fixed;
bottom:100px;
right:-50px;
display:none;
background: url('') no-repeat 0px 20px;
}
.scrollToTop:hover{
text-decoration:none;
}
.hdrange
{
font-family: calibri;
color :grey;
line-height: 26px;
margin-left: 50px;
}
.secrange
{
font-family: calibri;
margin: 0px;
color :white;
align:left;
font-size: 220%;
line-height: 50px;
}
.Thirdrange
{
font-family: calibri;
font-size: 110%;
color:gray;
display:block;
}
.Thirdrange:hover
{
font-family: calibri;
font-size: 110%;
color:#60a628;
display:block;
text-decoration: underline;
}
.boxbb
{
background: #ffffff;
float: left;
width: 100%;
padding: 50px;
margin: 2px;
height: 143px;
}
.link{
color: green;
text-decoration: none;
font-size:18px;
font-family: calibri;
}
.link:hover
{
color:orange;
}
.link1{
color: green;
margin-top:10px;
text-decoration: none;
font-size:18px;
font-family: calibri;
}
.link1:hover
{
color:orange;
}
.justifyfontEX{
text-align: justify;
}
.trialcontain{
background: #E98019;
text-align:center;
text-align: justify;
width:100%;
}
.trial-boxtxt{
font-size: 500%;
font-family: calibri;
color:#ffffff;
height:30px;
}
.box{
text-align: justify;
text-justify: inter-word;
}
.trial-sucesstext
{
color:white;
text-align: center;
}
.imageshadow {
opacity: 1;
filter: alpha(opacity=50); /* For IE8 and earlier */
box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);
}
.containimg {
position: relative;
width: 98%;
}
.image {
display: block;
width: 100%;
height: auto;
}
.overlay {
position: absolute;
bottom: 0;
left: 0;
right: 0;
background-color: transperent;
overflow: hidden;
width: 100%;
height: 0;
transition: .5s ease;
}
.containimg:hover .overlay {
height: 100%;
}
.text {
white-space: nowrap;
color: #FF9025;
background-color:transperent;
font-size: 25px;
position: absolute;
overflow: hidden;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
-ms-transform: translate(-50%, -50%);
}
.grow { transition: all .2s ease-in-out; }
.grow:hover { transform: scale(1.01); }
.notestyle{
background-color:#C0E3F1;
text-align: justify;
text-justify: inter-word;
color:black;
border: 1px solid white;
border-radius: 5px;
padding-top:10px;
padding-right:15px;
padding-left:15px;
padding-bottom:10px;
}
.syntax_notestyle{
border:1px solid lightgray;
color:black;
padding-top:10px;
padding-right:15px;
padding-left:15px;
padding-bottom:10px;
}
.tablecss {
border: 1px solid gray;
text-align: left;
padding: 8px;
}
table {
font-family: arial, sans-serif;
border-collapse: collapse;
width: 50px;
}
.topic_bordershadow{
box-shadow: 10px 4px 8px 0 rgba(0, 0, 0, 0.2), 10px 6px 20px 0 rgba(0, 0, 0, 0.19);
}
.shellimg{
width:100%;
}
.blackfont{
color:black;
}
.notecss
{
background-color:white;
text-align: justify;
text-justify: inter-word;
color:black;
border: 1px solid #7AA57B;
border-radius: 5px;
padding-top:10px;
padding-right:15px;
padding-left:15px;
padding-bottom:10px;
}
.tableborder{
border-collapse: collapse;
border: 1px solid #E4E4E4;
width:100%;
}
th, td {
border: 1px solid #E4E4E4;
}
</style>
<script>
$(document).ready(function(){
//Check to see if the window is top if not then display button
$(window).scroll(function(){
if ($(this).scrollTop() > 100) {
$('.scrollToTop').fadeIn();
} else {
$('.scrollToTop').fadeOut();
}
});
//Click event to scroll to top
$('.scrollToTop').click(function(){
$('html, body').animate({scrollTop : 0},800);
return false;
});
});
</script>
<body>
<div>
<br />
<div class="shellimg">
<img src="http://prodentechnologies.net/images/DataIngestionFramework.png" style="background-color: transparent;" />
</div>
<div style="height: 5px;">
</div>
<div class="notestyle">
<h1 style="text-align: left;">
Data Ingestion Framework for Hadoop</h1>
<p>
This version of the Data Ingestion Framework is a Script Engine you can use to ingest data from any database, data files (both fixed width and delimited) into Hadoop environment. This lets you get started, ingest the data in a matter of days.
</br>
If you are looking for a solution beyond Data Ingestion, please take a look at accel-DS for Data Integration. This solution let you ingest, clean and Transform data from a variety of data sources into Hadoop and vice versa.
</p>
</div>
<div style="margin-top: 3px;">
<h1>
Objective</h1>
<p>
To provide a simple, easy to use the framework to ingest data into Hadoop from a variety of data sources.
</p>
<div>
<h1 >
Framework</h1>
</div>
<p>
This is a set of shell script to which you can pass various parameters, such as source database or files details, target (Hadoop) details, target table name etc., </br>
This framework has a very small footprint and you can start ingesting data from day one.
</p>
<div>
<h1 >
Benefits</h1>
</div>
<div>
<ol style="text-align: left;">
<li>Ingest from a variety of data sources - database, data files (both fixed width and delimited)</li>
<li>Target Tables, Data Types are created by the Framework</li>
<li>Load multiple data files with a single call to the engine.</li>
</ol>
</div>
<div>
</div>
<h1>
How to Ingest Data</h1>
<div>
<ol style="text-align: left;">
<li>Download the Data Ingestion Framework.</li>
<li>Follow the instructions to copy it to your Hadoop environment.</li>
<li>Change Directory to the location where you have copied the scripts</li>
<li>Use any of the Data Ingestion commands listed under Sample Scripts section, to ingest data into Hadoop.</li>
</ol>
</div>
<div class="notestyle">
<h3>Note</h3>
<div style="text-align: left;">
Ensure Sqoop is installed and configured correctly. </div>
<div style="text-align: left;">
Use bash shell to execute the shell scripts in this Framework.</div>
</div>
<div>
<div>
<h2 style="text-align: left;">
Sample Scripts</h2>
<div style="height: 5px;">
</div>
<h4 style="text-align: left;">
In this article, sample scripts are provided for the following scenarios:</h4>
</div>
<div>
<ol >
<li>Create and Insert - Delimited File.</li>
<li>Create and Insert - Fixed width File.</li>
<li>Create and Insert - Table.</li>
<li>Create and Insert - SQL.</li>
<li>Create and Insert - XML File.</li>
<li>Insert only - Delimited File.</li>
<li>Insert only - Fixed Width File.</li>
<li>Insert only - Table.</li>
<li>Insert only - Query.</li>
<li>Insert only - XML File.</li>
</ol>
</div>
</div>
<div class="syntax_notestyle">
<h1>
1. Create and Insert - Delimited File</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create".Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Type_Of_Create</td>
<td>Options "External, Managed". If External is chosen, then the table created will be EXTERNAL HIVE table. #If Managed is chosen, then the table created will be HIVE MANAGED table.</td>
</tr>
<tr>
<td>3</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>4</td>
<td>Target_Table</td>
<td>Enter the table name to create and load.</td>
</tr>
<tr>
<td>5</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file. Choose Fixed width, #if the Source File is fixed width data file. Choose Table, if data needs to be imported from another database to hive.</td>
</tr>
<tr>
<td>6</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. #If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited. If the #Type_Of_Table is the Fixed width, then the Table Layout file should have column names, their data types, column start position and column end position, it must be tab delimited.</td>
</tr>
<tr>
<td>7</td>
<td>Table_Delimiter</td>
<td>Enter the Table delimiter.</td>
</tr>
<tr>
<td>8</td>
<td>File_Delimiter</td>
<td>Enter the column delimiter used in Source File.</td>
</tr>
<tr>
<td>9</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name. If you need to load multiple files, enter all the file names with path delimited by a comma.</td>
</tr>
<tr>
<td>10</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.</td>
</tr>
<tr>
<td>11</td>
<td>Null_Insert_Flag </td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>12</td>
<td>Table_Delete_Flag</td>
<td>Enter 'Delete Target Table', if you need to delete the target table, in case it exists.</td>
</tr>
<tr>
<td>13</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="default" Target_Table="Temp_Delimited_Data" Type_Of_Table="Delimited" Table_Layout_Path="/home/cloudera/Desktop/Table_Creation/Table_Layout_Delimited_Table.txt" Table_Delimiter="~" File_Delimiter="|" Load_Data_Path="/home/hadoop/Desktop/eds_request_data.txt" Transpose_Flag="Y" Null_Insert_Flag="Y" Table_Delete_Flag="Delete Target Table" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File.txt"./Data_Ing_Eng.sh
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:( Transpose )</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="default" Target_Table="EDS_Request_Prod_KV" Type_Of_Table="Delimited" Table_Layout_Path="/home/hadoop/Desktop/eds_request_layout.txt" Table_Delimiter="~" File_Delimiter="," Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Comma_File.txt" Transpose_Flag="Y" Null_Insert_Flag="Y" Table_Delete_Flag="Delete Target Table" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
2. Create and Insert - Fixed width File.</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th>#</th>
<th>Key Name</th>
<th>Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create".Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Type_Of_Create</td>
<td>Options "External, Managed". If External is chosen, then the table created will be EXTERNAL HIVE table. #If Managed is chosen, then the table created will be HIVE MANAGED table.</td>
</tr>
<tr>
<td>3</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>4</td>
<td>Target_Table</td>
<td>Enter the table name to create and load.</td>
</tr>
<tr>
<td>5</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file. Choose Fixed width, #if the Source File is fixed width data file.</td>
</tr>
<tr>
<td>6</td>
<td>Convert_Fixed_delimited_Flag</td>
<td>Options are Y or N. Enter Y, if you want to convert the fixed width data file to delimited data file. Enter N, #if you want to load data in fixed width format.</td>
</tr>
<tr>
<td>7</td>
<td>Fixed_Delimiter</td>
<td>Enter the delimiter, if you choose 'Y' for #Convert_Fixed_delimited_Flag. #This delimiter will be used to create the delimited data file.</td>
</tr>
<tr>
<td>8</td>
<td>Load_Data_Path</td>
<td>Enter Table layout file path and name. #If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited.If the #Type_Of_Table is the Fixed width, then the Table Layout file should have column names, their data types, column start position and column end position, it must be tab delimited.</td>
</tr>
<tr>
<td>9</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name. If you need to load multiple files, enter all the file names with path delimited by a comma.</td>
</tr>
<tr>
<td>10</td>
<td>Table_Delete_Flag </td>
<td>=Enter 'Delete Target Table', if you need to delete the target table, in case it exists.</td>
</tr>
<tr>
<td>11</td>
<td> Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example :(If Arg Convert_Fixed_delimited_Flag = "Y")</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="Managed" Data_Base_Name="default" Target_Table="Temp_Fixed_Data" Type_Of_Table="Fixed width" Convert_Fixed_delimited_Flag "Y" Fixed_Delimiter="|" Table_Layout_Path="/home/cloudera/Desktop/Table_Creation/Table_Layout_Fixed_Data.txt" Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Fixed_Width_sample_value.txt" Table_Delete_Flag="Delete Target Table" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File.txt" ./Data_Ing_Eng.sh
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example :(If Arg Convert_Fixed_delimited_Flag = "N")</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="Managed" Data_Base_Name="default" Target_Table="Temp_Fixed_Data" Type_Of_Table="Fixed width" Convert_Fixed_delimited_Flag="N" Fixed_Delimiter="NA" Table_Layout_Path="/home/cloudera/Desktop/Table_Creation/Table_Layout_Fixed_Data.txt" Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Fixed_Width_sample_value.txt" Table_Delete_Flag="Delete Target Table" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File.txt" ./Data_Ing_Eng.sh<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
3. Create and Insert - Table.</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create".Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Type_Of_Create</td>
<td>Options "External, Managed". If External is chosen, then the table created will be EXTERNAL HIVE table. #If Managed is chosen, then the table created will be HIVE MANAGED table.</td>
</tr>
<tr>
<td>3</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>4</td>
<td>Target_Table</td>
<td>Enter the table name to create and load.</td>
</tr>
<tr>
<td>5</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file. Choose Fixed width, #if the Source File is fixed width data file.</td>
</tr>
<tr>
<td>6</td>
<td>Create_Layout_Flag</td>
<td>Options are Y or N. Enter Y, if you want to create layout file automatically. Enter N, if you give layout file.</td>
</tr>
<tr>
<td>7</td>
<td>Table_Layout_Path</td>
<td>Enter layout file path and name, if you chose 'N' for #Create_Layout_Flag otherwise enter 'NA'.</td>
</tr>
<tr>
<td>8</td>
<td>Column_Name_Query</td>
<td>If you chose 'Y' for #Create_Layout_Flag, provide a Metadata SQL that can return the Source Table's, #column names, data type, precision, and scale. This will be used to build a comparable table in Hive.</td>
</tr>
<tr>
<td>9</td>
<td>Mapping_Data_Path</td>
<td>Enter the Source to Hive Data Type mapping file path and name. #This file will be used to convert Source Column Data Types to appropriate Hive Columns. Refer to supplied Oracle2HiveDataTypeMapping.txt.</td>
</tr>
<tr>
<td>10</td>
<td>Load_Data_Path</td>
<td>Enter source table connection string, username, password file path, source tablename and hdfs file path where the data from The #source table will be stored. This information should be entered and delimited by ','(comma) in the same order.</td>
</tr>
<tr>
<td>11</td>
<td>Table_Delimiter</td>
<td>Enter the table delimiter.</td>
</tr>
<tr>
<td>12</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest. #This option is not applicable if (Arg) Create_Layout_Flag is 'Y'</td>
</tr>
<tr>
<td>13</td>
<td>Audit_Columns</td>
<td>Enter the Audit Column details. It should contain audit column name, audit column datatype and function name,. #all details should be '~' delimited. Each audit column details should be delimited by ','.</td>
</tr>
<tr>
<td>14</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table. This option is not applicable if (Arg) Create_Layout_Flag is 'Y'.</td>
</tr>
<tr>
<td>15</td>
<td>Table_Delete_Flag</td>
<td>Enter 'Delete Target Table', if you need to delete the target table, in case it exists.</td>
</tr>
<tr>
<td>16</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example :(Auto creates the Table layout)</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="stage_db" Target_Table="EMP_TARGET" Type_Of_Table="Table" Create_Layout_Flag="Y" Table_Layout_Path="NA" Column_Name_Query="SELECT COLUMN_NAME,DATA_TYPE,DATA_PRECISION,DATA_SCALE FROM ALL_TAB_COLUMNS WHERE TABLE_NAME='EMP_TEMP' ##CONDITION## ORDER BY COLUMN_ID" Mapping_Data_Path="/home/hadoop/Desktop/Oracle2HiveDataTypeMapping.txt" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,EMP_TEMP,/user/hadoop/" Table_Delimiter="~" Transpose_Flag="N" Audit_Columns="AS_OF_DATE~DATE~CURRENT_DATE,CREATION_TS~VARCHAR(50)~CURRENT_TIMESTAMP,CREATED_BY~VARCHAR(50)~DEFAULT-J712798,PROCESS_CONTROL_ID~BIGINT~UNIQUE_VALUE" Null_Insert_Flag="N" Table_Delete_Flag="Delete Target Table" Log_File="/home/hadoop/Desktop/Temp_Fixed_Data_Log.txt" ./Data_Ing_Eng.sh
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example :(User provided Table layout)</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="default" Target_Table="Temp_Fixed_Data" Type_Of_Table="Table" Create_Layout_Flag="N" Table_Layout_Path="/home/hadoop/Desktop/Fixed_Length_Data.txt" Column_Name_Query="NA" Mapping_Data_Path="NA" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,EXPORT_SQOOP,/user/hadoop/" Table_Delimiter="~" Transpose_Flag="Y" Audit_Columns="NA" Null_Insert_Flag="Y" Table_Delete_Flag="Delete Target Table" Log_File="/home/hadoop/Desktop/Temp_Fixed_Data_Log.txt" ./Data_Ing_Eng.sh<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
4. Create and Insert - SQL</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Type_Of_Create</td>
<td>Options "External, Managed". If External is chosen, then the table created will be EXTERNAL HIVE table. #If Managed is chosen, then the table created will be HIVE MANAGED table.</td>
</tr>
<tr>
<td>3</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>4</td>
<td>Target_Table</td>
<td>Enter the table name to create and load.</td>
</tr>
<tr>
<td>5</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file. #Choose Fixed width of the Source File is fixed width data file. Choose Table, if data needs to be imported from another database to hive.</td>
</tr>
<tr>
<td>6</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. #If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited.</td>
</tr>
<tr>
<td>7</td>
<td>Source_SQL_Query</td>
<td>Enter the Source SQL query in which data will be exported to target table.</td>
</tr>
<tr>
<td>8</td>
<td>Load_Data_Path</td>
<td>Enter source table connection string, username, password file path and hdfs file path where the data from source table will be stored. #this information should be entered and delimited by ','(comma) in the same order.</td>
</tr>
<tr>
<td>9</td>
<td>Table_Delimiter</td>
<td>Enter the table delimiter.</td>
</tr>
<tr>
<td>10</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.</td>
</tr>
<tr>
<td>11</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>12</td>
<td>Table_Delete_Flag</td>
<td>Enter 'Delete Target Table', if you need to delete the target table, in case it exists.</td>
</tr>
<tr>
<td>13</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="Cygnus" Target_Table="EMPLOYEE_MASTER" Type_Of_Table="SQL" Table_Layout_Path="/home/hadoop/Desktop/EMPLOYEE_LAYOUT_FILE.txt" Source_SQL_Query="SELECT EMP_NO, NAME, POSITION, CLUB, NATIONALITY, BIRTHPLACE, HIREDATE, SALARY, PHONE_NO, EMAIL FROM EMPLOYEE WHERE HIREDATE > '25-05-2009' AND ##CONDITION##" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,/user/hadoop/" Table_Delimiter="~" Transpose_Flag="Y" Null_Insert_Flag="Y" Table_Delete_Flag="Delete Target Table" Log_File="/home/hadoop/Desktop/EMPLOYEE_MASTER.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
5. Create and Insert - XML File.</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Type_Of_Create</td>
<td>Options "External, Managed". If External is chosen, then the table created will be EXTERNAL HIVE table. #If Managed is chosen, then the table created will be HIVE MANAGED table.</td>
</tr>
<tr>
<td>3</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>4</td>
<td>Target_Table</td>
<td>Enter the table name to create and load.</td>
</tr>
<tr>
<td>5</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file. #Choose Fixed width of the Source File is fixed width data file. Choose Table, if data needs to be imported from another database to hive.</td>
</tr>
<tr>
<td>6</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. #If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited. If the #Type_Of_Table is the Fixed width, then the Table Layout file should have column names, their data types, column start position and column end position, it must be tab delimited.</td>
</tr>
<tr>
<td>7</td>
<td>Table_Delimiter </td>
<td>Enter the Table delimiter.</td>
</tr>
<tr>
<td>8</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name. If you need to load multiple files, enter all the file names with path delimited by a comma.</td>
</tr>
<tr>
<td>9</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.</td>
</tr>
<tr>
<td>10</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>11</td>
<td>Table_Delete_Flag</td>
<td>Enter 'Delete Target Table', if you need to delete the target table, in case it exists.</td>
</tr>
<tr>
<td>12</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
<tr>
<td>13</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Create & Insert" Type_Of_Create="External" Data_Base_Name="default" Target_Table="XML_Test" Type_Of_Table="XML_File" Table_Layout_Path="/home/hadoop/Desktop/xml/XML_Layout.txt" Load_Data_Path="/home/hadoop/Desktop/xml/log_file.xml" Table_Delimiter="~" Transpose_Flag="N" Null_Insert_Flag="N" Table_Delete_Flag="Delete Target Table" Log_File="/home/hadoop/Desktop/xml/XML_Test_Log.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
6. Insert only - Delimited File</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. #Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>3</td>
<td>Target_Table</td>
<td>Enter the table name to create and load..</td>
</tr>
<tr>
<td>4</td>
<td>Type_Of_Table</td>
<td>Enter Table layout file path and name. #If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited.</td>
</tr>
<tr>
<td>5</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. If the #Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited</td>
</tr>
<tr>
<td>6</td>
<td>Table_Delimiter</td>
<td>Enter the Table delimiter.</td>
</tr>
<tr>
<td>7</td>
<td>File_Delimiter</td>
<td>Enter the column delimiter used in Source File.</td>
</tr>
<tr>
<td>8</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name. If you need to load multiple files, enter all the file names with path delimited by a comma. NOTE: The column delimiter in this Source file should match with the delimiter configured in the target table.
</td>
</tr>
<tr>
<td>9</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.</td>
</tr>
<tr>
<td>10</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>11</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Insert" Data_Base_Name="default" Target_Table="Temp_Delimited_Data" Type_Of_Table="Delimited" Table_Layout_Path="/home/hadoop/Desktop/KV_LAYOUT.txt" Table_Delimiter="~" File_Delimiter="|" Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Comma_File.txt" Transpose_Flag="Y" Null_Insert_Flag="Y" Log_File"/home/cloudera/Desktop/Table_Creation/Log_File.txt"
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:(Transpose)</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Insert" Data_Base_Name="default" Target_Table="Individual_Prod_KV" Type_Of_Table="Delimited" Table_Layout_Path="/home/hadoop/Desktop/KV_LAYOUT.txt" Table_Delimiter="~" File_Delimiter="|" Load_Data_Path="/home/hadoop/Desktop/individual_data.txt" Transpose_Flag="Y" Null_Insert_Flag="Y" Log_File="/home/hadoop/Desktop/Individual_Prod_KV_Log.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
7. Insert only - Fixed Width File</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the table needs to be created.</td>
</tr>
<tr>
<td>3</td>
<td>Target_Table</td>
<td>Enter the table name to create and load..</td>
</tr>
<tr>
<td>4</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file.Choose Fixed width, if the Source File is fixed width data file. Choose Table, if data needs to be imported from another database to hive.</td>
</tr>
<tr>
<td>5</td>
<td>Convert_Fixed_delimited_Flag</td>
<td>Options are Y or N. Enter Y, if you want to convert the fixed width data file to delimited data file. Enter N, if you want to load data in fixed width format.</td>
</tr>
<tr>
<td>6</td>
<td>Fixed_Delimiter</td>
<td>Enter the delimiter, if you chose 'Y' for # Arg Convert_Fixed_delimited_Flag. This delimiter will be used to create the delimited data file.</td>
</tr>
<tr>
<td>7</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. If the # Arg Convert_Fixed_delimited_Flag is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited. If the # Arg Convert_Fixed_delimited_Flag is the Fixed width, then the Table Layout file should have column names, their data types, column start position and column end position, it must be tab delimited.</td>
</tr>
<tr>
<td>8</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name. If you need to load multiple files, enter all the file names with path delimited by a comma. NOTE: The Fixed width layout in this Source file should match with the Fixed width or delimiter configured in the target table.
</td>
</tr>
<tr>
<td>9</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:(If Arg Convert_Fixed_delimited_Flag = "Y")</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Insert" Data_Base_Name="default" Target_Table="Temp_Fixed_Data" Type_Of_Table="Fixed width" Convert_Fixed_delimited_Flag="Y" Fixed_Delimiter="|" Table_Layout_Path="/home/cloudera/Desktop/Table_Creation/Table_Layout_Fixed_Data.txt" Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Fixed_Width_sample_value.txt,/home/cloudera/Desktop/Table_Creation/Fixed_Width_sample_value1.txt" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File1.txt" ./Data_Ing_Eng.sh
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:(If Arg Convert_Fixed_delimited_Flag = "N")</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Insert" Data_Base_Name="default" Target_Table="Temp_Fixed_Data" Type_Of_Table="Fixed width" Convert_Fixed_delimited_Flag="N" Fixed_Delimiter="NA" Table_Layout_Path="/home/cloudera/Desktop/Table_Creation/Table_Layout_Fixed_Data.txt" Load_Data_Path="/home/cloudera/Desktop/Table_Creation/Fixed_Width_sample_value.txt" Log_File="/home/cloudera/Desktop/Table_Creation/Log_File1.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
8. Insert only - Table</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the source table is available.</td>
</tr>
<tr>
<td>3</td>
<td>Target_Table</td>
<td>Enter the table name to create and load..</td>
</tr>
<tr>
<td>4</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File and Table. Choose Delimited, if the Source File is a delimited file.Choose Fixed width, if the Source File is fixed width data file. Choose Table, if data needs to be imported from another database to hive.</td>
</tr>
<tr>
<td>5</td>
<td>Load_Data_Path</td>
<td>Enter source table connection string, username, password file path, source tablename and hdfs file path where the data from The source table will be stored. This information should be entered and delimited by ','(comma) in the same order.</td>
</tr>
<tr>
<td>6</td>
<td>Table_Delimiter </td>
<td>Enter the Column Delimiter to create the Target table.</td>
</tr>
<tr>
<td>7</td>
<td>Create_Layout_Flag</td>
<td>Options are Y or N. Enter Y if you want to create Table layout file automatically. Enter N, if you will provide the Table layout file.</td>
</tr>
<tr>
<td>8</td>
<td>Table_Layout_Path</td>
<td>Enter layout file path and name, if you chose 'N' for #Table_Layout_Path, otherwise enter 'NA'.
</td>
</tr>
<tr>
<td>9</td>
<td>Column_Name_Query</td>
<td>If you chose 'Y' for #Create_Layout_Flag, provide a Metadata SQL that can return the Source Table's column names, datatype, precision and scale. This will be used to build a comparable table in Hive.</td>
</tr>
<tr>
<td>10</td>
<td>Mapping_Data_Path</td>
<td>Enter the Source to Hive Data Type mapping file path and name. This file will be used to convert Source Column Data Types to appropriate Hive Columns. NOTE : Refer to supplied Oracle2HiveDataTypeMapping.txt.</td>
</tr>
<tr>
<td>11</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest. This option is not applicable if #Create_Layout_Flag is 'Y'.</td>
</tr>
<tr>
<td>12</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>13</td>
<td>Audit_Columns</td>
<td>Enter the additional columns that you like to ingest. It should contain the column name, data type, and function name, Functions supported are UNIQUE_VALUE, DEFAULT-<Value>, CURRENT_DATE, CURRENT_TIMESTAMP delimited by ~ (Tilde). If their multiple columns delimit them by, (Comma).</td>
</tr>
<tr>
<td>14</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:(If Arg Convert_Fixed_delimited_Flag = "Y")</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="INSERT" Data_Base_Name="STAGE_DB" Target_Table="EMP_TARGET" Type_Of_Table="Table" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,EXPORT_SQOOP,/user/hadoop/" Table_Delimiter="~" Create_Layout_Flag="Y" Table_Layout_Path="NA" Column_Name_Query="SELECT COLUMN_NAME,DATA_TYPE,DATA_PRECISION,DATA_SCALE FROM ALL_TAB_COLUMNS WHERE TABLE_NAME='EMP_TARGET' ##CONDITION## ORDER BY COLUMN_ID" Mapping_Data_Path="/home/hadoop/Desktop/Oracle2HiveDataTypeMapping.txt" Transpose_Flag="N" Null_Insert_Flag="Y" Audit_Columns="AS_OF_DATE~DATE~CURRENT_DATE,CREATION_TS~VARCHAR(50)~CURRENT_TIMESTAMP,CREATED_BY~VARCHAR(50)~DEFAULT-J712798,PROCESS_CONTROL_ID~BIGINT~UNIQUE_VALUE" Log_File="/home/hadoop/Desktop/Log_File.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
9. Insert only - Query.</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the source table is available.</td>
</tr>
<tr>
<td>3</td>
<td>Target_Table</td>
<td>Enter the target table name to load.</td>
</tr>
<tr>
<td>4</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File, Table and SQL. Choose Delimited, if the Source File is a delimited file. Choose Fixed width of the Source File is fixed width data file.Choose Table, if whole table data needs to be imported from another database to hive. Choose SQL if you are passing SQL as a data source. In this case, you need to provide the Table Layout file as well.</td>
</tr>
<tr>
<td>5</td>
<td>Load_Data_Path</td>
<td>Enter source table connection string, username, password file path and hdfs file path where the data from The source table will be stored.This information should be entered and delimited by ','(comma) in the same order</td>
</tr>
<tr>
<td>6</td>
<td>Table_Delimiter</td>
<td>Enter the Column Delimiter to create the Target.</td>
</tr>
<tr>
<td>7</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. If the Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited.</td>
</tr>
<tr>
<td>8</td>
<td>Source_SQL_Query</td>
<td>Enter the Source SQL query in which data will be exported to target table.
</td>
</tr>
<tr>
<td>9</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.</td>
</tr>
<tr>
<td>10</td>
<td> Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>11</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="INSERT" Data_Base_Name="stage_db" Target_Table="EMP_SQL_TEST" Type_Of_Table="SQL" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,/user/hadoop/" Table_Delimiter="~" Table_Layout_Path="/home/hadoop/Desktop/EMP_LAYOUT_FILE.txt" Source_SQL_Query="select * from emp_temp where hiredate > '30-05-2017' AND ##CONDITION##" Transpose_Flag="Y" Null_Insert_Flag="Y" Log_File="/home/hadoop/Desktop/Log_File.txt"
<br />
</span>
<h4>
<div style="height: 5px;">
</div>
<strong style="color: #55AA22;">Example:(Transpose)</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="INSERT" Data_Base_Name="default" Target_Table="STG_INDIVIDUAL_QUERY" Type_Of_Table="SQL" Load_Data_Path="jdbc:oracle:thin:@192.168.100.8:1521:orcl,scott,file:/home/hadoop/Desktop/pass.txt,/user/hadoop/" Table_Delimiter="~" Table_Layout_Path="/home/hadoop/Desktop/KV_LAYOUT.txt" Source_SQL_Query="select * from STG_GARWIN_INDIVIDUAL where CONTRACT_RELATIONSHIP = 'OWN' AND ##CONDITION##" Transpose_Flag="Y" Null_Insert_Flag="Y" Log_File="/home/hadoop/Desktop/Log_File.txt"
<br />
</span>
</div>
<div style="height: 10px;">
</div>
<div class="syntax_notestyle">
<h1>
10. Insert only - XML File.</h1>
<div style="height: 5px;">
</div>
<h4>
<strong style="color: #55AA22;">Command </strong><strong style="color: #55AA22;">Template</strong></h4>
<div style="height: 5px;">
</div>
<table class="tableborder">
<tr>
<th style="width:3%">#</th>
<th style="width:20%">Key Name</th>
<th style="width:77%">Description</th>
</tr>
<tr>
<td>1</td>
<td>Type_Of_Ingestion</td>
<td>Options are "Create & Insert, Insert, Create". Create & Insert option will create table and load data into the created table. Insert option will load data into the specified table. Create option will only create the table without loading any data.</td>
</tr>
<tr>
<td>2</td>
<td>Data_Base_Name</td>
<td>Enter the Database name in which the source table is available.</td>
</tr>
<tr>
<td>3</td>
<td>Target_Table</td>
<td>Enter the target table name to load.</td>
</tr>
<tr>
<td>4</td>
<td>Type_Of_Table</td>
<td>Options are: Delimited, Fixed width, XML_File, Table and SQL. Choose Delimited, if the Source File is a delimited file. Choose Fixed width of the Source File is fixed width data file.Choose Table, if whole table data needs to be imported from another database to hive. Choose SQL if you are passing SQL as a data source. In this case, you need to provide the Table Layout file as well.</td>
</tr>
<tr>
<td>5</td>
<td>Table_Layout_Path</td>
<td>Enter Table layout file path and name. If the Type_Of_Table is Delimited or Table, then the Table Layout File should have column names and their data types, tab delimited.</td>
</tr>
<tr>
<td>6</td>
<td>Table_Delimiter</td>
<td>Enter the Column Delimiter to create the Target table.</td>
</tr>
<tr>
<td>7</td>
<td>Load_Data_Path</td>
<td>Enter the Source File path with name.If you need to load multiple files, enter all the file names with path delimited by a comma. NOTE: The column delimiter in this Source file should match with the delimiter configured in the target table.</td>
</tr>
<tr>
<td>8</td>
<td>Transpose_Flag</td>
<td>Flag for Transpose. Enter 'Y' if you want to load data using Transpose Ingest.
</td>
</tr>
<tr>
<td>9</td>
<td>Null_Insert_Flag</td>
<td>Enter 'Y' if you don't want to insert null data into target table.</td>
</tr>
<tr>
<td>10</td>
<td>Log_File</td>
<td>Log file path and name. Stores the logs generated by this tool.</td>
</tr>
</table>
<h4>
<div style="height: 10px;">
</div>
<strong style="color: #55AA22;">Example</strong></h4>
<div style="height: 5px;">
</div>
<span class="justifyfontEX">
Type_Of_Ingestion="Insert" Data_Base_Name="default" Target_Table="XML_Test" Type_Of_Table="XML_File" Table_Layout_Path="/home/hadoop/Desktop/xml/XML_Layout.txt" Table_Delimiter="~" Load_Data_Path="/home/hadoop/Desktop/xml/log_file.xml" Transpose_Flag="N" Null_Insert_Flag="N" Log_File="/home/hadoop/Desktop/xml/XML_File_Log.txt" ./Data_Ing_Eng.sh
<br />
</span>
</div>
<div style="height: 5px;">
</div>
<div class="notestyle">
<div style="margin-top: 5px;">
<h2 style="color: #F63B02; font-family: "times";">
Disclaimer </h2>
<h3 style="color: #55AA22;">
Please ensure you read and understand the following general disclaimer:</h3>
<div class="blackfont">
<div class="justifyfontEX">
<h4>
IMPORTANT:</h4>
THIS SOFTWARE END USER LICENSE AGREEMENT (“EULA”) IS A LEGAL AGREEMENT BETWEEN YOU AND PRODEN TECHNOLOGIES, INC. READ IT CAREFULLY BEFORE COMPLETING THE INSTALLATION PROCESS AND USING THE SOFTWARE. IT PROVIDES A LICENSE TO USE THE SOFTWARE AND CONTAINS WARRANTY INFORMATION AND LIABILITY DISCLAIMERS. BY INSTALLING AND USING THE SOFTWARE, YOU ARE CONFIRMING YOUR ACCEPTANCE OF THE SOFTWARE AND AGREEING TO BECOME BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO BE BOUND BY THESE TERMS, THEN SELECT THE "CANCEL" BUTTON. DO NOT PROCEED TO REGISTER & INSTALL THE SOFTWARE.
LIABILITY DISCLAIMER•THE accel<>DS PROGRAM IS DISTRIBUTED "AS IS". NO WARRANTY OF ANY KIND IS EXPRESSED OR IMPLIED. YOU USE IT AT YOUR OWN RISK. NEITHER THE AUTHORS NOR PRODEN TECHNOLOGIES, INC. WILL BE LIABLE FOR DATA LOSS, DAMAGES AND LOSS OF PROFITS OR ANY OTHER KIND OF LOSS WHILE USING OR MISUSING THIS SOFTWARE.
</div>
<div style="height: 4px;">
</div>
<div class="justifyfontEX">
<h4>
RESTRICTIONS:</h4>
You may not use, copy, emulate, clone, rent, lease, sell, modify, decompile, disassemble, otherwise reverse engineer, or transfer any version of the Software, or any subset of it, except as provided for in this agreement. Any such unauthorized use shall result in immediate and automatic termination of this license and may result in criminal and/or civil prosecution.
</div>
<div style="height: 4px;">
</div>
<div class="justifyfontEX">
<h4>
TERMS:</h4>
This license is effective until terminated. You may terminate it by destroying the program, the documentation and copies thereof. This license will also terminate if you fail to comply with any terms or conditions of this agreement. You agree upon such termination to destroy all copies of the program and of the documentation, or return them to the author.</div>
</div>
</div>
</div>
<br />
</div>
</div>
</body></html>
<div class="trialcontain">
<div class="trial-sucesstext">
<a href="http://www.prodentechnologies.net/download.php?ProductName=Data%20Ingestion%20Scripts" style="color: white; font-family: "calibri"; font-size: 250%;">Data Ingestion Framework for Hadoop (Free)</a></br>
</div>
</div>
<div>
<a class="scrollToTop" href="https://www.blogger.com/blogger.g?blogID=1292635231032939247#">
<img class="scrollimg" src="http://prodentechnologies.net/images/scrolltop.png" style="background-color: transparent;" title="scroll to Top" />
</a>
</div>
</div>
Proden Technologieshttp://www.blogger.com/profile/08742796894235273079noreply@blogger.com1