Informatica Big Data Management User Guide

Informatica Big Data Management User Guide


Table of Contents for above guide.

Preface ....................................................................... 8
Informatica Resources................................................... 8
Informatica My Support Portal........................................... 8
Informatica Documentation............................................. 8
Informatica Product Availability Matrixes..................................... 8
Informatica Web Site................................................. 9
Informatica How-To Library............................................. 9
Informatica Knowledge Base............................................ 9
Informatica Support YouTube Channel...................................... 9
Informatica Marketplace............................................... 9
Informatica Velocity.................................................. 9
Informatica Global Customer Support...................................... 9
Chapter 1: Introduction to Informatica Big Data Management.................. 11
Informatica Big Data Management Overview.................................... 11
Example........................................................ 12
Big Data Management Tasks .............................................. 12
Read from and Write to Big Data Sources and Targets........................... 12
Perform Data Discovery.............................................. 13
Perform Data Lineage on Big Data Sources.................................. 13
Stream Machine Data................................................ 14
Manage Big Data Relationships......................................... 14
Big Data Process...................................................... 14
Step 1. Collect the Data.............................................. 15
Step 2. Cleanse the Data............................................. 15
Step 3. Transform the Data............................................ 15
Step 4. Process the Data.............................................. 15
Step 5. Monitor Jobs................................................ 16
Big Data Management Component Architecture.................................. 16
Clients and Tools................................................... 16
Application Services................................................. 17
Repositories...................................................... 17
Third-Party Applications.............................................. 18
Big Data Management Connectivity Architecture.................................. 18
Hadoop Ecosystem Architecture............................................ 19
Chapter 2: Connections...................................................... 20
Connections Overview.................................................. 20
Hadoop Connection Properties............................................. 21
HDFS Connection Properties.............................................. 28
4 Table of Contents
HBase Connection Properties.............................................. 29
Hive Connection Properties............................................... 30
Creating a Connection to Access Sources or Targets............................... 36
Creating a Hadoop Connection............................................. 37
Chapter 3: Mappings in a Hadoop Environment................................ 39
Mappings in a Hadoop Environment Overview................................... 39
Data Warehouse Optimization Mapping Example ................................. 40
Hive Engine Architecture................................................. 42
Informatica Blaze Engine Architecture........................................ 43
High-Level Steps to Run a Mapping in the Hadoop Environment....................... 45
Sources in a Hadoop Environment........................................... 45
Flat File Sources................................................... 46
Hive Sources..................................................... 46
Relational Sources.................................................. 46
Targets in a Hadoop Environment........................................... 47
Flat File Targets................................................... 47
HDFS Flat File Targets............................................... 47
Hive Targets...................................................... 47
Relational Targets.................................................. 48
Transformations in a Hadoop Environment..................................... 48
Variable Ports in a Hadoop Environment.................................... 51
Functions in a Hadoop Environment.......................................... 51
Mappings in a Hadoop Environment.......................................... 52
Data Types in a Hadoop Environment........................................ 53
Parameters in a Hadoop Environment........................................ 53
Parameter Usage.................................................. 54
Create and Use Hadoop Parameters...................................... 55
Workflows that Run Mappings in a Hadoop Environment............................ 56
Configuring a Mapping to Run in a Hadoop Environment............................ 56
Mapping Execution Plans................................................ 57
Hive Engine Execution Plan Details....................................... 58
Blaze Engine Execution Plan Details...................................... 58
Viewing the Execution Plan for a Mapping in the Developer Tool.................... 59
Monitor Jobs......................................................... 60
Accessing the Monitoring URL.......................................... 61
Monitor Blaze Engine Jobs............................................ 62
Monitoring a Mapping................................................ 63
Hadoop Environment Logs................................................ 63
Blaze Engine Logs.................................................. 64
Hive Engine Logs.................................................. 64
Viewing Hadoop Environment Logs in the Administrator Tool....................... 65
Viewing Logs in the Blaze Job Monitor..................................... 65
Table of Contents 5
Optimization for the Hadoop Environment...................................... 65
Truncating Partitions in a Hive Target...................................... 66
Enabling Data Compression on Temporary Staging Tables........................ 66
Parallel Sorting.................................................... 67
Troubleshooting a Mapping in a Hadoop Environment.............................. 67
Chapter 4: Mappings in the Native Environment............................... 69
Mappings in the Native Environment Overview................................... 69
Data Processor Mappings................................................ 69
HDFS Mappings...................................................... 70
HDFS Data Extraction Mapping Example................................... 70
Hive Mappings....................................................... 71
Hive Mapping Example............................................... 72
Social Media Mappings.................................................. 72
Twitter Mapping Example............................................. 73
Chapter 5: Profiles........................................................... 74
Profiles Overview..................................................... 74
Native and Hadoop Environments........................................... 75
Supported Data Source and Run-time Environments............................ 75
Run-time Environment Setup and Validation................................. 76
Run-time Environment and Profile Performance............................... 77
Profile Types on Hadoop................................................. 77
Column Profiles on Hadoop............................................ 77
Rule Profiles on Hadoop.............................................. 77
Data Domain Discovery on Hadoop....................................... 78
Running a Profile on Hadoop in the Developer Tool................................ 78
Running a Profile on Hadoop in the Analyst Tool................................. 78
Running Multiple Data Object Profiles on Hadoop................................. 79
Monitoring a Profile.................................................... 80
Troubleshooting...................................................... 80
Chapter 6: Native Environment Optimization.................................. 82
Native Environment Optimization Overview..................................... 82
Processing Big Data on a Grid............................................. 82
Data Integration Service Grid........................................... 83
Grid Optimization................................................... 83
Processing Big Data on Partitions........................................... 83
Partitioned Model Repository Mappings.................................... 83
Partition Optimization................................................ 84
High Availability....................................................... 84
6 Table of Contents
Appendix A: Data Type Reference............................................ 86
Data Type Reference Overview............................................ 86
Hive Complex Data Types................................................ 86
Hive Data Types and Transformation Data Types................................. 87

Informatica MDM Big Data Relationship Management Guide

Informatica MDM Big Data Relationship Management Guide


Table of Contents for above guide
Installation..................................................................................... 1
MDM Big Data Relationship Management Version 10.0.0 HotFix 3.................................. 2
Fixed Limitations................................................................ 2
Known Limitations............................................................... 2
MDM Big Data Relationship Management Version 10.0.0 HotFix 2.................................. 3
Known Limitations............................................................... 3
MDM Big Data Relationship Management Version 10.0.0 HotFix 1.................................. 3
Fixed Limitations................................................................ 3
MDM Big Data Relationship Management Version 10.0.0........................................... 4
Known Limitations............................................................... 4
Informatica Global Customer Support............................................................. 4

Informatica Big Data Management Security Guide

Informatica Big Data Management Security Guide


Table of Contents for above
Preface ....................................................................... 6
Informatica Resources................................................... 6
Informatica My Support Portal........................................... 6
Informatica Documentation............................................. 6
Informatica Product Availability Matrixes..................................... 6
Informatica Web Site................................................. 7
Informatica How-To Library............................................. 7
Informatica Knowledge Base............................................ 7
Informatica Support YouTube Channel...................................... 7
Informatica Marketplace............................................... 7
Informatica Velocity.................................................. 7
Informatica Global Customer Support...................................... 7
Chapter 1: Introduction to Big Data Management Security ...................... 9
Overview............................................................ 9
Authentication........................................................ 10
Kerberos Authentication.............................................. 10
Authorization........................................................ 11
HDFS Permissions.................................................. 11
User Impersonation................................................. 11
Data and Metadata Management........................................... 12
Cloudera Navigator and Metadata Manager.................................. 12
Data Security........................................................ 12
Chapter 2: Running Mappings with Kerberos Authentication................... 14
Running Mappings with Kerberos Authentication Overview........................... 14
Requirements for the Hadoop Cluster......................................... 15
Prerequisite Tasks for Running Mappings on a Hadoop Cluster with Kerberos Authentication..... 15
Install the JCE Policy File for AES-256 Encryption............................. 16
Verify Permissions on the Hadoop Warehouse................................ 16
Configure Kerberos Security Properties in hive-site.xml.......................... 16
Setting up the Kerberos Configuration File.................................. 18
Running Mappings in the Hadoop Environment when Informatica Does not Use Kerberos
Authentication........................................................ 20
Step 1. Create Matching Operating System Profile Names........................ 21
Step 2. Create the Principal Names and Keytab File in the AD KDC.................. 21
Step 3. Specify the Kerberos Authentication Properties for the Data Integration Service..... 21
Running Mappings in a Hadoop Environment When Informatica Uses Kerberos Authentication.... 22
Step 1. Set up the One-Way Cross-Realm Trust............................... 22
Step 2. Create Matching Operating System Profile Names........................ 24
4 Table of Contents
Step 3. Create an SPN and Keytab File in the Active Directory Server................. 24
Step 4. Specify the Kerberos Authentication Properties for the Data Integration Service..... 25
Running Mappings in the Native Environment when Informatica Uses Kerberos Authentication.... 25
Running Mappings in the Native Environment When Informatica Does not Use Kerberos
Authentication........................................................ 26
Kerberos Authentication for a Hive Connection................................... 26
Kerberos Authentication for an HBase Connection................................ 27
Kerberos Authentication for an HDFS Connection................................. 28
Metadata Import in the Developer Tool........................................ 28
Create and Configure the Analyst Service...................................... 29
Chapter 3: User Impersonation with Kerberos Authentication.................. 30
User Impersonation.................................................... 30
Create a Proxy Directory for Clusters that Run MapR.............................. 31
User Impersonation in the Hadoop Environment.................................. 31
Step 1. Enable the SPN of the Data Integration Service to Impersonate Another User....... 32
Step 2. Specify a User Name in the Hadoop Connection......................... 32
User Impersonation in the Native Environment................................... 33
Step 1. Login to the Kerberos Realm...................................... 33
Step 2. Specify the Kerberos Authentication Properties for the Data Integration Service..... 33
Step 3. Configure the Execution Options for the Data Integration Service............... 34
Step 4. Specify the URL for the Hadoop Cluster in the Connection Properties............ 34
Step 5. Configure the Mapping Impersonation Property.......................... 34
Chapter 4: Blaze Engine Security............................................. 35
Blaze Engine Security Overview............................................ 35
Setting up a Blaze User Account............................................ 35

Informatica Big Data Management Installation and Configuration Guide

Informatica Big Data Management Installation and Configuration Guide

Table of contents for above guide.

Preface ....................................................................... 7
Informatica Resources................................................... 7
Informatica My Support Portal........................................... 7
Informatica Documentation............................................. 7
Informatica Product Availability Matrixes..................................... 7
Informatica Web Site................................................. 8
Informatica How-To Library............................................. 8
Informatica Knowledge Base............................................ 8
Informatica Support YouTube Channel...................................... 8
Informatica Marketplace............................................... 8
Informatica Velocity.................................................. 8
Informatica Global Customer Support...................................... 8
Chapter 1: Installation and Configuration..................................... 10
Installation and Configuration Overview....................................... 10
Informatica Big Data Management Installation Process.......................... 10
Before You Begin..................................................... 11
Install and Configure the Informatica Domain and Clients......................... 11
Install and Configure PowerExchange Adapters............................... 11
Install and Configure Data Replication..................................... 12
Pre-Installation Tasks for a Single Node Environment........................... 12
Pre-Installation Tasks for a Cluster Environment............................... 12
Informatica Big Data Management Installation................................... 14
Installing in a Single Node Environment.................................... 14
Installing in a Cluster Environment from the Primary NameNode Using SCP Protocol....... 15
Installing in a Cluster Environment from the Primary NameNode Using FTP, HTTP, or NFS
Protocol......................................................... 15
Installing in a Cluster Environment from any Machine........................... 16
Installing Big Data Management Using Cloudera Manager........................ 17
After You Install....................................................... 17
Configure Hadoop Pushdown Properties for the Data Integration Service............... 18
Reference Data Requirements.......................................... 19
Hive Variables for Mappings in a Hadoop Environment.......................... 19
Update Hadoop Cluster Configuration Parameters............................. 20
Library Path and Path Variables for Mappings in a Hadoop Environment............... 21
Configure the Blaze Engine Log Directories.................................. 21
Hadoop Environment Properties File...................................... 21
Informatica Developer Files and Variables................................... 22
Open the Required Ports for the Blaze Engine................................ 22
Enable Support for Lookup Transformations with Teradata Data Objects............... 22
4 Table of Contents
Informatica Big Data Management Uninstallation................................. 23
Uninstalling Big Data Management....................................... 23
Chapter 2: Mappings on Hadoop Distributions................................ 25
Mappings on Hadoop Distributions Overview.................................... 25
Big Data Management Configuration Utility..................................... 26
Use Cloudera Manager............................................... 27
Use SSH........................................................ 27
Use a Shared Directory............................................... 28
Mappings on Cloudera CDH............................................... 29
Configure Hadoop Cluster Properties on the Data Integration Service Machine........... 29
Create a Staging Directory on HDFS...................................... 30
Configure Virtual Memory Limits......................................... 31
Add hbase_protocol.jar to the Hadoop classpath.............................. 31
Configure the Blaze Engine............................................ 31
Mappings on Hortonworks HDP............................................ 34
Configure Hadoop Cluster Properties for the Data Integration Service................. 35
Enable Tez....................................................... 37
Add hbase_protocol.jar to the Hadoop classpath.............................. 39
Enable HBase Support............................................... 39
Configure the Hadoop Cluster for the Blaze Engine............................. 40
Mappings on IBM BigInsights.............................................. 41
User Account for the JDBC and Hive Connections ............................. 42
Mappings on MapR.................................................... 42
Verify the Cluster Details.............................................. 42
Configure hive-site.xml on the Data Integration Service Machine for MapReduce 1........ 43
Configure hive-site.xml on Every Node in the Hadoop Cluster for MapReduce 1.......... 44
Configure Hadoop Cluster Properties on the Data Integration Service Machine for
MapReduce 2..................................................... 44
Configure yarn-site.xml on Every Node in the Cluster for MapReduce 2................ 45
Configure MapR Distribution Variables for Mappings in a Hadoop Environment........... 47
Configure the Heap Space for the MapR-FS................................. 47
Enable Hadoop Pushdown for HBase...................................... 47
Configure the Application Timeline Server................................... 48
Mappings on Pivotal HD................................................. 49
Configure Hadoop Cluster Properties for Pivotal HD in yarn-site.xml.................. 49
Configure Virtual Memory Limits......................................... 51
Chapter 3: High Availability.................................................. 52
Configure High Availability................................................ 52
Configuring Big Data Management for a Highly Available Cloudera CDH Cluster............. 53
Configuring Big Data Management for a Highly Available Hortonworks HDP Cluster........... 54
Configuring Big Data Management for a Highly Available IBM BigInsights Cluster............ 55
Table of Contents 5
Configuring Big Data Management for a Highly Available MapR Cluster................... 56
Configuring Big Data Management for a Highly Available Pivotal Cluster.................. 57
Appendix A: Upgrade Big Data Management.................................. 58
Upgrading Big Data Management........................................... 58

Informatica Big Data Management Tutorials

• Informatica Big Data Management Installation and Configuration Guide
• Informatica Big Data Management Security Guide
• Informatica MDM Big Data Relationship Management Guide
• Informatica Big Data Management User Guide