Informatica Big Data Management Installation and Configuration Guide

Informatica Big Data Management Installation and Configuration Guide

Table of contents for above guide.

Preface ....................................................................... 7
Informatica Resources................................................... 7
Informatica My Support Portal........................................... 7
Informatica Documentation............................................. 7
Informatica Product Availability Matrixes..................................... 7
Informatica Web Site................................................. 8
Informatica How-To Library............................................. 8
Informatica Knowledge Base............................................ 8
Informatica Support YouTube Channel...................................... 8
Informatica Marketplace............................................... 8
Informatica Velocity.................................................. 8
Informatica Global Customer Support...................................... 8
Chapter 1: Installation and Configuration..................................... 10
Installation and Configuration Overview....................................... 10
Informatica Big Data Management Installation Process.......................... 10
Before You Begin..................................................... 11
Install and Configure the Informatica Domain and Clients......................... 11
Install and Configure PowerExchange Adapters............................... 11
Install and Configure Data Replication..................................... 12
Pre-Installation Tasks for a Single Node Environment........................... 12
Pre-Installation Tasks for a Cluster Environment............................... 12
Informatica Big Data Management Installation................................... 14
Installing in a Single Node Environment.................................... 14
Installing in a Cluster Environment from the Primary NameNode Using SCP Protocol....... 15
Installing in a Cluster Environment from the Primary NameNode Using FTP, HTTP, or NFS
Protocol......................................................... 15
Installing in a Cluster Environment from any Machine........................... 16
Installing Big Data Management Using Cloudera Manager........................ 17
After You Install....................................................... 17
Configure Hadoop Pushdown Properties for the Data Integration Service............... 18
Reference Data Requirements.......................................... 19
Hive Variables for Mappings in a Hadoop Environment.......................... 19
Update Hadoop Cluster Configuration Parameters............................. 20
Library Path and Path Variables for Mappings in a Hadoop Environment............... 21
Configure the Blaze Engine Log Directories.................................. 21
Hadoop Environment Properties File...................................... 21
Informatica Developer Files and Variables................................... 22
Open the Required Ports for the Blaze Engine................................ 22
Enable Support for Lookup Transformations with Teradata Data Objects............... 22
4 Table of Contents
Informatica Big Data Management Uninstallation................................. 23
Uninstalling Big Data Management....................................... 23
Chapter 2: Mappings on Hadoop Distributions................................ 25
Mappings on Hadoop Distributions Overview.................................... 25
Big Data Management Configuration Utility..................................... 26
Use Cloudera Manager............................................... 27
Use SSH........................................................ 27
Use a Shared Directory............................................... 28
Mappings on Cloudera CDH............................................... 29
Configure Hadoop Cluster Properties on the Data Integration Service Machine........... 29
Create a Staging Directory on HDFS...................................... 30
Configure Virtual Memory Limits......................................... 31
Add hbase_protocol.jar to the Hadoop classpath.............................. 31
Configure the Blaze Engine............................................ 31
Mappings on Hortonworks HDP............................................ 34
Configure Hadoop Cluster Properties for the Data Integration Service................. 35
Enable Tez....................................................... 37
Add hbase_protocol.jar to the Hadoop classpath.............................. 39
Enable HBase Support............................................... 39
Configure the Hadoop Cluster for the Blaze Engine............................. 40
Mappings on IBM BigInsights.............................................. 41
User Account for the JDBC and Hive Connections ............................. 42
Mappings on MapR.................................................... 42
Verify the Cluster Details.............................................. 42
Configure hive-site.xml on the Data Integration Service Machine for MapReduce 1........ 43
Configure hive-site.xml on Every Node in the Hadoop Cluster for MapReduce 1.......... 44
Configure Hadoop Cluster Properties on the Data Integration Service Machine for
MapReduce 2..................................................... 44
Configure yarn-site.xml on Every Node in the Cluster for MapReduce 2................ 45
Configure MapR Distribution Variables for Mappings in a Hadoop Environment........... 47
Configure the Heap Space for the MapR-FS................................. 47
Enable Hadoop Pushdown for HBase...................................... 47
Configure the Application Timeline Server................................... 48
Mappings on Pivotal HD................................................. 49
Configure Hadoop Cluster Properties for Pivotal HD in yarn-site.xml.................. 49
Configure Virtual Memory Limits......................................... 51
Chapter 3: High Availability.................................................. 52
Configure High Availability................................................ 52
Configuring Big Data Management for a Highly Available Cloudera CDH Cluster............. 53
Configuring Big Data Management for a Highly Available Hortonworks HDP Cluster........... 54
Configuring Big Data Management for a Highly Available IBM BigInsights Cluster............ 55
Table of Contents 5
Configuring Big Data Management for a Highly Available MapR Cluster................... 56
Configuring Big Data Management for a Highly Available Pivotal Cluster.................. 57
Appendix A: Upgrade Big Data Management.................................. 58
Upgrading Big Data Management........................................... 58