HDFS High Availability

HDFS High Availability

Why High Availability : Without HA implemented on the HDFS NameNode, the NameNode is a single point of failure for the entire filesystem. A SecondaryNameNode, while helpful in reducing recovery times and providing an alternate storage location for the NameNode’s metadata, is not a high availability or hot standby solution. When can a NameNode become unavailable: - Unplanned Reasons: hardware or software failure. - Planned Reasons: Restart for  software upgrade or configuration change. How is HA implemented at high level: Using NameNode high availability (HA), two NameNodes are deployed managing the same HDFS namespace: one active and one standby NameNode. The Standby NameNode takes control of the filesystem, managing client read-and-write requests and metadata updates, if the active NameNode fails. Clients only ever connect to the active NameNode. The active NameNode writes its metadata to a quorum of JournalNodes, managed by the QuorumJournalManager (QJM) process, which is built into the NameNode. Quorums are deployed in odd numbers greater than one (3 or 5 for example). Metadata updates (edits) written to a majority of JournalNodes (a quorum) are considered to be consistent. The Standby NameNode reads consistent updates from the JournalNodes to apply changes to its in-memory metadata representation. DataNodes in the cluster send their heartbeats and block reports to both NameNodes using an abstraction called a nameservice. The nameservice is the same method used by clients to connect to the “active” NameNode. Using HA, the Standby NameNode is transactionally consistent with the active NameNode, so a failure and subsequent failover will not result in a loss of data or significant disruption. Impact of HA on Secondary NN: The Standby NameNode performs the checkpointing functions normally provided by the SecondaryNameNode, so in a HA configuration the SecondaryNameNode is no longer required.

Fencing:   Clients connect to an active NameNode in a high availability HDFS cluster. Only one NameNode can be active in the cluster at any given time. If more than one NameNode were active it could lead to inconsistencies or corruption of the filesystem. Fencing is a technique employed by the QJM to ensure that clients are not able to access a NameNode that is in standby mode due to a failover. Fencing isolates a NameNode, preventing further requests from being accepted. Following are the fencing methods available: Sshfence: Connects to the active NameNode using SSH, and uses fuser to kill the process listening on the service’s TCP port Shell: Runs an arbitrary shell command to fence the Active NameNode Fencing method are implemented using the dfs.ha.fencing.methods property in the hdfs-site.xml configuration file. <property>   <name>dfs.ha.fencing.methods</name>   <value>sshfence</value> </property> The QJM does not allow the active NameNode to take further metadata update requests until the previously active NameNode has been fenced.

Types of Failover:   Failover of a NameNode, or changing the state of a NameNode from standby to active, can be either  1)automatic (system-detected and initiated) or 2)manual (user-initiated).   Manual Failover: Manual failover can be accomplished using the hdfs haadmin command hdfs haadmin -failover nn1 nn2 Automatic Failover: Automatic failover is implemented using an external service called the ZooKeeper Failover Controller(ZKFC) . The ZKFC process runs on each NameNode and uses a quorum of ZooKeeper nodes to maintain the state of the active node and automatically initiate a failover in the event of unavailability of the active NameNode.

Deploying HA:   Deploying NN HA is a disruptive exercise as you will need to stop and reconfigure HDFS services. First and foremost , we need a second host to act in the NameNode HA pair. This host will ideally have the same configuration and specifications as the existing NameNode as it will need to perform all of the functions of the primary NameNode. NameService:  A nameservice provides an abstraction for cluster clients and services to connect to one of the two available NameNodes. This is done through configuration, whereby the fs.defaultFS refers to a nameservice ID as opposed to a NameNode host. Nameservices available are defined in the dfs.nameservices property in the hdfs-site.xml file.   References to the hosts involved in the particular nameservice are configured using the dfs.ha.namenodes.<nameservice_id> property in the hdfs-site.xml configuration file.  The hosts themselves are then defined in dfs.namenode.rpc-address and dfs.namenode.http-address properties, which provide specific connection details to the RPC, and web services respectively for each NameNode.

Following Shows examples of above mentioned properties: In core-site.xml we set  fs.defaultFS property to hdfs://testhacluster Following property values are set in hdfs-site.xml property>   <name>dfs.nameservices</name>   <value>testhacluster</value> </property> <property>   <name>dfs.ha.namenodes.testhacluster</name>   <value>nn1,nn2</value> </property> <property>   <name>dfs.namenode.rpc-address.testhacluster.nn1</name>   <value>namenode1:8020</value> </property> <property>   <name>dfs.namenode.rpc-address.testhacluster.nn2</name>   <value>namenode2:8020</value> </property> <property>   <name>dfs.namenode.http-address.testhacluster.nn1</name>   <value>namenode1:50070</value> </property> <property>   <name>dfs.namenode.http-address.testhacluster.nn2</name>   <value>namenode2:50070</value> </property> <property>   <name>dfs.namenode.shared.edits.dir</name>   <value>qjournal://journalnode1:8485/testhacluster</value> </property> <property>   <name>dfs.journalnode.edits.dir</name>   <value>/tmp/dfs/jn</value> </property> <property>   <name>dfs.client.failover.proxy.provider.testhacluster</name>   <value>org.apache.hadoop.hdfs.server.namenode.ha.Configured FailoverProxyProvider</value> </property> <property>   <name>dfs.ha.fencing.methods</name>   <value>shell(/bin/true)</value> </property> <property>   <name>dfs.ha.automatic-failover.enabled</name>   <value>false</value> </property

HDFS Rack Awareness

HDFS Rack Awareness

What is Rank Awareness? Rack awareness enables HDFS to understand a cluster topology that may include multiple racks of servers or multiple data centers, and to orchestrate its block placement accordingly. This allows us to achieve the goals of Data locality, fault tolerance, and resiliency .   Advantage of Rank Awareness: Implementing a rack topology and using rack awareness in HDFS, data can be dispersed across racks or data centers to provide further fault tolerance in the event of a rack failure, switch or network failure, or even a data center outage.   With a rack topology defined, Hadoop will place blocks on nodes according to the following strategy: The first replica of a block is placed on a node in the cluster. This is the same node as the Hadoop client if the client is running within the cluster. The second replica of a block is placed on a node residing on a different rack from the first replica. The third replica, assuming a default replication factor of 3, is placed on a different node on the same rack as the second replica. How is Rank Awareness implemented: Rack awareness is implemented using a user-supplied script, which could be in any scripting language available on the cluster. Common scripting languages used include Python, Ruby, or BASH. The script, called a rack topology script, provides Hadoop with an identifier for any given node, telling Hadoop to which rack that particular node belongs. A rack could be a physical rack (e.g., a 19” server rack) or a particular subnet, or even an abstraction representing a data center. The script needs to return a hierarchical location ID for a host passed in as a script argument. The hierarchical location ID is in the form /datacenter/rack. This can be pseudo-coded as follows: topology_script([datanode_hosts]) -> [rackids] What happens when there is no topology script: Rack awareness is still implemented if a rack topology script is not supplied. In that case, all nodes have a default location ID of /default-rack. The script can be implemented in several different ways with no strict definition of how to accomplish this. For instance, you could embed the location ID in the hostname itself and use simple string manipulation to determine the location, as in the following example: dc01rack01node02. You could also create a lookup table or map and store this in a file or in a database. Where is the topology script located: Rack awareness is implemented on the HDFS client or client application, so the script, its chosen interpreter, and any supporting data such as lookup files need to be available on the client. A sample rack topology script is provided below #!/usr/bin/env python # input: hostname or ipaddress of a node or nodes # output: rack id for each node # example: # input of dc01rack01node02 # outputs /dc01/rack01 import sys DEFAULT_RACK = "/dc01/default-rack" for host in sys.argv[1:]:   if len(host) == 16:    dcid = host[:4]    rackid = host[4:10]    print "/" + dcid + "/" + rackid   else:    print DEFAULT_RACK This script is enabled using the net.topology.script.file.name configuration property in the core-site.xml configuration <property>   <name>net.topology.script.file.name</name>   <value>/etc/hadoop/conf/rack-topology-script.py</value> </property>

Configuration files for hadoop eco-system components

Configuration files for hadoop eco-system components

Hive: Hive will usually inherit its HDFS and YARN configuration from the Hadoop configuration files just spoken about (core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml). However, many other properties specific to Hive are maintained in the hive-site.xml configuration file, typically located in /etc/hive/conf. The most significant settings in hive-site.xml are the connection details to a shared Hive metastore Pig: As with Hive, most HDFS- and YARN-specific configuration details are sourced from the Hadoop configuration on the host, typically a client including the Pig and Hadoop client libraries. Additional Pig-specific properties are located in the pig.properties file, which is in /etc/pig/conf on most distributions.  Note that the pig.properties file, unlike many of the other Hadoop configuration files, is not an XML document. Spark: Spark configuration properties are set through the spark-defaults.conf file located in $SPARK_HOME/conf. This configuration file is read by Spark applications and daemons upon startup. The spark-defaults.conf file, like pig.properties, is also not in the standard Hadoop XML configuration format.Spark configuration properties can also be set programmatically in your driver code using the SparkConf object Hbase: HBase configuration is typically stored in /etc/hbase/conf, the primary configuration file being the hbase-site.xml file. This will govern the behavior of HBase and will be used by the HMaster and RegionServers alike.

core-site.xml and hdfs-site.xml Imp properties

HDFS Configuration Parameters(Detailed):

Common Properties (core-site.xml)

Used to store many properties that are required by different processes in a cluster, including client processes, master node processes, and slave node processes. fs.defaultFS fs.defaultFS is the property that specifies the filesystem to be used by the cluster. Most of the time this is HDFS. However, the filesystem used by the cluster could be an object store, such as S3 or Ceph, or a local or network filesystem. Note that the value specifies the filesystem scheme as well as hostname and port information for the filesystem. When using filesystems such as S3, you will need to provide additional AWS authentication parameters, including fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey, to supply your credentials to the remote platform. This may also be the case with other remote filesystems or object stores. When high-availability and federation is used fs.defaultFS will use a construct called nameservices instead of specifying hosts explicitly. Example: <property>   <name>fs.defaultFS</name>   <value>hdfs://mynamenode:8020</value> </property>

hdfs-site.xml

dfs.namenode.name.dir and dfs.namenode.edits.dir   The dfs.namenode.name.dir  property specifies the location on the filesystem where the NameNode stores its on-disk metadata, specifically the fsimage file(s).This value is also used as the default directory for the edits files used for the NameNodes journalling function. However, this location can be set to a different directory using the dfs.namenode.edits.dir property. <property>   <name>dfs.namenode.name.dir</name>   <value>file:///disk1/dfs/nn,file:///disk2/dfs/nn</value> </property> Note that there are no spaces between the comma-delimited values. Example value is /opt/app/daya01/hdfs/nn A loss of the NameNode’s metadata would result in the loss of all of the data stored in the cluster’s distributed filesystem—potentially petabytes of data. The on-disk metadata structures are there to provide durability and crash consistency for the NameNode’s metadata, which is otherwise stored in volatile memory on the NameNode host.   The value for the dfs.namenode.name.dir property is a comma-separated list of directories (on a local or network file system—not on HDFS) where the fsimage files and edits files by default will be stored by default. The default value for dfs.namenode.name.dir is file://${hadoop.tmp.dir}/dfs/name. This should be changed on a production system. The NameNodes metadata is written to each directory in the comma-separated list of directories specified by dfs.namenode.name.dir synchronously. If any directory or volume specified in the list is unavailable, it will be removed from the cached list of directories and no further attempts will be made to write to this directory until the NameNode is restarted. The parallel write operations to multiple directories provide additional fault tolerance to the NameNode’s critical metadata functions. For this reason, you should always provide more than one directory, residing on a different physical disk, volume, or disk controller to minimize the risk of outages if one volume or channel fails. In some cases such as non-HA deployments should, specify an NFS mount point in the list of directories in dfs.namenode.name.dir, which will then store a copy of the metadata on another host, providing further fault tolerance and recovery options. Note that if you do this, you should soft mount and configure retries for the NFS mount point; otherwise the NameNode process may hang if the mount is temporarily unavailable for whatever reason. The dfs.namenode.name.dir and dfs.namenode.edits.dir properties are read by the NameNode daemon upon startup. Any changes to these properties will require a NameNode restart to take effect.

dfs.namenode.checkpoint.dir/period/txns   There are three significant configuration properties that relate to the checkpointing function.   The dfs.namenode.checkpoint.dir property is a comma-separated list of directories, analogous to the dfs.namenode.name.dir property discussed earlier, used to store the temporary edits to merge during the checkpointing process. The dfs.namenode.checkpoint.period property specifies the maximum delay between two consecutive checkpoints with a default value of one hour. The dfs.namenode.checkpoint.txns property specifies the number of transactions at which the NameNode will force a checkpoint. Checkpointing will occur when either threshold is met (time interval or transactions).

dfs.datanode.data.dir The dfs.datanode.data.dir is the property that specifies where the DataNode will store the physical HDFS blocks.  Like the dfs.namenode.name.dir property this too is a comma-separated list of directories with no spaces between. Unlike the dfs.namenode.name.dir setting, writes to the directories specified in the dfs.datanode.data.dir property are performed in a round-robin fashion (i.e., the first block on one directory, the next block on the next, and so on). The configuration for each DataNode may differ . On different slave nodes, the volumes and directories may differ. However, when planning your cluster, it is best to try to homogenize as many configuration settings as possible, making the cluster easier to manage.

dfs.datanode.du.reserved HDFS is a “greedy” file system. Volumes associated with directories on the DataNodes specified by dfs.datanode.data.dir will be 100% filled with HDFS block data if left unmanaged. This is problematic because slave nodes require working space for intermediate data storage. If available disk storage is completely consumed by HDFS block storage, data locality suffers as processing activities may not be possible on the node. The dfs.datanode.du.reserved configuration property in hdfs-site.xml specifies the amount of space in bytes on each volume that must be reserved, and thus cannot be used for HDFS block storage. It is generally recommended to set this value to 25% of the available space or at least 1 GB, depending upon the local storage on each DataNode

dfs.blocksize The property dfs.blocksize specifies the block size in bytes for new files written by clients, including files produced as the result of an application or job run by the client. The default is 134217728, or 128MB. Although commonly thought of as a cluster- or server-based setting, dfs.blocksize is actually a client setting.   This property can be influenced by administrators using the <final> tag on a server node as discussed earlier

dfs.replication   The dfs.replication property located in the hdfs-site.xml file determines the number of block replicas created when a file is written to HDFS.   The default is 3, which is also the recommended value.   As with the dfs.blocksize property, the dfs.replication property is a client-side setting

Examples of above Properties:

<property>       <name>dfs.namenode.name.dir</name>       <value>/opt/app/data01/hdfs/nn</value>     </property>    <property>       <name>dfs.namenode.checkpoint.dir</name>       <value>/opt/app/data01/hdfs/snn</value>     </property>     <property>       <name>dfs.namenode.checkpoint.edits.dir</name>       <value>${dfs.namenode.checkpoint.dir}</value>     </property>     <property>       <name>dfs.namenode.checkpoint.period</name>       <value>21600</value>     </property>     <property>       <name>dfs.namenode.checkpoint.txns</name>       <value>10000000</value>     </property>    <property>       <name>dfs.datanode.data.dir</name>       <value>/opt/data/data01/hdfs/dn,/opt/data/data02/hdfs/dn,/opt/data/data03/hdfs/dn,/opt/data/data04/hdfs/dn,/opt/data/data05/hdfs/dn,/opt/data/data06/hdfs/dn,/opt/data/data07/hdfs/dn,/opt/data/data08/hdfs/dn,/opt/data/data09/hdfs/dn,/opt/data/data10/hdfs/dn,/opt/data/data11/hdfs/dn,/opt/data/data12/hdfs/dn</value>    </property> <property>       <name>dfs.datanode.du.reserved</name>       <value>1073741824</value>     </property>     <property>       <name>dfs.blocksize</name>       <value>134217728</value>     </property>     <property>       <name>dfs.replication</name>       <value>3</value>     </property>     <property>       <name>dfs.replication.max</name>       <value>50</value>     </property>

Other Hadoop Environment Scripts and Configuration Files

Other Hadoop Environment Scripts and Configuration Files

We are familiar with many configuration files like core-site.xml,yarn-site.xml,mapred-site.xml,and hdfs-site.xml. Apart from these there are some more configuration files in the configuration directory.

[sukul@server1 ~]$ cd $HADOOP_HOME/conf

[sukul@server1 conf]$ ls *xml

capacity-scheduler.xml  hadoop-policy.xml  mapred-site.xml  ssl-server.xml

core-site.xml           hdfs-site.xml      ssl-client.xml   yarn-site.xml

[sukul@server1 conf]$ ls *sh

hadoop-env.sh  mapred-env.sh  yarn-env.sh

A] hadoop-env.sh/yarn-env.sh/mapred-env.sh

• The hadoop-env.sh is used to source environment variables for Hadoop daemons and processes.
• This can include daemon JVM settings such as heap size or Java options, as well as basic variables required by many processes such as HADOOP_LOG_DIRorJAVA_HOME (Following shows just few lines of the hadoop-env.sh script)

[sukul@server1 conf]$ cat hadoop-env.sh  | grep -v '^#' | sed '/^$/d' export JAVA_HOME=/opt/app/java/jdk/jdk180/ export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.6.5.4-1/hadoop} export JSVC_HOME=/usr/lib/bigtop-utils export HADOOP_HEAPSIZE="4096" export HADOOP_NAMENODE_INIT_HEAPSIZE="-Xms233472m" export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS}" export HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/opt/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=25600m -XX:MaxNewSize=25600m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -Xloggc:/opt/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms233472m -Xmx233472m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,RFAAUDIT ${HADOOP_NAMENODE_OPTS}"

• Basically, if you need to pass any environment variables to any Hadoop process, thehadoop-env.shfile is the file to do this in, as it is sourced by all Hadoop control scripts. 
• Similarly, there may be other environment shell scripts such as yarn-env.shandmapred-env.sh that are used by these specific processes to source necessary environment variables. (Following shows just few lines of the mapred-env.sh and yarn-env.sh scripts)

[sukul@server1 conf]$ cat mapred-env.sh  | grep -v '^#' | sed '/^$/d' export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=16384 export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA export HADOOP_JOB_HISTORYSERVER_OPTS=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/opt/log/hadoop-mapreduce/mapred/gc_trace.log -XX:ErrorFile=/opt/log/hadoop-mapreduce/mapred/java_error.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/log/hadoop-mapreduce/mapred/heap_dump.hprof export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"

[sukul@server1 conf]$ cat yarn-env.sh  | grep -v '^#' | sed '/^$/d' export HADOOP_YARN_HOME=/usr/hdp/2.6.5.4-1/hadoop-yarn export YARN_LOG_DIR=/opt/log/hadoop-yarn/$USER export YARN_PID_DIR=/var/run/hadoop-yarn/$USER export HADOOP_LIBEXEC_DIR=/usr/hdp/2.6.5.4-1/hadoop/libexec export JAVA_HOME=/opt/app/java/jdk/jdk180/ export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn} export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}" if [ "$JAVA_HOME" != "" ]; then   #echo "run java in $JAVA_HOME"   JAVA_HOME=$JAVA_HOME fi if [ "$JAVA_HOME" = "" ]; then   echo "Error: JAVA_HOME is not set."   exit 1 fi

B] log4j.properties

• Hadoop usesLog4J(the Java logging framework) to store and manage its log files. Log files are produced by nearly every process in Hadoop, including daemons, applications, and tasks. 
• Thelog4j.propertiesfile provides configuration for log file management, including how to write log records, where to write them, and how to manage rotation and retention of log files.
• Following shows sample log4j.properties file:

[sukul@server1 ~]$ cd $HADOOP_HOME/conf [sukul@server1 conf]$ ls log4j.properties log4j.properties [sukul@server1 conf]$ cat log4j.properties | grep -v '^#' | sed '/^$/d' hadoop.root.logger=INFO,console hadoop.log.dir=. hadoop.log.file=hadoop.log log4j.rootLogger=${hadoop.root.logger}, EventCounter log4j.threshhold=ALL log4j.appender.RFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file} log4j.appender.RFA.DatePattern=.yyyy-MM-dd log4j.appender.RFA.MaxBackupIndex=45 log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n hadoop.tasklog.taskid=null hadoop.tasklog.iscleanup=false hadoop.tasklog.noKeepSplits=4 hadoop.tasklog.totalLogFileSize=100 hadoop.tasklog.purgeLogSplits=true hadoop.tasklog.logsRetainHours=12 log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender log4j.appender.TLA.taskId=${hadoop.tasklog.taskid} log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup} log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize} log4j.appender.TLA.layout=org.apache.log4j.PatternLayout log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n hadoop.security.logger=INFO,console hadoop.security.log.maxfilesize=256MB hadoop.security.log.maxbackupindex=20 log4j.category.SecurityLogger=WARN,console hadoop.security.log.file=SecurityAuth.audit log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd log4j.appender.RFAS=org.apache.log4j.RollingFileAppender log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize} log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex} hdfs.audit.logger=INFO,console log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.RFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.RFAAUDIT.MaxBackupIndex=180 log4j.appender.RFAAUDIT.MaxFileSize=16106127360 mapred.audit.logger=INFO,console log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger} log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd hadoop.metrics.log.level=INFO log4j.logger.org.apache.hadoop.metrics2=${hadoop.metrics.log.level} log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN log4j.logger.BlockStateChange=ERROR log4j.logger.org.apache.hadoop.hdfs.StateChange=WARN yarn.log.dir=. hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger} hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender yarn.server.resourcemanager.appsummary.log.file=hadoop-mapreduce.jobsummary.log yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger} log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender log4j.appender.RMSUMMARY.File=${yarn.log.dir}/${yarn.server.resourcemanager.appsummary.log.file} log4j.appender.RMSUMMARY.MaxFileSize=256MB log4j.appender.RMSUMMARY.MaxBackupIndex=20 log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.JSA.layout=org.apache.log4j.PatternLayout log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n log4j.appender.JSA.DatePattern=.yyyy-MM-dd log4j.appender.JSA.layout=org.apache.log4j.PatternLayout log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false

• In some cases different components may have their own specificlog4j.propertiesfile, which may be located in the Hadoop configuration directory, such askms-log4j.propertiesandhttpfs-log4j.properties. 

C] hadoop-metrics.properties

• We may also have hadoop-metrics.properties and/or hadoop-metrics2.propertiesfiles in your Hadoop configuration directory. These are used to define application and platform metrics to collect. Following are the sample hadoop-metrics.properties

[sukul@server1 conf]$ cat hadoop-metrics2.properties | grep -v '^#' | sed '/^$/d' *.period=10 *.sink.timeline.plugin.urls=file:///usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink.jar *.sink.timeline.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink *.sink.timeline.period=10 *.sink.timeline.sendInterval=60000 *.sink.timeline.slave.host.name=serv084.zbc.xyz.com *.sink.timeline.zookeeper.quorum=serv269.zbc.xyz.com:2181,serv271.zbc.xyz.com:2181,serv267.zbc.xyz.com:2181 *.sink.timeline.protocol=http *.sink.timeline.port=6188 *.sink.timeline.truststore.path = /etc/security/clientKeys/all.jks *.sink.timeline.truststore.type = jks *.sink.timeline.truststore.password = bigdata datanode.sink.timeline.collector.hosts=serv287.zbc.xyz.com namenode.sink.timeline.collector.hosts=serv287.zbc.xyz.com resourcemanager.sink.timeline.collector.hosts=serv287.zbc.xyz.com nodemanager.sink.timeline.collector.hosts=serv287.zbc.xyz.com jobhistoryserver.sink.timeline.collector.hosts=serv287.zbc.xyz.com journalnode.sink.timeline.collector.hosts=serv287.zbc.xyz.com maptask.sink.timeline.collector.hosts=serv287.zbc.xyz.com reducetask.sink.timeline.collector.hosts=serv287.zbc.xyz.com applicationhistoryserver.sink.timeline.collector.hosts=serv287.zbc.xyz.com resourcemanager.sink.timeline.tagsForPrefix.yarn=Queue

D] Other Configuration Files:   

• Slaves files: Used by the cluster startup scripts in the Hadoop sbin directory. This contains list of slave nodes.

• hadoop-policy.xml,kms-site.xml, orssl-server.xml: configuration files related to security or access control policies, SSL configuration, or key management