Change the Hadoop MapReduce v2 (YARN) ShuffleHandler Port

Introduction

If you are running Hadoop on a development machine, then it’s likely that you’ll run into a situation where multiple services require port 8080. I recently ran into this issue where both the Pentaho User Console and the Hadoop MapReduce ShuffleHandler were trying to use port 8080.

One solution is to change the port used by the Hadoop MapReduce ShuffleHandler, which is what I’m going to configure below.

Configuration

sudo vi /etc/hadoop/conf/mapred-site.xml

Add the following as a new property by adding it just before the </configuration> element.

  <property>
    <name>mapreduce.shuffle.port</name>
    <value>8080</value>
    <description>Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers.</description>
  </property>

Then restart the YARN daemons.

Advertisements

One Comment

  1. […] There are many reasons for changing the Pentaho BI Server’s default port, but one of the most common is a conflict with another server. I ran into this issue where both the Pentaho User Console and the Hadoop MapReduce ShuffleHandler were trying to use port 8080. I wrote an earlier article on how to change the default port for Hadoop MapReduce ShuffleHandler. […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s