HADOOP discussion found on the web

HADOOP discussion

http://cloudcamp.pbworks.com/Hadoop

A room of attendees (est. > 20) attended this discussion on Hadoop. 2 people had extensive discussions: Chris Wensel (Concurrent, inc.) and Adrian Cockcroft (Netflix).

Summary of topics:

1. HDFS

a. Designed for write-once, read many (more appropriate for applications depending on this characteristic)
b. Data is access serially
c. Access has latency
d. Multiple Hadoop instances loose seek time
e. Hadoop is not a transaction system
2. Virtualization

a. Few companies doing Hadoop level virtualization (no data who’s doing this)

3. MAP-REDUCE

a. To use Hadoop, need to program in map-reduce, which might not be easy; may need 2-3 guys to write in map-reduce
b. Also generates lots of data
c. Concurrent offers an abstraction on top of map-reduce so programmers can think in terms of streams, can now use Groovy or Jython; also offer options to deal with node failure (whether to continue or fail immediately); also compatible with PIG
d. One hadoop implementation does not just have one map-reduce, but has 5, 10, 20 such programs to manage

4. Database on Hadoop?

a. PIG
b. Facebook HIVE
b. Business.com CloudBase
c. Endpoint reducer write to file system, People loading HBASE that way
d. HBASE is a big-table implementation in Apache (built by Powerset; Powerset was purchased by Microsoft; Powerset went dark for a while, but now back)
5. Running on Amazon

a. Amazon is opaque: not much control on servers, so rack-awareness feature in Hadoop not useful;

6. Performance
a. Block size at 128MB (Yahoo, Facebook)
b. Rack-awareness policy is fairly static; build tree of data center to rack to …
c. EC2’s biggest problem – store in S3 in small chucks, could impact HDFS block sizes
d. Some people are using HDFS rather than S3 on 1000-node EC2
e. Hadoop usage offers infinite file size, caching is setting attributes, paying additional penalty with preloading
f. Data replication of name node is an issue – higher replication -> increases latency; replication is pipelined
g. HDFS does not work well with lots of small files (name node gets crushed)
h. New Hadoop has a way to bundle a bunch of small files into one file and still have a way to seek into them to find the files
i. It takes a minute to start or stop Hadopo, as there is a high latency to starting JVMs – “If the time to start a JVM is significant for your application, then your problem is not big enough to use Hadoop”
j. Gating factor is bandwidth - gets I/O bound (even with data locality) still need to do the replication
k. Scales linearly as you increase in size
l. Set number of map and reduce tasks higher than number of cores
m. At the map-reduce levels, several schedulers / strategies are available
n. A big issue is run-time monitoring to avoid current situation of idle nodes in a hadoop system (e.g. reduce task could be waiting for map task, but sitting idle)
o. George Porter (Sun Labs) is working on sensing Xtrace to do run-time monitoring to accumulate tracing to optimize (you doing the optimization). Could be most interesting in debugging in hadoop application. Would not know about CPU utilization (that's where Ganglia data come in)
p. SSD could make sense for the MAP phase
q. Currently no data (simulation or otherwise) on where to partition compute vs disk vs I/O
r. Supercomputer is a computer that turns a compute-bound problem to an I/O bound problem

7. Usage of Hadoop

a. Log analysis
b. Powerset search
c. Metaweb uses Hadoop (heavy Python user)
d. Hadoop is pluggable (KFS, …)
e. Using Hypertable
f. Companies running 1K nodes - primarily using it for storage (likely using Amazon small node - cost is low)
g. Google sponsors a 1,065 (approx) node cluster (hosted at IBM) for research purposes; Given Hadoop job, many tenants (university) already… you ask for how many nodes you want, (e.g. Maryland asks for 40 nodes), control number of universities who get access to it. No stability in number of nodes you can get - One day, you could get 200 nodes, another 500 nodes. Problem is how you split up your data, you don't move it around.

8. Hadoop single point of failure

a. Hadoop name node is single node, no clustering; name node holds all metadata
b. Failure could take up to 30 minutes for a system with 100 million files
c. A lot of people want redundant name nodes, but no one is working on it
d. No logging in HDFS
e. Concurrent using THRIFT?, which is type safe, … THRIFT? Is a great archival system