HDFS3. X new feature – erasure code
EC (erasure code) is a coding technique that, prior to HDFS, was most widely used in inexpensive redundant array of disks (RAID) (RAID Introduction: RAID...
Read More
Category: hadoop
【Hadoop Learning 】HDFS basic principles
December 14, 2023
by Casey Bowater-Parkes
No Comments
HDFS Introduction HDSF {code... } features {code... } System structure Master and Slave structure. There are three roles: NameNode, SecondarynaNode and DataNode. NameNode {code... }...
Read More
Hadoop Tutorial – HDFS Client Development
December 14, 2023
by Samantha Rice
No Comments
The Hadoop environment Hadoop version is {code... } Client development introduces dependencies (using Maven) {code... } write code {code... If you want to run it...
Read More
The history of Dataphin, a platform for intelligent data construction and management: the origin
December 14, 2023
by Gina Harrell
No Comments
Brief introduction: The OneData methodology proposed by Alibaba helps enterprises to clarify the management ideas of the whole life cycle of data, and more, it...
Read More
Hadoop-5-HDFS for big data
December 14, 2023
by 張婷婷
No Comments
HDFS (Hadoop Distributed File System) is the core sub-project of Hadoop project. In the development of big data, massive data are stored and managed through...
Read More
A must-see for ETL engineers! Super practical task optimization and breakpoint execution scheme
December 14, 2023
by Bruce Moreno
No Comments
With the rapid development of the era of big data, enterprises need to store, calculate and analyze trillions of data every day, and at the...
Read More
Hadoop Learning Note 2: MapReduce IO Types & Files Slice
December 14, 2023
by Raghav Kaur
No Comments
What to do: Provide a set of interfaces (core classes: InputFormat, OutputFormat, Mapper, Reducer, Driver) that enable users to implement distributed computing tasks with custom...
Read More
HDFS – What is Metadata
December 14, 2023
by Dana Mullen
No Comments
When a file is uploaded to an HDFS server, it is divided into multiple blocks and stored as multiple copies on the server. How do...
Read More
(2) MapReduce principle analysis of Hadoop
December 14, 2023
by Tracy Crosby MD
No Comments
MapReduce is a programming framework for distributed computing programs, which is the core framework for users to develop "data analysis applications based on Hadoop". The...
Read More
HDFS – What about more and more files
December 14, 2023
by Hrishita Choudhary
No Comments
With the development of the business, the server will store more and more files, in the initial stage, we will directly through the expansion to...
Read More
(3) Understand how MapReduce works in a simple way
December 14, 2023
by Kay Waters
No Comments
In front of the written a \(two \)Hadoop MapReduce principle analysis, and then as if to read several times, even I feel too boring, even...
Read More
How is the HDFS-Checkpoint mechanism implemented
December 14, 2023
by Sarah Glenn
No Comments
We mentioned the CheckPoint mechanism, which basically merges multiple Edits files. NameNode is already under a lot of pressure, so it is not the NameNode...
Read More
HDFS – the stuff that writes data
December 14, 2023
by Bethan Connor
No Comments
As mentioned earlier, when a client needs to write data, it will tell the NameNode that it is uploading the file to a directory, and...
Read More
Deploy Cloudera Manager 5 and CDH 5.12.1 offline and use CDH to deploy Hadoop cluster services
December 14, 2023
by Mr. Denis Woodward
No Comments
CDH is short for Cloudera Distribution Hadoop. As the name implies, it is the version of Hadoop published by Cloudera, which encapsulates Apache Hadoop and...
Read More
SQOOP learning summary
December 14, 2023
by Kristina Walker
No Comments
SQOOP is an open source project of Apache. It belongs to the Hadoop family. The core function of SQOOP is to convert and transfer data...
Read More
(1)Hadoop MapReduce basic entry example WordCount detail
December 14, 2023
by Norman Chapman
No Comments
MapReduce is a computing framework, and since it's a framework for doing computation, the representation has an input. MapReduce operates this input and obtains an...
Read More
This article lowers the threshold of big data learning to the horizon
December 14, 2023
by Graham Curtis
No Comments
The idea is to chop up large chunks of data, make three copies of each piece and put them on three cheap machines, so that...
Read More
HDFS – How does the dual caching mechanism guarantee high concurrent requests for metadata
December 14, 2023
by Jorge Coffey
No Comments
As we mentioned in HDFS - What is Metadata, metadata is stored in memory and disk. Memory is used to improve response time, disk is...
Read More
How to improve the efficiency of Hadoop accessing Object Storage US3? We did these technical practices
December 14, 2023
by Jasmine Simmons
No Comments
In the era of big data with information explosion, how to solve the storage problem of massive data with lower cost has become an important...
Read More
HDFS – What is the security mode
December 14, 2023
by 梁筱涵
No Comments
High availability of HDFS-NameNode mentions that NameNode has active and standby states, and it also has another state, which is SafeMode, which is safemode. In...
Read More
CDH off-line setup
December 14, 2023
by Samantha Cragg
No Comments
CDH Offline Construction Actually, we have been engaged in CDH for a long time since 3 years ago. At that time, we also planned to...
Read More
Ambari environment setup (offline setup)
December 14, 2023
by Megan Miles
No Comments
Ambari is a top-tier open source project from the Apache Software Foundation. It is a tool for centrally deploying, managing, and monitoring Hadoop distributed clusters....
Read More
Kerberos-based Hadoop security cluster construction
December 14, 2023
by Jessica Elliott
No Comments
In my last job, I was mainly responsible for the construction of big data platform, during which I accumulated some notes on the construction and...
Read More
The hadoop tutorial – graphs
December 14, 2023
by Rhys Bruce-Griffin
No Comments
MapReduce is a computing model adopted by Hadoop for multi-node computing, and to put it more simply, it is a set of methodology for Hadoop...
Read More
Ambari2.6.0+HDP2.6.3 deploys the Hadoop cluster
December 14, 2023
by Gabriel Washington
No Comments
1. Modify hostname(all nodes) to take effect temporarily {code... } in perpetuity {code... } 2. Configure SSH free secret (all nodes) to generate a secret...
Read More
The Hadoop Framework: MapReduce Rationals and Primer Cases
December 13, 2023
by 李淑娟
No Comments
One of the core components of Hadoop: MapReduce, a distributed computing scheme, is a programming model for parallel operations of large-scale data sets, including Map...
Read More
High availability of HDFS-NameNode
December 13, 2023
by Aniruddh Sule
No Comments
We already know that when a file is read or uploaded, it needs to go through the NameNode. If the NameNode is down, the client...
Read More
Hadoop MapReduce Job Stuck for a Long Time
December 13, 2023
by Lori Lopez
No Comments
At present, we preprocess source data files from upstream by writing Hadoop MapReduce program. After the source data file is sent to the Hadoop cluster,...
Read More
How does HDFS-NameNode perceive the DataNode
December 13, 2023
by Sheila Davis
No Comments
In HDFS-HDFS, when the NameNode is started, it will open the RPC service, called ServicerPCServer. There are many protocols in this Server, one of which...
Read More
HDFS – The four roles of HDFS
December 13, 2023
by Rati Bajaj
No Comments
In the last article we have known that HDFS is to cut the file into blocks, and then stored in a number of servers, and...
Read More