大数据产品管理平台Apache Ambari研究

1、环境准备1.1、配置时间同步

centos7开始使用chrony进行始终同步 安装chrony并配置始终同步 设置为开机启动

yum -y install chrony? #系统默认已经安装 如未安装 请执行以下命令安装

1.2、配置主机名、映射设置主机名

[root cdh1 ~]hostnamectl set-hostname hdp1
[root cdh2 ~]hostnamectl set-hostname hdp3
[root cdh3 ~]hostnamectl set-hostname hdp5

设置主机映射

vi /etc/hosts 添加以下内容【每台都需要】
10.10.101.1 hdp1
10.10.101.3 hdp3
10.10.101.5 hdp5

1.3、取消大叶内存

注意 【所有节点都需要】

sysctl -w vm.swappiness 0 echo vm.swappiness 0 /etc/sysctl.conf echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabled

1.4、数据库驱动配置

注意 【所有节点都需要】

第一步 重命名Mysql驱动包 把版本号去掉

mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar

第二步 将jar包移到java共享目录下

mv mysql-connector-java.jar /usr/share/java/

第三步 将Mysql驱动包分发到另外两台服务器

scp mysql-connector-java.jar admin 10.10.101.3:/home/admin/
scp mysql-connector-java.jar admin 10.10.101.5:/home/admin/
sudo mkdir /usr/share/java/
sudo cp /home/admin/mysql-connector-java.jar /usr/share/java/

1.5、安装JDK【/etc/profile】

注意 要设置java环境变量 每个机器都要配置

从10.10.101.4上scp目录 /usr/local/jdk1.8.0_112 可以自行下载解压

cd /usr/local/设置软连接 sudo ln -s jdk1.8.0_112 jdk设置环境变量 sudo vi /etc/profileexport JAVA_HOME /usr/local/jdkexport PATH $PATH:$JAVA_HOME/binsource /etc/profile1.6、免密

切换root sudo -i -uroot

查看密钥 cat ~/.ssh/id_rsa

生成密钥 ssh-keygen

ssh-copy-id hdp1

?

ssh-copy-id -i

1.7、关闭防火墙

查看防火墙状态 firewall-cmd -state

停止firewall?

systemctl stop firewalld.service

禁止firewall开机启动

systemctl disable firewalld.service

1.8、关闭selinux获取当前selinux状态 getenforceEnforcing为开启Disabled为关闭sudo vim /etc/sysconfig/selinux替换 SELINUX disabled重启 reboot1.9、设置unlimit参数

注意 【所有节点】

官方建议大于等于10000

cd /etc/security/limits.d/
sudo vi hadoop.conf

输入以下内容

* soft noproc 65535
* hard noproc 65535
* soft nofile 65535
* hard nofile 65535

退出当前会话 重新登录生效

?

2、制作本地yum源

使用【10.10.101.251】,我的实验环境服务器 需替换。

2.1、安装包准备http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari-2.6.2.2-centos7.tar.gzhttp://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gzhttp://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/HDP-GPL-2.6.5.0-centos7-gpl.tar.gz2.2、yum文件准备

下载方式

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari.repohttp://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/hdp.repohttp://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/hdp.gpl.repo

本地方式

在各自的安装包中存在。比如 ambari中 ambari/centos7/2.6.2.2-1/ambari.repo

2.3、安装http

yum install httpd
systemctl start httpd
systemctl enable httpd

2.4、制作源

mkdir /var/www/html/ambari/HDP -P
mkdir /var/www/html/ambari/HDP-UTILS
cd

2.5、编辑yum文件

cd /etc/yum.repos.d
sudo vi ambari.repo
#VERSION_NUMBER 2.6.2.2-1
[ambari-2.6.2.2]
name ambari Version - ambari-2.6.2.2
baseurl http://10.10.101.251/hdp/ambari/centos7/2.6.2.2-1/
gpgcheck 1
gpgkey http://10.10.101.251/hdp/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled 1
priority 1

sudo vi hdp.repo
#VERSION_NUMBER 2.6.5.0-292
[HDP-2.6.5.0]
name HDP Version - HDP-2.6.5.0
baseurl http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/
gpgcheck 1
gpgkey http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled 1
priority 1
[HDP-UTILS-1.1.0.22]
name HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck 1
gpgkey http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled 1
priority 1

?

sudo vi hdp.gpl.repo
#VERSION_NUMBER 2.6.5.0-292
[HDP-GPL-2.6.5.0]
name HDP-GPL Version - HDP-GPL-2.6.5.0
baseurl http://10.10.101.251/hdp/HDP-GPL/centos7/2.6.5.0-292/
gpgcheck 1
gpgkey http://10.10.101.251/hdp/HDP-GPL/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled 1
priority 1

3、数据库准备

此处不再赘述 实验环境使用存在的数据库 ambari_dianxin

地址 10.10.101.10User ambari_dianxinPwd ambari_dianxin进入数据库 mysql -uambari_dianxin -pambari_dianxin4、安装ambari-server下载

sudo yum install -y ambari-server

设置

sudo ambari-server setup
[admin hdp1 yum.repos.d]$ sudo ambari-server reset
Using python /usr/bin/python
Resetting ambari-server
**** WARNING **** You are about to reset and clear the Ambari Server database. This will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard.
Are you SURE you want to perform the reset [yes/no] (no)? yes
ERROR: Exiting with exit code 1.
REASON: Ambari doesn t support resetting exernal DB automatically. To reset Ambari Server schema you must first drop and then create it using DDL scripts from /var/lib/ambari-server/resources/ 
[admin hdp1 yum.repos.d]$ sudo ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is disabled 
Ambari-server daemon is configured to run under user ambari . Change this setting [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)?
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
Enter choice (3):
Hostname (10.10.101.10):
Port (3309):
Database name (ambari_dianxin):
Username (ambari_dianxin):
Enter Database Password (ambari_dianxin):
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server setup completed successfully.

执行sql 由于sql文件与数据库不在一个节点 所以需要转发过去。

scp /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql admin 10.10.101.10:/home/admin/

执行ambari的建表语句

source /home/admin/Ambari-DDL-MySQL-CREATE.sql

启动服务

ambari-server start

成功提示 Ambari Server start completed successfully在界面验证 http://10.10.1.1:8080 账户 admin 密码 admin失败查看日志 /var/log/ambari-server/ambari-server.log5、搭建集群

登录Ambari界面 http://10.10.1.1:8080

输入地址 IP是我的实验环境服务器 需替换

http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/

http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/

查看密钥

cat ~/.ssh/id_rsa

然后输入到文本框

如果这一步出现如下错误 通过以下方式解决

错误

ERROR 2018-05-30 00:12:25,280 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-05-30 00:12:25,280 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

解决方式

1.修改/etc/python/cert-verification.cfg配置文件 
# vim /etc/python/cert-verification.cfg
[https]
verify disable
2、编辑 /etc/ambari-agent/conf/ambari-agent.ini 配置文件 在 [security] 节部分 确保设置如下两个值 其它值保持不变 
 [root ambari ~]# vi /etc/ambari-agent/conf/ambari-agent.ini
 [security]
 ssl_verify_cert 0
 force_https_protocol PROTOCOL_TLSv1_2
保存退出 重启 ambari-agent 
 [root ambari ~]# ambari-agent restart

若还是不能注册ambari-agent,使用下面介绍的方式。

如果上面方式还不能注册 说明jdk版本不对 使用默认oracle啊jdk 2.6.2.2版本ambari需要使用java version 1.8.0_112

如果一个节点重启后agent无法连接的话解决方法如下

ambari管理大数据集群 节点失去心跳 操作方法 
1、systemctl stop ambari-agent
2、在失去心跳节点打开配置
vi /etc/ambari-agent/conf/ambari-agent.ini
在[security] 下添加
force_https_protocol PROTOCOL_TLSv1_2
3、关闭状态
vi /etc/python/cert-verification.cfg
[https]
verify disable
4、systemctl stop ambari-agent

5.1、安装HDP

选择服务 全部为默认配置【除开密码配置以及数据库配置】

如果测试的时候报错 则设置mysql驱动

ambari-server setup --jdbc-db mysql --jdbc-driver /usr/share/java/mysql-connector-java.jar

5.2、设置oozie

等待安装即可。

6、卸载Ambari

停服务

sudo ambari-agent stop
sudo ambari-server stop

卸载所有组件

sudo yum remove -y hadoop_2* hdp-select* ranger_2* zookeeper* bigtop*atlas-metadata* ambari* spark* slide* strom* hive* oozie_2*

如有遗漏自行添加

查看ambari是否被卸载

rpm -qa|grep ambari
rpm -e ambari-server-2.6.2.2-1.x86_64

删除目录

sudo rm -rf /var/lib/ambari*
sudo rm -rf /usr/lib/python2.6/site-packages/ambari_*
sudo rm -rf /usr/lib/python2.6/site-packages/resource_management
sudo rm -rf /usr/lib/ambari-*
sudo rm -rf /etc/ambari-*
sudo rm -rf /etc/hadoop
sudo rm -rf /etc/hbase
sudo rm -rf /etc/hive
sudo rm -rf /etc/hive2
sudo rm -rf /etc/oozie
sudo rm -rf /etc/sqoop
sudo rm -rf /etc/zookeeper
sudo rm -rf /etc/flume
sudo rm -rf /etc/storm
sudo rm -rf /etc/tez_hive2
sudo rm -rf /etc/spark2
sudo rm -rf /etc/phoenix
sudo rm -rf /etc/pig
sudo rm -rf /etc/hive-hcatalog
sudo rm -rf /etc/tez
sudo rm -rf /etc/falcon
sudo rm -rf /etc/knox
sudo rm -rf /etc/hive-webhcat
sudo rm -rf /etc/kafka
sudo rm -rf /etc/slider
sudo rm -rf /etc/storm-slider-client
sudo rm -rf /etc/spark
sudo rm -rf /var/run/spark
sudo rm -rf /var/run/hadoop
sudo rm -rf /var/run/hbase
sudo rm -rf /var/run/zookeeper
sudo rm -rf /var/run/flume
sudo rm -rf /var/run/storm
sudo rm -rf /var/run/webhcat
sudo rm -rf /var/run/hadoop-yarn
sudo rm -rf /var/run/hadoop-mapreduce
sudo rm -rf /var/run/kafka
sudo rm -rf /var/run/hive 
sudo rm -rf /var/run/oozie 
sudo rm -rf /var/run/sqoop 
sudo rm -rf /var/run/hive-hcatalog
sudo rm -rf /var/run/falcon 
sudo rm -rf /var/run/hadoop-hdfs 
sudo rm -rf /var/run/ambari-metrics-collector
sudo rm -rf /var/run/ambari-metrics-monitor 
sudo rm -rf /var/log/hadoop-hdfs 
sudo rm -rf /var/log/hive-hcatalog
sudo rm -rf /var/log/ambari-metrics-monitor
sudo rm -rf /var/log/hadoop
sudo rm -rf /var/log/hbase
sudo rm -rf /var/log/flume
sudo rm -rf /var/log/sqoop
sudo rm -rf /var/log/ambari-server
sudo rm -rf /var/log/ambari-agent
sudo rm -rf /var/log/storm
sudo rm -rf /var/log/hadoop-yarn
sudo rm -rf /var/log/hadoop-mapreduce
sudo rm -rf /var/log/knox
sudo rm -rf /var/lib/slider
sudo rm -rf /var/lib/pgsql/
sudo rm -rf /usr/lib/flume
sudo rm -rf /usr/lib/storm
sudo rm -rf /var/lib/hive
sudo rm -rf /var/lib/oozie
sudo rm -rf /var/lib/flume
sudo rm -rf /var/lib/hadoop-yarn
sudo rm -rf /var/lib/hadoop-mapreduce
sudo rm -rf /var/lib/hadoop-hdfs
sudo rm -rf /var/lib/zookeeper
sudo rm -rf /var/lib/knox
sudo rm -rf /var/log/hive
sudo rm -rf /var/log/oozie
sudo rm -rf /var/log/zookeeper
sudo rm -rf /var/log/falcon
sudo rm -rf /var/log/webhcat
sudo rm -rf /var/log/spark
sudo rm -rf /var/tmp/oozie
sudo rm -rf /tmp/ambari-qa
sudo rm -rf /tmp/hive
sudo rm -rf /var/hadoop
sudo rm -rf /hadoop/falcon
sudo rm -rf /tmp/hadoop
sudo rm -rf /tmp/hadoop-hdfs
sudo rm -rf /usr/hdp
sudo rm -rf /usr/hadoop
sudo rm -rf /opt/hadoop
sudo rm -rf /tmp/hadoop
sudo rm -rf /var/hadoop
sudo rm -rf /hadoop
sudo rm -rf /usr/bin/worker-lanucher
sudo rm -rf /usr/bin/zookeeper-client
sudo rm -rf /usr/bin/zookeeper-server
sudo rm -rf /usr/bin/zookeeper-server-cleanup
sudo rm -rf /usr/bin/yarn
sudo rm -rf /usr/bin/storm
sudo rm -rf /usr/bin/storm-slider
sudo rm -rf /usr/bin/worker-lanucher
sudo rm -rf /usr/bin/storm
sudo rm -rf /usr/bin/storm-slider
sudo rm -rf /usr/bin/sqoop
sudo rm -rf /usr/bin/sqoop-codegen
sudo rm -rf /usr/bin/sqoop-create-hive-table
sudo rm -rf /usr/bin/sqoop-eval
sudo rm -rf /usr/bin/sqoop-export
sudo rm -rf /usr/bin/sqoop-help
sudo rm -rf /usr/bin/sqoop-import
sudo rm -rf /usr/bin/sqoop-import-all-tables
sudo rm -rf /usr/bin/sqoop-job
sudo rm -rf /usr/bin/sqoop-list-databases
sudo rm -rf /usr/bin/sqoop-list-tables
sudo rm -rf /usr/bin/sqoop-merge
sudo rm -rf /usr/bin/sqoop-metastore
sudo rm -rf /usr/bin/sqoop-version
sudo rm -rf /usr/bin/slider
sudo rm -rf /usr/bin/ranger-admin-start
sudo rm -rf /usr/bin/ranger-admin-stop
sudo rm -rf /usr/bin/ranger-kms
sudo rm -rf /usr/bin/ranger-usersync-start
sudo rm -rf /usr/bin/ranger-usersync-stop
sudo rm -rf /usr/bin/pig
sudo rm -rf /usr/bin/phoenix-psql
sudo rm -rf /usr/bin/phoenix-queryserver
sudo rm -rf /usr/bin/phoenix-sqlline
sudo rm -rf /usr/bin/phoenix-sqlline-thin
sudo rm -rf /usr/bin/oozie
sudo rm -rf /usr/bin/oozied.sh
sudo rm -rf /usr/bin/mapred
sudo rm -rf /usr/bin/mahout
sudo rm -rf /usr/bin/kafka
sudo rm -rf /usr/bin/hive
sudo rm -rf /usr/bin/hiveserver2
sudo rm -rf /usr/bin/hbase
sudo rm -rf /usr/bin/hcat
sudo rm -rf /usr/bin/hdfs
sudo rm -rf /usr/bin/hadoop
sudo rm -rf /usr/bin/flume-ng
sudo rm -rf /usr/bin/falcon
sudo rm -rf /usr/bin/beeline
sudo rm -rf /usr/bin/atlas-start
sudo rm -rf /usr/bin/atlas-stop
sudo rm -rf /usr/bin/accumulo

如有遗漏自行添加

如果安装过程中在某个组件报错 那么直接删除对应组件的所有东西 重新安装即可 比如 symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink.

删除安装包

sudo yum remove oozie_2* -y

找到oozie的rpm包

rpm -qa|grep oozie

删除包

sudo rpm -e oozie_2_6_5_0_292-4.2.0.2.6.5.0-292.noarch

如果报错 /var/tmp/rpm-tmp.YhTbCT: line 1: pushd: /usr/hdp/2.6.5.0-292/oozie: No such file or directory

先创建目录

sudo mkdir –p /usr/hdp/2.6.5.0-292/oozie

再删除

sudo rpm -e oozie_2_6_5_0_292-4.2.0.2.6.5.0-292.noarch

删除目录

sudo rm -rf /usr/hdp/current/oozie-client/
sudo rm -rf /usr/hdp/2.6.5.0-292/oozie
sudo rm -rf /etc/oozie/
sudo rm -rf /var/lib/oozie/
sudo rm -rf /var/log/oozie/
sudo userdel oozie
sudo rm -rf /home/oozie

THE END

本文转自网络，原文链接：https://developer.aliyun.com/article/784962
本站部分内容转载于网络，版权归原作者所有，转载之目的在于传播更多优秀技术内容，如有侵权请联系QQ/微信：153890879删除，谢谢！