big-data-europe/docker-hadoop-spark-workbench
Fork: 373 Star: 688 (更新于 2024-11-29 02:50:39)
license: 暂无
Language: Makefile .
[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.
How to use HDFS/Spark Workbench
To start an HDFS/Spark Workbench:
docker-compose up -d
docker-compose does not work to scale up spark-workers, for distributed setup see swarm folder
Starting workbench with Hive support
Before starting the next command, check that the previous service is running correctly (with docker logs servicename).
docker-compose -f docker-compose-hive.yml up -d namenode hive-metastore-postgresql
docker-compose -f docker-compose-hive.yml up -d datanode hive-metastore
docker-compose -f docker-compose-hive.yml up -d hive-server
docker-compose -f docker-compose-hive.yml up -d spark-master spark-worker spark-notebook hue
Interfaces
- Namenode: http://localhost:50070
- Datanode: http://localhost:50075
- Spark-master: http://localhost:8080
- Spark-notebook: http://localhost:9001
- Hue (HDFS Filebrowser): http://localhost:8088/home
Important
When opening Hue, you might encounter NoReverseMatch: u'about' is not a registered namespace
error after login. I disabled 'about' page (which is default one), because it caused docker container to hang. To access Hue when you have such an error, you need to append /home to your URI: http://docker-host-ip:8088/home
Docs
Count Example for Spark Notebooks
val spark = SparkSession
.builder()
.appName("Simple Count Example")
.getOrCreate()
val tf = spark.read.textFile("/data.csv")
tf.count()
Maintainer
- Ivan Ermilov @earthquakesan
Note: this repository was a part of BDE H2020 EU project and no longer actively maintained by the project participants.
最近版本更新:(数据更新于 2024-10-18 17:36:14)
big-data-europe/docker-hadoop-spark-workbench同语言 Makefile最近更新仓库
2024-03-15 23:00:57 breakings/OpenWrt
2022-09-22 04:08:49 teaxyz/white-paper
2022-05-11 23:01:18 YuzukiHD/Buildroot-YuzukiSBC
2021-05-26 08:57:35 jmeubank/tdm-gcc
2020-11-18 20:35:35 shadowsocks/openwrt-shadowsocks
2020-07-09 14:00:22 CrafterPika/reboot_to_hekate