sparkpdf

1. 《SparkinAction》pdf下载在线阅读，求百度网盘云资源

《Spark in Action》（Marko Bonaći）电子书网盘下载免费在线阅读

资源链接：

链接：

书名：Spark in Action

作者：Marko Bonaći

豆瓣评分：7.7

出版社：Manning

出版年份：2016-1

页数：400

内容简介：Working with big data can be complex and challenging, in part because of the multiple analysis frameworks and tools required. Apache Spark is a big data processing framework perfect for analyzing near-real-time streams and discovering historical patterns in batched data sets. But Spark goes much further than other frameworks. By including machine learning and graph processing capabilities, it makes many specialized data processing platforms obsolete. Spark's unified framework and programming model significantly lowers the initial infrastructure investment, and Spark's core abstractions are intuitive for most Scala, java, and Python developers.

Spark in Action teaches you to use Spark for stream and batch data processing. It starts with an introction to the Spark architecture and ecosystem followed by a taste of Spark's command line interface. You then discover the most fundamental concepts and abstractions of Spark, particularly Resilient Distributed Datasets (RDDs) and the basic data transformations that RDDs provide. The first part of the book also introces you to writing Spark applications using the the core APIs. Next, you learn about different Spark components: how to work with structured data using Spark SQL, how to process near-real time data with Spark Streaming, how to apply machine learning algorithms with Spark MLlib, how to apply graph algorithms on graph-shaped data using Spark GraphX, and a clear introction to Spark clustering.

作者简介：Marko Bonaći has worked with Java for 13 years. He currently works as IBM Enterprise Content Management team lead at SV Group. Petar Zečević is a CTO at SV Group. During the last 14 years he has worked on various projects as a Java developer, team leader, consultant and software specialist. He is the founder and, with Marko, organizer of popular Spark@Zg meetup group.

2. sparksession 作用域

Apache Spark 2.0引入了SparkSession，其为用户提供了一个统一的切入点来使用Spark的各项功能，并且允许用户通过它调用DataFrame和Dataset相关API来编写Spark程序。最重要的是，它减少了用户需要了解的一些概念，使得我们可以很容易地与Spark交互。
创建SparkSession
在2.0版本之前，与Spark交互之前必须先创建SparkConf和SparkContext，代码如下：

//set up the spark configuration and create contextsval sparkConf = new SparkConf().setAppName("SparkSessionZipsExample").setMaster("local")// your handle to SparkContext to access other context like SQLContextval sc = new SparkContext(sparkConf).set("spark.some.config.option", "some-value")val sqlContext = new org.apache.spark.sql.SQLContext(sc)
然而在Spark 2.0中，我们可以通过SparkSession来实现同样的功能，而不需要显式地创建SparkConf, SparkContext 以及 SQLContext，因为这些对象已经封装在SparkSession中。使用生成器的设计模式(builder design pattern)，如果我们没有创建SparkSession对象，则会实例化出一个新的SparkSession对象及其相关的上下文。
// Create a SparkSession. No need to create SparkContext// You automatically get it as part of the SparkSessionval warehouseLocation = "file:${system:user.dir}/spark-warehouse"val spark = SparkSession
.builder()
.appName("SparkSessionZipsExample")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
到现在我们可以使用上面创建好的spark对象，并且访问其public方法。
配置Spark运行相关属性
一旦我们创建好了SparkSession，我们就可以配置Spark运行相关属性。比如下面代码片段我们修改了已经存在的运行配置选项。
//set new runtime optionsspark.conf.set("spark.sql.shuffle.partitions", 6)
spark.conf.set("spark.executor.memory", "2g")//get all settingsval configMap:Map[String, String] = spark.conf.getAll()

获取Catalog元数据
通常我们想访问当前系统的Catalog元数据。SparkSession提供了catalog实例来操作metastore。这些方法放回的都是Dataset类型的，所有我们可以使用Dataset相关的API来访问其中的数据。如下代码片段，我们展示了所有的表并且列出当前所有的数据库：
//fetch metadata data from the catalog
scala> spark.catalog.listDatabases.show(false)+--------------+---------------------+--------------------------------------------------------+|name |description |locationUri |+--------------+---------------------+--------------------------------------------------------+|default |Default Hive database|hdfs://iteblogcluster/user/iteblog/hive/warehouse |+--------------+---------------------+--------------------------------------------------------+
scala> spark.catalog.listTables.show(false)+----------------------------------------+--------+-----------+---------+-----------+|name |database|description|tableType|isTemporary|+----------------------------------------+--------+-----------+---------+-----------+|iteblog |default |null |MANAGED |false ||table2 |default |null |EXTERNAL |false ||test |default |null |MANAGED |false |+----------------------------------------+--------+-----------+---------+-----------+

创建Dataset和Dataframe
使用SparkSession APIs创建 DataFrames 和 Datasets的方法有很多，其中最简单的方式就是使用spark.range方法来创建一个Dataset。当我们学习如何操作Dataset API的时候，这个方法非常有用。操作如下：
scala> val numDS = spark.range(5, 100, 5)
numDS: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> numDS.orderBy(desc("id")).show(5)+---+| id|+---+| 95|| 90|| 85|| 80|| 75|+---+only showing top 5 rows
scala> numDS.describe().show()+-------+------------------+|summary| id|+-------+------------------+| count| 19|| mean| 50.0|| stddev|28.136571693556885|| min| 5|| max| 95|+-------+------------------+scala> val langPercentDF = spark.createDataFrame(List(("Scala", 35), | ("Python", 30), ("R", 15), ("Java", 20)))
langPercentDF: org.apache.spark.sql.DataFrame = [_1: string, _2: int]

scala> val lpDF = langPercentDF.withColumnRenamed("_1", "language").withColumnRenamed("_2", "percent")
lpDF: org.apache.spark.sql.DataFrame = [language: string, percent: int]

scala> lpDF.orderBy(desc("percent")).show(false)+--------+-------+ |language|percent|+--------+-------+|Scala |35 ||Python |30 ||Java |20 ||R |15 |+--------+-------+

使用SparkSession读取CSV
创建完SparkSession之后，我们就可以使用它来读取数据，下面代码片段是使用SparkSession来从csv文件中读取数据：

val df = sparkSession.read.option("header","true").
csv("src/main/resources/sales.csv")

上面代码非常像使用SQLContext来读取数据，我们现在可以使用SparkSession来替代之前使用SQLContext编写的代码。下面是完整的代码片段：

package com.iteblogimport org.apache.spark.sql.SparkSession/*** Spark Session example**/object SparkSessionExample {def main(args: Array[String]) {val sparkSession = SparkSession.builder.master("local").appName("spark session example").getOrCreate()val df = sparkSession.read.option("header","true").csv("src/main/resources/sales.csv")df.show()}}

使用SparkSession API读取JSON数据
我们可以使用SparkSession来读取JSON、CVS或者TXT文件，甚至是读取parquet表。比如在下面代码片段里面，我将读取邮编数据的JSON文件，并且返回DataFrame对象：
// read the json file and create the dataframe
scala> val jsonFile = "/user/iteblog.json"
jsonFile: String = /user/iteblog.json
scala> val zipsDF = spark.read.json(jsonFile)
zipsDF: org.apache.spark.sql.DataFrame = [_id: string, city: string ... 3 more fields]

scala> zipsDF.filter(zipsDF.col("pop") > 40000).show(10, false)+-----+----------+-----------------------+-----+-----+|_id |city |loc |pop |state|+-----+----------+-----------------------+-----+-----+|01040|HOLYOKE |[-72.626193, 42.202007]|43704|MA ||01085|MONTGOMERY|[-72.754318, 42.129484]|40117|MA ||01201|PITTSFIELD|[-73.247088, 42.453086]|50655|MA ||01420|FITCHBURG |[-71.803133, 42.579563]|41194|MA ||01701|FRAMINGHAM|[-71.425486, 42.300665]|65046|MA ||01841|LAWRENCE |[-71.166997, 42.711545]|45555|MA ||01902|LYNN |[-70.941989, 42.469814]|41625|MA ||01960|PEABODY |[-70.961194, 42.532579]|47685|MA ||02124|DORCHESTER|[-71.072898, 42.287984]|48560|MA ||02146|BROOKLINE |[-71.128917, 42.339158]|56614|MA |+-----+----------+-----------------------+-----+-----+only showing top 10 rows

在SparkSession中还用Spark SQL
通过SparkSession我们可以访问Spark SQL中所有函数，正如你使用SQLContext访问一样。下面代码片段中，我们创建了一个表，并在其中使用SQL查询：
// Now create an SQL table and issue SQL queries against it without// using the sqlContext but through the SparkSession object.// Creates a temporary view of the DataFrame
scala> zipsDF.createOrReplaceTempView("zips_table")

scala> zipsDF.cache()
res3: zipsDF.type = [_id: string, city: string ... 3 more fields]

scala> val resultsDF = spark.sql("SELECT city, pop, state, _id FROM zips_table")
resultsDF: org.apache.spark.sql.DataFrame = [city: string, pop: bigint ... 2 more fields]

scala> resultsDF.show(10)+------------+-----+-----+-----+| city| pop|state| _id|+------------+-----+-----+-----+| AGAWAM|15338| MA|01001|| CUSHMAN|36963| MA|01002|| BARRE| 4546| MA|01005|| BELCHERTOWN|10579| MA|01007|| BLANDFORD| 1240| MA|01008|| BRIMFIELD| 3706| MA|01010|| CHESTER| 1688| MA|01011||CHESTERFIELD| 177| MA|01012|| CHICOPEE|23396| MA|01013|| CHICOPEE|31495| MA|01020|+------------+-----+-----+-----+only showing top 10 rows

使用SparkSession读写Hive表
下面我们将使用SparkSession创建一个Hive表，并且对这个表进行一些SQL查询，正如你使用HiveContext一样：

3. 求免费《Spark 》电子版书籍网盘资源

《Spark》电子版书籍网盘资源

链接: https://pan..com/s/14BzwQ4ncZKBHWNHzB4kBkA 提取码：fnbn

基本介绍书籍目录点评信息书籍内容 主要内容 ·大数据技术和Spark概述。 ·通过实例学习DataFrame、SQL、Dataset等Spark的核心API。 ·了解Spark的低级API实现,包括RDD以及SQL和DataFrame的执行...

4. 飞利浦Spark2 的一些问题

1:mp3,wma,wav,mp4,rmvb,rm,avi,3gp
2:PDF阅读器

5. 《SparkinAction》pdf下载在线阅读全文，求百度网盘云资源

《Spark in Action》（Marko Bonaći）电子书网盘下载免费在线阅读

资源链接：

链接:

书名：Spark in Action

作者：Marko Bonaći

豆瓣评分：7.7

出版社：Manning

出版年份：2016-1

页数：400

内容简介：

Working with big data can be complex and challenging, in part because of the multiple analysis frameworks and tools required. Apache Spark is a big data processing framework perfect for analyzing near-real-time streams and discovering historical patterns in batched data sets. But Spark goes much further than other frameworks. By including machine learning and graph processing capabilities, it makes many specialized data processing platforms obsolete. Spark's unified framework and programming model significantly lowers the initial infrastructure investment, and Spark's core abstractions are intuitive for most Scala, Java, and Python developers.

Spark in Action teaches you to use Spark for stream and batch data processing. It starts with an introction to the Spark architecture and ecosystem followed by a taste of Spark's command line interface. You then discover the most fundamental concepts and abstractions of Spark, particularly Resilient Distributed Datasets (RDDs) and the basic data transformations that RDDs provide. The first part of the book also introces you to writing Spark applications using the the core APIs. Next, you learn about different Spark components: how to work with structured data using Spark SQL, how to process near-real time data with Spark Streaming, how to apply machine learning algorithms with Spark MLlib, how to apply graph algorithms on graph-shaped data using Spark GraphX, and a clear introction to Spark clustering.

作者简介：

Marko Bonaći has worked with Java for 13 years. He currently works as IBM Enterprise Content Management team lead at SV Group. Petar Zečević is a CTO at SV Group. During the last 14 years he has worked on various projects as a Java developer, team leader, consultant and software specialist. He is the founder and, with Marko, organizer of popular Spark@Zg meetup group.

6. spark streaming应用日志怎么看

支持mysql的，下面是示例sparkstreaming使用数据源方式插入mysql数据importjava.sql.{Connection,ResultSet}importcom.jolbox.bonecp.{BoneCP,BoneCPConfig}importorg.slf4j.{vallogger=LoggerFactory.getLogger(this.getClass)privatevalconnectionPool={try{Class.forName("com.mysql.jdbc.Driver")valconfig=newBoneCPConfig()config.setJdbcUrl("jdbc:mysql://192.168.0.46:3306/test")config.setUsername("test")config.setPassword("test")config.setMinConnectionsPerPartition(2)config.setMaxConnectionsPerPartition(5)config.setPartitionCount(3)config.setCloseConnectionWatch(true)config.setLogStatementsEnabled(true)Some(newBoneCP(config))}catch{caseexception:Exception=>logger.warn(""+exception.printStackTrace())None}}defgetConnection:Option[Connection]={connectionPoolmatch{caseSome(connPool)=>Some(connPool.getConnection)caseNone=>None}}defcloseConnection(connection:Connection):Unit={if(!connection.isClosed)connection.close()}}importjava.sql.{Connection,DriverManager,PreparedStatement}importorg.apache.spark.streaming.kafka.KafkaUtilsimportorg.apache.spark.streaming.{Seconds,StreamingContext}importorg.apache.spark.{SparkConf,SparkContext}importorg.slf4j.LoggerFactory/***记录最近五秒钟的数据*/objectRealtimeCount1{caseclassLoging(vtime:Long,muid:String,uid:String,ucp:String,category:String,autoSid:Int,dealerId:String,tuanId:String,newsId:String)caseclassRecord(vtime:Long,muid:String,uid:String,item:String,types:String)vallogger=LoggerFactory.getLogger(this.getClass)defmain(args:Array[String]){valargc=newArray[String](4)argc(0)="10.0.0.37"argc(1)="test-1"argc(2)="test22"argc(3)="1"valArray(zkQuorum,group,topics,numThreads)=argcvalsparkConf=newSparkConf().setAppName("RealtimeCount").setMaster("local[2]")valsc=newSparkContext(sparkConf)valssc=newStreamingContext(sc,Seconds(5))valtopicMap=topics.split(",").map((_,numThreads.toInt)).toMapvallines=KafkaUtils.createStream(ssc,zkQuorum,group,topicMap).map(x=>x._2)valsql="insertintologing_realtime1(vtime,muid,uid,item,category)values(?,?,?,?,?)"valtmpdf=lines.map(_.split("\t")).map(x=>Loging(x(9).toLong,x(1),x(0),x(3),x(25),x(18).toInt,x(29),x(30),x(28))).filter(x=>(x.muid!=null&&!x.muid.equals("null")&&!("").equals(x.muid))).map(x=>Record(x.vtime,x.muid,x.uid,getItem(x.category,x.ucp,x.newsId,x.autoSid.toInt,x.dealerId,x.tuanId),getType(x.category,x.ucp,x.newsId,x.autoSid.toInt,x.dealerId,x.tuanId)))tmpdf.filter(x=>x.types!=null).foreachRDD{rdd=>//rdd.foreach(println)rdd.foreachPartition(partitionRecords=>{valconnection=ConnectionPool.getConnection.getOrElse(null)if(connection!=null){partitionRecords.foreach(record=>process(connection,sql,record))ConnectionPool.closeConnection(connection)}})}ssc.start()ssc.awaitTermination()}defgetItem(category:String,ucp:String,newsId:String,autoSid:Int,dealerId:String,tuanId:String):String={if(category!=null&&!category.equals("null")){valpattern=""valmatcher=ucp.matches(pattern)if(matcher){ucp.substring(33,42)}else{null}}elseif(autoSid!=0){autoSid.toString}elseif(dealerId!=null&&!dealerId.equals("null")){dealerId}elseif(tuanId!=null&&!tuanId.equals("null")){tuanId}else{null}}defgetType(category:String,ucp:String,newsId:String,autoSid:Int,dealerId:String,tuanId:String):String={if(category!=null&&!category.equals("null")){valpattern="100000726;100000730;\\d{9};\\d{9}"valmatcher=category.matches(pattern)valpattern1=""valmatcher1=ucp.matches(pattern1)if(matcher1&&matcher){"nv"}elseif(newsId!=null&&!newsId.equals("null")&&matcher1){"ns"}elseif(matcher1){"ne"}else{null}}elseif(autoSid!=0){"as"}elseif(dealerId!=null&&!dealerId.equals("null")){"di"}elseif(tuanId!=null&&!tuanId.equals("null")){"ti"}else{null}}defprocess(conn:Connection,sql:String,data:Record):Unit={try{valps:PreparedStatement=conn.prepareStatement(sql)ps.setLong(1,data.vtime)ps.setString(2,data.muid)ps.setString(3,data.uid)ps.setString(4,data.item)ps.setString(5,data.types)ps.executeUpdate()}catch{caseexception:Exception=>logger.warn("Errorinexecutionofquery"+exception.printStackTrace())}}}

7. 《Spark高级数据分析》pdf下载在线阅读，求百度网盘云资源

《Spark高级数据分析》（[美] Sandy Ryza）电子书网盘下载免费在线阅读

链接：

书名：Spark高级数据分析

作者：[美] Sandy Ryza

译者：龚少成

豆瓣评分：8.1

出版社：人民邮电出版社

出版年份：2015-11

页数：244

内容简介：

本书是使用Spark进行大规模数据分析的实战宝典，由着名大数据公司Cloudera的数据科学家撰写。四位作者首先结合数据科学和大数据分析的广阔背景讲解了Spark，然后介绍了用Spark和Scala进行数据处理的基础知识，接着讨论了如何将Spark用于机器学习，同时介绍了常见应用中几个最常用的算法。此外还收集了一些更加新颖的应用，比如通过文本隐含语义关系来查询Wikipedia或分析基因数据。

作者简介：

Sandy Ryza

是Cloudera公司资深数据科学家，Apache Spark项目的活跃代码贡献者。最近领导了Cloudera公司的Spark开发工作。他还是Hadoop项目管理委员会委员。

Uri Laserson

是Cloudera公司资深数据科学家，专注于Hadoop生态系统中的Python部分。

Sean Owen

是Cloudera公司EMEA地区的数据科学总监，也是Apache Spark项目的代码提交者。他创立了基于Spark、Spark Streaming和Kafka的Hadoop实时大规模学习项目Oryx（之前称为Myrrix）。

Josh Wills

是Cloudera公司的高级数据科学总监，Apache Crunch项目的发起者和副总裁。

8. 如何搭建spark集群.pdf

安装环境简介

硬件环境：两台四核cpu、4G内存、500G硬盘的虚拟机。

软件环境：64位Ubuntu12.04
LTS；主机名分别为spark1、spark2，IP地址分别为1**.1*.**.***/***。JDK版本为1.7。集群上已经成功部署了Hadoop2.2，详细的部署过程可以参见另一篇文档Yarn的安装与部署。

2. 安装Scala2.9.3

9. 求Spark 2.0中文 翻译， PDF最好

网络搜索： ApacheCN Spark 2.0.1 中文文档 | 那伊抹微笑

10. 求Spark大数据处理-高彦杰书籍电子版百度云资源

Spark大数据处理-高彦杰书籍电子版网络网盘txt 最新全集下载

链接:

《Spark大数据处理技术》以Spark 0.9版本为基础进行编写，是一本全面介绍Spark及Spark生态圈相关技术的书籍，是国内首本深入介绍Spark原理和架构的技术书籍。主要内容有Spark基础功能介绍及内部重要模块分析，包括部署模式、调度框架、存储管理以及应用监控；同时也详细介绍了Spark生态圈中其他的软件和模块，包括SQL处理引擎Shark和Spark SQL、流式处理引擎Spark Streaming、图计算框架Graphx以及分布式内存文件系统Tachyon。《Spark大数据处理技术》从概念和原理上对Spark核心框架和生态圈做了详细的解读，并对Spark的应用现状和未来发展做了一定的介绍，旨在为大数据从业人员和Spark爱好者提供一个更深入学习的平台。