sparkpdf

1. 《SparkinAction》pdf下載在線閱讀，求百度網盤雲資源

《Spark in Action》（Marko Bonaći）電子書網盤下載免費在線閱讀

資源鏈接：

鏈接：

書名：Spark in Action

作者：Marko Bonaći

豆瓣評分：7.7

出版社：Manning

出版年份：2016-1

頁數：400

內容簡介：Working with big data can be complex and challenging, in part because of the multiple analysis frameworks and tools required. Apache Spark is a big data processing framework perfect for analyzing near-real-time streams and discovering historical patterns in batched data sets. But Spark goes much further than other frameworks. By including machine learning and graph processing capabilities, it makes many specialized data processing platforms obsolete. Spark's unified framework and programming model significantly lowers the initial infrastructure investment, and Spark's core abstractions are intuitive for most Scala, java, and Python developers.

Spark in Action teaches you to use Spark for stream and batch data processing. It starts with an introction to the Spark architecture and ecosystem followed by a taste of Spark's command line interface. You then discover the most fundamental concepts and abstractions of Spark, particularly Resilient Distributed Datasets (RDDs) and the basic data transformations that RDDs provide. The first part of the book also introces you to writing Spark applications using the the core APIs. Next, you learn about different Spark components: how to work with structured data using Spark SQL, how to process near-real time data with Spark Streaming, how to apply machine learning algorithms with Spark MLlib, how to apply graph algorithms on graph-shaped data using Spark GraphX, and a clear introction to Spark clustering.

作者簡介：Marko Bonaći has worked with Java for 13 years. He currently works as IBM Enterprise Content Management team lead at SV Group. Petar Zečević is a CTO at SV Group. During the last 14 years he has worked on various projects as a Java developer, team leader, consultant and software specialist. He is the founder and, with Marko, organizer of popular Spark@Zg meetup group.

2. sparksession 作用域

Apache Spark 2.0引入了SparkSession，其為用戶提供了一個統一的切入點來使用Spark的各項功能，並且允許用戶通過它調用DataFrame和Dataset相關API來編寫Spark程序。最重要的是，它減少了用戶需要了解的一些概念，使得我們可以很容易地與Spark交互。
創建SparkSession
在2.0版本之前，與Spark交互之前必須先創建SparkConf和SparkContext，代碼如下：

//set up the spark configuration and create contextsval sparkConf = new SparkConf().setAppName("SparkSessionZipsExample").setMaster("local")// your handle to SparkContext to access other context like SQLContextval sc = new SparkContext(sparkConf).set("spark.some.config.option", "some-value")val sqlContext = new org.apache.spark.sql.SQLContext(sc)
然而在Spark 2.0中，我們可以通過SparkSession來實現同樣的功能，而不需要顯式地創建SparkConf, SparkContext 以及 SQLContext，因為這些對象已經封裝在SparkSession中。使用生成器的設計模式(builder design pattern)，如果我們沒有創建SparkSession對象，則會實例化出一個新的SparkSession對象及其相關的上下文。
// Create a SparkSession. No need to create SparkContext// You automatically get it as part of the SparkSessionval warehouseLocation = "file:${system:user.dir}/spark-warehouse"val spark = SparkSession
.builder()
.appName("SparkSessionZipsExample")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
到現在我們可以使用上面創建好的spark對象，並且訪問其public方法。
配置Spark運行相關屬性
一旦我們創建好了SparkSession，我們就可以配置Spark運行相關屬性。比如下面代碼片段我們修改了已經存在的運行配置選項。
//set new runtime optionsspark.conf.set("spark.sql.shuffle.partitions", 6)
spark.conf.set("spark.executor.memory", "2g")//get all settingsval configMap:Map[String, String] = spark.conf.getAll()

獲取Catalog元數據
通常我們想訪問當前系統的Catalog元數據。SparkSession提供了catalog實例來操作metastore。這些方法放回的都是Dataset類型的，所有我們可以使用Dataset相關的API來訪問其中的數據。如下代碼片段，我們展示了所有的表並且列出當前所有的資料庫：
//fetch metadata data from the catalog
scala> spark.catalog.listDatabases.show(false)+--------------+---------------------+--------------------------------------------------------+|name |description |locationUri |+--------------+---------------------+--------------------------------------------------------+|default |Default Hive database|hdfs://iteblogcluster/user/iteblog/hive/warehouse |+--------------+---------------------+--------------------------------------------------------+
scala> spark.catalog.listTables.show(false)+----------------------------------------+--------+-----------+---------+-----------+|name |database|description|tableType|isTemporary|+----------------------------------------+--------+-----------+---------+-----------+|iteblog |default |null |MANAGED |false ||table2 |default |null |EXTERNAL |false ||test |default |null |MANAGED |false |+----------------------------------------+--------+-----------+---------+-----------+

創建Dataset和Dataframe
使用SparkSession APIs創建 DataFrames 和 Datasets的方法有很多，其中最簡單的方式就是使用spark.range方法來創建一個Dataset。當我們學習如何操作Dataset API的時候，這個方法非常有用。操作如下：
scala> val numDS = spark.range(5, 100, 5)
numDS: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> numDS.orderBy(desc("id")).show(5)+---+| id|+---+| 95|| 90|| 85|| 80|| 75|+---+only showing top 5 rows
scala> numDS.describe().show()+-------+------------------+|summary| id|+-------+------------------+| count| 19|| mean| 50.0|| stddev|28.136571693556885|| min| 5|| max| 95|+-------+------------------+scala> val langPercentDF = spark.createDataFrame(List(("Scala", 35), | ("Python", 30), ("R", 15), ("Java", 20)))
langPercentDF: org.apache.spark.sql.DataFrame = [_1: string, _2: int]

scala> val lpDF = langPercentDF.withColumnRenamed("_1", "language").withColumnRenamed("_2", "percent")
lpDF: org.apache.spark.sql.DataFrame = [language: string, percent: int]

scala> lpDF.orderBy(desc("percent")).show(false)+--------+-------+ |language|percent|+--------+-------+|Scala |35 ||Python |30 ||Java |20 ||R |15 |+--------+-------+

使用SparkSession讀取CSV
創建完SparkSession之後，我們就可以使用它來讀取數據，下面代碼片段是使用SparkSession來從csv文件中讀取數據：

val df = sparkSession.read.option("header","true").
csv("src/main/resources/sales.csv")

上面代碼非常像使用SQLContext來讀取數據，我們現在可以使用SparkSession來替代之前使用SQLContext編寫的代碼。下面是完整的代碼片段：

package com.iteblogimport org.apache.spark.sql.SparkSession/*** Spark Session example**/object SparkSessionExample {def main(args: Array[String]) {val sparkSession = SparkSession.builder.master("local").appName("spark session example").getOrCreate()val df = sparkSession.read.option("header","true").csv("src/main/resources/sales.csv")df.show()}}

使用SparkSession API讀取JSON數據
我們可以使用SparkSession來讀取JSON、CVS或者TXT文件，甚至是讀取parquet表。比如在下面代碼片段裡面，我將讀取郵編數據的JSON文件，並且返回DataFrame對象：
// read the json file and create the dataframe
scala> val jsonFile = "/user/iteblog.json"
jsonFile: String = /user/iteblog.json
scala> val zipsDF = spark.read.json(jsonFile)
zipsDF: org.apache.spark.sql.DataFrame = [_id: string, city: string ... 3 more fields]

scala> zipsDF.filter(zipsDF.col("pop") > 40000).show(10, false)+-----+----------+-----------------------+-----+-----+|_id |city |loc |pop |state|+-----+----------+-----------------------+-----+-----+|01040|HOLYOKE |[-72.626193, 42.202007]|43704|MA ||01085|MONTGOMERY|[-72.754318, 42.129484]|40117|MA ||01201|PITTSFIELD|[-73.247088, 42.453086]|50655|MA ||01420|FITCHBURG |[-71.803133, 42.579563]|41194|MA ||01701|FRAMINGHAM|[-71.425486, 42.300665]|65046|MA ||01841|LAWRENCE |[-71.166997, 42.711545]|45555|MA ||01902|LYNN |[-70.941989, 42.469814]|41625|MA ||01960|PEABODY |[-70.961194, 42.532579]|47685|MA ||02124|DORCHESTER|[-71.072898, 42.287984]|48560|MA ||02146|BROOKLINE |[-71.128917, 42.339158]|56614|MA |+-----+----------+-----------------------+-----+-----+only showing top 10 rows

在SparkSession中還用Spark SQL
通過SparkSession我們可以訪問Spark SQL中所有函數，正如你使用SQLContext訪問一樣。下面代碼片段中，我們創建了一個表，並在其中使用SQL查詢：
// Now create an SQL table and issue SQL queries against it without// using the sqlContext but through the SparkSession object.// Creates a temporary view of the DataFrame
scala> zipsDF.createOrReplaceTempView("zips_table")

scala> zipsDF.cache()
res3: zipsDF.type = [_id: string, city: string ... 3 more fields]

scala> val resultsDF = spark.sql("SELECT city, pop, state, _id FROM zips_table")
resultsDF: org.apache.spark.sql.DataFrame = [city: string, pop: bigint ... 2 more fields]

scala> resultsDF.show(10)+------------+-----+-----+-----+| city| pop|state| _id|+------------+-----+-----+-----+| AGAWAM|15338| MA|01001|| CUSHMAN|36963| MA|01002|| BARRE| 4546| MA|01005|| BELCHERTOWN|10579| MA|01007|| BLANDFORD| 1240| MA|01008|| BRIMFIELD| 3706| MA|01010|| CHESTER| 1688| MA|01011||CHESTERFIELD| 177| MA|01012|| CHICOPEE|23396| MA|01013|| CHICOPEE|31495| MA|01020|+------------+-----+-----+-----+only showing top 10 rows

使用SparkSession讀寫Hive表
下面我們將使用SparkSession創建一個Hive表，並且對這個表進行一些SQL查詢，正如你使用HiveContext一樣：

3. 求免費《Spark 》電子版書籍網盤資源

《Spark》電子版書籍網盤資源

鏈接: https://pan..com/s/14BzwQ4ncZKBHWNHzB4kBkA 提取碼：fnbn

基本介紹書籍目錄點評信息書籍內容 主要內容 ·大數據技術和Spark概述。 ·通過實例學習DataFrame、SQL、Dataset等Spark的核心API。 ·了解Spark的低級API實現,包括RDD以及SQL和DataFrame的執行...

4. 飛利浦Spark2 的一些問題

1:mp3,wma,wav,mp4,rmvb,rm,avi,3gp
2:PDF閱讀器

5. 《SparkinAction》pdf下載在線閱讀全文，求百度網盤雲資源

《Spark in Action》（Marko Bonaći）電子書網盤下載免費在線閱讀

資源鏈接：

鏈接:

書名：Spark in Action

作者：Marko Bonaći

豆瓣評分：7.7

出版社：Manning

出版年份：2016-1

頁數：400

內容簡介：

Working with big data can be complex and challenging, in part because of the multiple analysis frameworks and tools required. Apache Spark is a big data processing framework perfect for analyzing near-real-time streams and discovering historical patterns in batched data sets. But Spark goes much further than other frameworks. By including machine learning and graph processing capabilities, it makes many specialized data processing platforms obsolete. Spark's unified framework and programming model significantly lowers the initial infrastructure investment, and Spark's core abstractions are intuitive for most Scala, Java, and Python developers.

Spark in Action teaches you to use Spark for stream and batch data processing. It starts with an introction to the Spark architecture and ecosystem followed by a taste of Spark's command line interface. You then discover the most fundamental concepts and abstractions of Spark, particularly Resilient Distributed Datasets (RDDs) and the basic data transformations that RDDs provide. The first part of the book also introces you to writing Spark applications using the the core APIs. Next, you learn about different Spark components: how to work with structured data using Spark SQL, how to process near-real time data with Spark Streaming, how to apply machine learning algorithms with Spark MLlib, how to apply graph algorithms on graph-shaped data using Spark GraphX, and a clear introction to Spark clustering.

作者簡介：

Marko Bonaći has worked with Java for 13 years. He currently works as IBM Enterprise Content Management team lead at SV Group. Petar Zečević is a CTO at SV Group. During the last 14 years he has worked on various projects as a Java developer, team leader, consultant and software specialist. He is the founder and, with Marko, organizer of popular Spark@Zg meetup group.

6. spark streaming應用日誌怎麼看

支持mysql的，下面是示例sparkstreaming使用數據源方式插入mysql數據importjava.sql.{Connection,ResultSet}importcom.jolbox.bonecp.{BoneCP,BoneCPConfig}importorg.slf4j.{vallogger=LoggerFactory.getLogger(this.getClass)privatevalconnectionPool={try{Class.forName("com.mysql.jdbc.Driver")valconfig=newBoneCPConfig()config.setJdbcUrl("jdbc:mysql://192.168.0.46:3306/test")config.setUsername("test")config.setPassword("test")config.setMinConnectionsPerPartition(2)config.setMaxConnectionsPerPartition(5)config.setPartitionCount(3)config.setCloseConnectionWatch(true)config.setLogStatementsEnabled(true)Some(newBoneCP(config))}catch{caseexception:Exception=>logger.warn(""+exception.printStackTrace())None}}defgetConnection:Option[Connection]={connectionPoolmatch{caseSome(connPool)=>Some(connPool.getConnection)caseNone=>None}}defcloseConnection(connection:Connection):Unit={if(!connection.isClosed)connection.close()}}importjava.sql.{Connection,DriverManager,PreparedStatement}importorg.apache.spark.streaming.kafka.KafkaUtilsimportorg.apache.spark.streaming.{Seconds,StreamingContext}importorg.apache.spark.{SparkConf,SparkContext}importorg.slf4j.LoggerFactory/***記錄最近五秒鍾的數據*/objectRealtimeCount1{caseclassLoging(vtime:Long,muid:String,uid:String,ucp:String,category:String,autoSid:Int,dealerId:String,tuanId:String,newsId:String)caseclassRecord(vtime:Long,muid:String,uid:String,item:String,types:String)vallogger=LoggerFactory.getLogger(this.getClass)defmain(args:Array[String]){valargc=newArray[String](4)argc(0)="10.0.0.37"argc(1)="test-1"argc(2)="test22"argc(3)="1"valArray(zkQuorum,group,topics,numThreads)=argcvalsparkConf=newSparkConf().setAppName("RealtimeCount").setMaster("local[2]")valsc=newSparkContext(sparkConf)valssc=newStreamingContext(sc,Seconds(5))valtopicMap=topics.split(",").map((_,numThreads.toInt)).toMapvallines=KafkaUtils.createStream(ssc,zkQuorum,group,topicMap).map(x=>x._2)valsql="insertintologing_realtime1(vtime,muid,uid,item,category)values(?,?,?,?,?)"valtmpdf=lines.map(_.split("\t")).map(x=>Loging(x(9).toLong,x(1),x(0),x(3),x(25),x(18).toInt,x(29),x(30),x(28))).filter(x=>(x.muid!=null&&!x.muid.equals("null")&&!("").equals(x.muid))).map(x=>Record(x.vtime,x.muid,x.uid,getItem(x.category,x.ucp,x.newsId,x.autoSid.toInt,x.dealerId,x.tuanId),getType(x.category,x.ucp,x.newsId,x.autoSid.toInt,x.dealerId,x.tuanId)))tmpdf.filter(x=>x.types!=null).foreachRDD{rdd=>//rdd.foreach(println)rdd.foreachPartition(partitionRecords=>{valconnection=ConnectionPool.getConnection.getOrElse(null)if(connection!=null){partitionRecords.foreach(record=>process(connection,sql,record))ConnectionPool.closeConnection(connection)}})}ssc.start()ssc.awaitTermination()}defgetItem(category:String,ucp:String,newsId:String,autoSid:Int,dealerId:String,tuanId:String):String={if(category!=null&&!category.equals("null")){valpattern=""valmatcher=ucp.matches(pattern)if(matcher){ucp.substring(33,42)}else{null}}elseif(autoSid!=0){autoSid.toString}elseif(dealerId!=null&&!dealerId.equals("null")){dealerId}elseif(tuanId!=null&&!tuanId.equals("null")){tuanId}else{null}}defgetType(category:String,ucp:String,newsId:String,autoSid:Int,dealerId:String,tuanId:String):String={if(category!=null&&!category.equals("null")){valpattern="100000726;100000730;\\d{9};\\d{9}"valmatcher=category.matches(pattern)valpattern1=""valmatcher1=ucp.matches(pattern1)if(matcher1&&matcher){"nv"}elseif(newsId!=null&&!newsId.equals("null")&&matcher1){"ns"}elseif(matcher1){"ne"}else{null}}elseif(autoSid!=0){"as"}elseif(dealerId!=null&&!dealerId.equals("null")){"di"}elseif(tuanId!=null&&!tuanId.equals("null")){"ti"}else{null}}defprocess(conn:Connection,sql:String,data:Record):Unit={try{valps:PreparedStatement=conn.prepareStatement(sql)ps.setLong(1,data.vtime)ps.setString(2,data.muid)ps.setString(3,data.uid)ps.setString(4,data.item)ps.setString(5,data.types)ps.executeUpdate()}catch{caseexception:Exception=>logger.warn("Errorinexecutionofquery"+exception.printStackTrace())}}}

7. 《Spark高級數據分析》pdf下載在線閱讀，求百度網盤雲資源

《Spark高級數據分析》（[美] Sandy Ryza）電子書網盤下載免費在線閱讀

鏈接：

書名：Spark高級數據分析

作者：[美] Sandy Ryza

譯者：龔少成

豆瓣評分：8.1

出版社：人民郵電出版社

出版年份：2015-11

頁數：244

內容簡介：

本書是使用Spark進行大規模數據分析的實戰寶典，由著名大數據公司Cloudera的數據科學家撰寫。四位作者首先結合數據科學和大數據分析的廣闊背景講解了Spark，然後介紹了用Spark和Scala進行數據處理的基礎知識，接著討論了如何將Spark用於機器學習，同時介紹了常見應用中幾個最常用的演算法。此外還收集了一些更加新穎的應用，比如通過文本隱含語義關系來查詢Wikipedia或分析基因數據。

作者簡介：

Sandy Ryza

是Cloudera公司資深數據科學家，Apache Spark項目的活躍代碼貢獻者。最近領導了Cloudera公司的Spark開發工作。他還是Hadoop項目管理委員會委員。

Uri Laserson

是Cloudera公司資深數據科學家，專注於Hadoop生態系統中的Python部分。

Sean Owen

是Cloudera公司EMEA地區的數據科學總監，也是Apache Spark項目的代碼提交者。他創立了基於Spark、Spark Streaming和Kafka的Hadoop實時大規模學習項目Oryx（之前稱為Myrrix）。

Josh Wills

是Cloudera公司的高級數據科學總監，Apache Crunch項目的發起者和副總裁。

8. 如何搭建spark集群.pdf

安裝環境簡介

硬體環境：兩台四核cpu、4G內存、500G硬碟的虛擬機。

軟體環境：64位Ubuntu12.04
LTS；主機名分別為spark1、spark2，IP地址分別為1**.1*.**.***/***。JDK版本為1.7。集群上已經成功部署了Hadoop2.2，詳細的部署過程可以參見另一篇文檔Yarn的安裝與部署。

2. 安裝Scala2.9.3

9. 求Spark 2.0中文 翻譯， PDF最好

網路搜索： ApacheCN Spark 2.0.1 中文文檔 | 那伊抹微笑

10. 求Spark大數據處理-高彥傑書籍電子版百度雲資源

Spark大數據處理-高彥傑書籍電子版網路網盤txt 最新全集下載

鏈接:

《Spark大數據處理技術》以Spark 0.9版本為基礎進行編寫，是一本全面介紹Spark及Spark生態圈相關技術的書籍，是國內首本深入介紹Spark原理和架構的技術書籍。主要內容有Spark基礎功能介紹及內部重要模塊分析，包括部署模式、調度框架、存儲管理以及應用監控；同時也詳細介紹了Spark生態圈中其他的軟體和模塊，包括SQL處理引擎Shark和Spark SQL、流式處理引擎Spark Streaming、圖計算框架Graphx以及分布式內存文件系統Tachyon。《Spark大數據處理技術》從概念和原理上對Spark核心框架和生態圈做了詳細的解讀，並對Spark的應用現狀和未來發展做了一定的介紹，旨在為大數據從業人員和Spark愛好者提供一個更深入學習的平台。