Flink入门之 DataSet API实现Word Count
以下是一个简单的flink word count任务demo代码,首先生产上不建议使用DataSet方式,这里demo仅供学习;
(图片来源网络,侵删)
pom配置文件
simple-demo org.example 1.0-SNAPSHOT 4.0.0 flink 8 8 1.17.0 org.apache.flink flink-streaming-java ${flink.version} org.apache.flink flink-clients ${flink.version}
任务编写
package com.xx.flink.task; import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.java.operators.AggregateOperator; import org.apache.flink.api.java.operators.DataSource; import org.apache.flink.api.java.operators.FlatMapOperator; import org.apache.flink.api.java.operators.UnsortedGrouping; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.util.Collector; /** * TODO DataSet API 实现 wordcount(不推荐) */ public class WordCountBatchDemo { public static void main(String[] args) throws Exception { // TODO 1. 创建执行环境 ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); // TODO 2.读取数据:从文件中读取 DataSource lineDS = env.readTextFile("E:\code-opensource\simple-demo\flink\input\word.txt"); // TODO 3.切分、转换 (word,1) FlatMapOperator wordAndOne = lineDS.flatMap(new FlatMapFunction() { @Override public void flatMap(String value, Collector out) throws Exception { // TODO 3.1 按照 空格 切分单词 String[] words = value.split(" "); // TODO 3.2 将 单词 转换为 (word,1) for (String word : words) { Tuple2 wordTuple2 = Tuple2.of(word, 1); //TODO 3.3 使用 Collector 向下游发送数据 out.collect(wordTuple2); } } }); // TODO 4.按照 word 分组 UnsortedGrouping wordAndOneGroupby = wordAndOne.groupBy(0); // TODO 5.各分组内聚合 AggregateOperator sum = wordAndOneGroupby.sum(1); // 1是位置,表示第二个元素 // TODO 6.输出 sum.print(); } }
结果输出
over~~
(图片来源网络,侵删)
(图片来源网络,侵删)
文章版权声明:除非注明,否则均为主机测评原创文章,转载或复制请以超链接形式并注明出处。
还没有评论,来说两句吧...