pig的安装及使用

1.下载软件:

我们提供的服务有：网站设计、网站制作、微信公众号开发、网站优化、网站认证、庐山ssl等。为上千家企事业单位解决了网站和推广的问题。提供周到的售前咨询和贴心的售后服务，是有科学管理、有技术的庐山网站制作公司

wget http://apache.fayea.com/pig/pig-0.15.0/pig-0.15.0.tar.gz

2.解压

tar -zxvf pig-0.15.0.tar.gz

mv pig-0.15.0 /usr/local/

ln -s pig-0.15.0 pig

3.配置环境变量:

export PATH=PATH=$HOME/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/pig/bin:$PATH;

export PIG_CLASSPATH=/usr/local/hadoop/etc/hadoop;

4.进入grunt shell:

以本地模式登录pig: 该方式的所有文件和执行过程都在本地,一般用于测试程序

[hadoop@host61 ~]$ pig -x local

15/10/03 01:14:09 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL

15/10/03 01:14:09 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType

2015-10-03 01:14:09,756 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35

2015-10-03 01:14:09,758 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1443860049744.log

2015-10-03 01:14:10,133 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found

2015-10-03 01:14:12,648 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

2015-10-03 01:14:12,656 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

2015-10-03 01:14:12,685 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///

2015-10-03 01:14:13,573 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

grunt>

以Mapreduce模式登录:实际工作模式:

[hadoop@host63 ~]$ pig

15/10/03 02:11:54 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL

15/10/03 02:11:54 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE

15/10/03 02:11:54 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType

2015-10-03 02:11:55,086 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35

2015-10-03 02:11:55,087 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1443863515062.log

2015-10-03 02:11:55,271 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found

2015-10-03 02:11:59,735 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

2015-10-03 02:11:59,740 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

2015-10-03 02:11:59,742 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://host61:9000/

2015-10-03 02:12:06,256 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

2015-10-03 02:12:06,257 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: host61:9001

2015-10-03 02:12:06,265 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

grunt>

5.pig的运行方式有如下三种:

1.脚本

2.grunt

3.嵌入式

6.登录pig,并使用常用的命令:

[hadoop@host63 ~]$ pig

15/10/03 06:01:01 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL

15/10/03 06:01:01 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE

15/10/03 06:01:01 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType

2015-10-03 06:01:01,412 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35

2015-10-03 06:01:01,413 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1443877261408.log

2015-10-03 06:01:01,502 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found

2015-10-03 06:01:03,657 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

2015-10-03 06:01:03,657 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

2015-10-03 06:01:03,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://host61:9000/

2015-10-03 06:01:05,968 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

2015-10-03 06:01:05,968 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: host61:9001

2015-10-03 06:01:05,979 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

grunt> help

Commands:

; - See the PigLatin manual for details: http://hadoop.apache.org/pig

File system commands:

fs - Equivalent to Hadoop dfs command: http://hadoop.apache.org/common/docs/current/hdfs_shell.html

Diagnostic commands:

describe [::

explain [-script ] [-out ] [-brief] [-dot|-xml] [-param =]

[-param_file ] [] - Show the execution plan to compute the alias or for entire script.

-script - Explain the entire script.

-out - Store the output into directory rather than print to stdout.

-brief - Don't expand nested plans (presenting a smaller graph for overview).

-dot - Generate the output in .dot format. Default is text format.

-xml - Generate the output in .xml format. Default is text format.

-param

-param_file - See parameter substitution for details.

alias - Alias to explain.

dump - Compute the alias and writes the results to stdout.

Utility Commands:

exec [-param =param_value] [-param_file ]