HiveとPrestoのメモ

HIVEとPrestoについてメモを残す

HIVE

安定度はPrestoより上だが、速度劣る

Presto

速度はHiveより上、ただし安定度は劣る

コマンド

[fuga-user@fuga ~]$ cip=$(curl http://hogehoge/fugafuga/emr/XXXX/YYYY | jq -r .private_dns) && ssh hadoop@$cip
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100   213    0   213    0     0   2349      0 --:--:-- --:--:-- --:--:--  2366
The authenticity of host 'ip-hogehoge.ap-northeast-1.compute.internal ' can't be established.
ECDSA key fingerprint is 68:43:d0:66:4b:4f:f2:df:9c:0f:a7:1a:a9:ce:e7:2e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hogehoge.141' (ECDSA) to the list of known hosts.
      __|  __|_  )
      _|  (     /   Amazon Linux AMI
     ___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2017.03-release-notes/
14 package(s) needed for security, out of 240 available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2018.03 is available.
EEEEEEEEEEEEEEEEEEEE MMMMMMMM           MMMMMMMM RRRRRRRRRRRRRRR
E::::::::::::::::::E M:::::::M         M:::::::M R::::::::::::::R
EE:::::EEEEEEEEE:::E M::::::::M       M::::::::M R:::::RRRRRR:::::R
 E::::E       EEEEE M:::::::::M     M:::::::::M RR::::R      R::::R
 E::::E             M::::::M:::M   M:::M::::::M   R:::R      R::::R
 E:::::EEEEEEEEEE   M:::::M M:::M M:::M M:::::M   R:::RRRRRR:::::R
 E::::::::::::::E   M:::::M  M:::M:::M  M:::::M   R:::::::::::RR
 E:::::EEEEEEEEEE   M:::::M   M:::::M   M:::::M   R:::RRRRRR::::R
 E::::E             M:::::M    M:::M    M:::::M   R:::R      R::::R
 E::::E       EEEEE M:::::M     MMM     M:::::M   R:::R      R::::R
EE:::::EEEEEEEE::::E M:::::M             M:::::M   R:::R      R::::R
E::::::::::::::::::E M:::::M             M:::::M RR::::R      R::::R
EEEEEEEEEEEEEEEEEEEE MMMMMMM             MMMMMMM RRRRRRR      RRRRRR

Prestoの場合
[hadoop@hogehoge ~]$ presto-cli --catalog hive --schema prd_ac_1003 --user hadoop

Hiveの場合
[Hive@hogehoge ~]$ hive

#logが置かれる場所

[ec2-user@ip-172-30-137-215 hive]$ pwd
/mnt/var/log/hive
[ec2-user@ip-172-30-137-215 hive]$ ll
total 9216
-rw-r--r-- 1 hive hive 8377081 Dec 13 12:28 hive-server2.log
-rw-r--r-- 1 hive hive  516604 Dec 13 12:28 hive-server2.out
drwxrwxrwt 4 root root      30 Dec 13 08:03 user

雑談

以下の記事はとってもよいと思いました。
データサイエンティストと一口にいってもいろいろあるのだと実感。