Quick file upload to HDFS

{ pv -bfp FILE 2>&3 | ssh HDFS_USER@REMOTE_HOST "hadoop dfs -put - /path/to/hdfs/FILE" 2> /dev/null ;} 3>&1 | tr '\015' '\012'

Example:


$ touch file.txt
$ { pv -bfp file.txt 2>&3 | ssh hdfs@ff1 "hadoop dfs -put - /user/admin/file.txt" 2> /dev/null ;} 3>&1 | tr '\015' '\012'
Share on FacebookTweet about this on TwitterShare on Google+Share on StumbleUponShare on RedditShare on TumblrShare on LinkedInEmail this to someone