Return to site

Osx Ssh File Transfer

broken image


Transfer files by dragging them to and from the Finder. A free alternative to Fetch that may be used from OS X, Windows and Linux is is FileZilla. To install FileZilla in OS X: Download the FileZilla client for Mac OS X from here. Open the FileZilla installation file to unpack the application. OpenSSH is a free and open source SSH protocol with a major focus on encryption and data protection. Data, including passwords, transmitted using OpenSSH are encrypted with multiple protocols to ensure full security of the contents on your server. Other than SSH, OpenSSH can also be used for file transfer using commands like SCP and SFTP. As B Layer pointed out whenever, you are in a ssh tunnel your command/shell doesn't really know this. Basicly your command/shell works just like it wasn't in a tunnel (the beauty about ssh). That's also why tunnels inside tunnels work! Goodies gave the answer you were looking for I think. SCP is a very nice tool using ssh to copy files in both a pull and push way depending your situation.

Ssh To Transfer Files

The Secure File Transfer Protocol is a network protocol that provides secure file access, file transfer, and file management functionality. The following are procedures to use SFTP connection for FileZilla for Mac OS X.

A. Set up a secure file transfer connection

  1. Start FileZilla (download it free from
    http://filezilla-project.org/download.php (Intel version))
  2. Click File -> Site Manager.


  3. In Site Manager Windows, click New Site button.


  4. Type in the remote system you intend to connect to in the Host field.

    Depending on the remote system you intend to connect, the remote host server names are as follows:

    Remote Server

    Purpose

    web.hku.hk

    personal websites

    hpc2015-file.hku.hk

    high performance computing cluster

    htcfile.hku.hk

    high throughput computing service

    For example, if your personal website is http://web.hku.hk/~yourUID, then enter web.hku.hk in the Host field.

    If your department website is http://www.hku.hk/yourUID, then enter www.hku.hk in the Host field.

  5. In Port field, type 22.
  6. In the Server-Type drop-down menu, select SFTP - SSH File Transfer Protocol.


  7. For Logon Type field, select Normal.
  8. Type in your account name (HKU Portal UID) in the User field (you must have an account on the web host server).
  9. Type in your account password (HKU Portal PIN) in the Password field.
  10. In Site Manager windows, click on the Connect button.


  11. You will see a pop-up window below. This is a feature of the SSH protocol. It is designed to protect you against a network attack known as 'spoofing' - secretly redirecting your connection to a different computer, so that they can get your password. Click OK to continue.


  12. In the FileZilla window, the local system (your PC) appears on the left window and the remote system appears on the right window.


  13. To upload a file, drag the file from the the left window to the right window.
  14. To download a file, drag the file from the right windows to the left window.

D. Exit FileZilla

  1. To end your session, click on the menu FileZilla on the top menu => Quit FileZilla.
Osx Ssh File Transfer

How to transfer files using SCP, SFTP, Fuse and S3

Some people are choosing to move files via the web console since you are only asked for your Duo authentication when first logging in and then can keep the window open to transfer files selectively. To learn more about this option take a look at the documentation for Open OnDemand

Why use SSH/SCP/SFTP for file transfer?

SCP and SFTP both run over ssh and are thus encrypted. There are implementations available for all common operating systems including Linux, Windows, and Mac OS X.

Windows

GUI:
  • WinSCP
    • Host: login.hpc.caltech.edu
    • Enter your username and password.
  • FileZilla
Command Line:
  • pscp, psftp - part of the PUTTY tools

Linux

Command Line:
  • Start Terminal (Applications->Accessories->Terminal)
    • To copy files from your computer to the central cluster
      • Type scp local_filename username@login.hpc.caltech.edu:~/username/
    • To copy files from to the central cluster to your computer
      • Type scp username@login.hpc.caltech.edu:/home/username/remote_filename .

Mac OS X

Command Line:
  • Start Terminal (Applications->Utilities->Terminal)
    • To copy files from your computer to the central cluster
      • Type scp local_filename username@login.hpc.caltech.edu:~/username/
    • To copy files from the central cluster to your computer
      • Type scp username@login.hpc.caltech.edu:~/username/remote_filename
SSHFS on Mac OS X
If you prefer filesystem like access you may use FuseOS together with SSHFS. This works over SSH protocol and is therefore encrypted as with standard SSH/SCP/SFTP but with the added benefit of drag and drop transfers.

Ssh File Transfer Download

  • Download and install FUSE and SSHFS here.
  • Make a local mount directory on your Mac. mkdir ~/Desktop/HPC-Mount
  • Run a command similar to the following, swapping out your username and directory name.
  • sshfs -o allow_other,defer_permissions,auto_cache remote-username@login.hpc.caltech.edu:/home/remote-username ~/Desktop/HPC-Mount
GUI:
  • Cyberduck.
    • Cyberduck can be made to work with 2 factor
      • Click on 'Open Connection'
      • choose 'SFTP'
      • enter you username and password, then click connect
      • In the 'Provide additional login credentials' box, enter 1 in the password field and hit enter if using the smartphone app.
      • You should be prompted on you cell phone to allow the connection
      • If using a yubikey, you can touch it when prompted to complete the login.
Globus is a fast, reliable file transfer service that makes it easy for users to move data between two GridFTP servers or between a GridFTP server and a user's machine (Windows, Mac or Linux).
Setup the Globus endpoint via their website.
  1. Go to this address to setup a new globus endpoint https://app.globus.org/file-manager/gcp
  2. Select California Institute of Technology > Continue
  3. Sign in with your access.caltech credentials
  4. Set the Endpoint display name to 'central-hpc' (or something similar)
  5. Click 'Generate setup key' and copy that to a secure location.
Setup the Globus Personal client under your account on the Central HPC.
SSH to the Central HPC then run the following.
  1. module load globusconnectpersonal/3.0.2
  2. globusconnectpersonal -setup
  3. globusconnectpersonal -start &
The daemon should now be running in the background and connected to the external Globus service. You should be able to browse your home directory and transfer data to and from it.
If you need to allow Globus access to another directory (for instance /central/groups/xxx) perform the following.
Edit ~/.globusonline/lta/config-paths, adding the following line after the existing one. The one in the configuration below sets the directory as read/write. Setting to 0 will set the directory to read-only in Globus.
/central/groups//,0,1
Restart the Globus daemon to pickup the changes.
  1. globusconnectpersonal -stop
  2. globusconnectpersonal -start &

Ssh Transfer File To Remote

You should now be able to navigate to the additional directory via the Globus website. Keep in mind that even though you've added a directory to allow globus access, existing unix permissions will determine what files and directories you have access to on the cluster.
If your data is in Amazon S3 you may use the awscli tools which are already installed as a module on the cluster.
  • Log into the cluster and run module load awscli/1.15.27
  • Type aws configure and enter your Amazon Web Services API key and private key. (You generate these in the IAM credential page in the AWS console).
  • Run a command similar to the following to copy data from S3 to your cluster home directory.
  • aws s3 cp --recursive s3://my-bucket-name/subfolder/ ~/destination-directory/
  • Run a command similar to the following to copy data from the cluster to a pre-existing S3 bucket.
  • aws s3 cp --recursive ~/source-directory/ s3://my-bucket-name/subfolder/
  • More s3 examples are available here.
If your data is in Google Cloud Storage you may use the gsutil which is installed as a module on the cluster.

Linux Ssh File Transfer

  • Log into the cluster and run module load python/2.7.15 gcloud/latest
  • Run gcloud auth login to configure the Google SDK for your GCP account if needed.

  • Run following command to copy data from the cluster to Google Cloud Storage.

  • gsutil cp ~/kitten.png gs://my-awesome-bucket

  • More gsutil examples here.





broken image