Macs in Chemistry

Insanely great science


Applescript and curl

Way back in the days of Mac OS 8.6 Apple provided basic internet functions using a component called URL Access, this was made available to scripts using a small applications called URL Access Scripting. This rather rough and ready solution provided a means to upload or download files. Unfortunately I occasionally had problems with it and judging by the questions on a couple of applescript forums others have also. Around the time that Mac OS X became available I changed all my scripts to use curl instead of URL Access Scripting and things have worked pretty smoothy since. To qote from the cURL website

"curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks."

Open up the Terminal and type curl -h to get a list of options.

curl -h
Usage: curl [options...] 
Options: (H) means HTTP/HTTPS only, (F) means FTP only
    --anyauth       Pick "any" authentication method (H)
 -a/--append        Append to target file when uploading (F/SFTP)
    --basic         Use HTTP Basic Authentication (H)
    --cacert  CA certificate to verify peer against (SSL)
    --capath  CA directory to verify peer against (SSL)
 -E/--cert  Client certificate file and password (SSL)
    --cert-type  Certificate file type (DER/PEM/ENG) (SSL)
    --ciphers  SSL ciphers to use (SSL)
    --compressed    Request compressed response (using deflate or gzip)
 -K/--config  Specify which config file to read
    --connect-timeout  Maximum time allowed for connection
 -C/--continue-at  Resumed transfer offset
 -b/--cookie  Cookie string or file to read cookies from (H)
 -c/--cookie-jar  Write cookies to this file after operation (H)
    --create-dirs   Create necessary local directory hierarchy
    --crlf          Convert LF to CRLF in upload
    --crlfile  Get a CRL list in PEM format from the given file
 -d/--data    HTTP POST data (H)
    --data-ascii   HTTP POST ASCII data (H)
    --data-binary  HTTP POST binary data (H)
    --data-urlencode  HTTP POST data url encoded (H)
    --digest        Use HTTP Digest Authentication (H)
    --disable-eprt  Inhibit using EPRT or LPRT (F)
    --disable-epsv  Inhibit using EPSV (F)
 -D/--dump-header  Write the headers to this file
    --egd-file  EGD socket path for random data (SSL)
    --engine   Crypto engine to use (SSL). "--engine list" for list
 -f/--fail          Fail silently (no output at all) on HTTP errors (H)
 -F/--form  Specify HTTP multipart POST data (H)
    --form-string  Specify HTTP multipart POST data (H)
    --ftp-account  Account data to send when requested by server (F)
    --ftp-alternative-to-user  String to replace "USER [name]" (F)
    --ftp-create-dirs Create the remote dirs if not present (F)
    --ftp-method [multicwd/nocwd/singlecwd] Control CWD usage (F)
    --ftp-pasv      Use PASV/EPSV instead of PORT (F)
Use PORT with address instead of PASV (F) --ftp-skip-pasv-ip Skip the IP address for PASV (F) --ftp-ssl Try SSL/TLS for ftp transfer (F) --ftp-ssl-ccc Send CCC after authenticating (F) --ftp-ssl-ccc-mode [active/passive] Set CCC mode (F) --ftp-ssl-control Require SSL/TLS for ftp login, clear for transfer (F) --ftp-ssl-reqd Require SSL/TLS for ftp transfer (F) -G/--get Send the -d data with a HTTP GET (H) -g/--globoff Disable URL sequences and ranges using {} and [] -H/--header Custom header to pass to server (H) -I/--head Show document info only -h/--help This help text --hostpubmd5 Hex encoded MD5 string of the host public key. (SSH) -0/--http1.0 Use HTTP 1.0 (H) --ignore-content-length Ignore the HTTP Content-Length header -i/--include Include protocol headers in the output (H/F) -k/--insecure Allow connections to SSL sites without certs (H) --interface Specify network interface/address to use -4/--ipv4 Resolve name to IPv4 address -6/--ipv6 Resolve name to IPv6 address -j/--junk-session-cookies Ignore session cookies read from file (H) --keepalive-time Interval between keepalive probes --key Private key file name (SSL/SSH) --key-type Private key file type (DER/PEM/ENG) (SSL) --krb Enable Kerberos with specified security level (F) --libcurl Dump libcurl equivalent code of this command line --limit-rate Limit transfer speed to this rate -l/--list-only List only names of an FTP directory (F) --local-port [-num] Force use of these local port numbers -L/--location Follow Location: hints (H) --location-trusted Follow Location: and send auth to other hosts (H) -M/--manual Display the full manual --max-filesize Maximum file size to download (H/F) --max-redirs Maximum number of redirects allowed (H) -m/--max-time Maximum time allowed for the transfer --negotiate Use HTTP Negotiate Authentication (H) -n/--netrc Must read .netrc for user name and password --netrc-optional Use either .netrc or URL; overrides -n -N/--no-buffer Disable buffering of the output stream --no-keepalive Disable keepalive use on the connection --no-sessionid Disable SSL session-ID reusing (SSL) --noproxy Comma-separated list of hosts which do not use proxy --ntlm Use HTTP NTLM authentication (H) -o/--output Write output to instead of stdout --pass Pass phrase for the private key (SSL/SSH) --post301 Do not switch to GET after following a 301 redirect (H) --post302 Do not switch to GET after following a 302 redirect (H) -#/--progress-bar Display transfer progress as a progress bar -x/--proxy Use HTTP proxy on given port --proxy-anyauth Pick "any" proxy authentication method (H) --proxy-basic Use Basic authentication on the proxy (H) --proxy-digest Use Digest authentication on the proxy (H) --proxy-negotiate Use Negotiate authentication on the proxy (H) --proxy-ntlm Use NTLM authentication on the proxy (H) -U/--proxy-user Set proxy user and password --proxy1.0 Use HTTP/1.0 proxy on given port -p/--proxytunnel Operate through a HTTP proxy tunnel (using CONNECT) --pubkey Public key file name (SSH) -Q/--quote Send command(s) to server before file transfer (F/SFTP) --random-file File for reading random data from (SSL) -r/--range Retrieve only the bytes within a range --raw Pass HTTP "raw", without any transfer decoding (H) -e/--referer Referer URL (H) -O/--remote-name Write output to a file named as the remote file --remote-name-all Use the remote file name for all URLs -R/--remote-time Set the remote file's time on the local output -X/--request Specify request command to use --retry Retry request times if transient problems occur --retry-delay When retrying, wait this many seconds between each --retry-max-time Retry only within this period -S/--show-error Show error. With -s, make curl show errors when they occur -s/--silent Silent mode. Don't output anything --socks4 SOCKS4 proxy on given host + port --socks4a SOCKS4a proxy on given host + port --socks5 SOCKS5 proxy on given host + port --socks5-hostname SOCKS5 proxy, pass host name to proxy --socks5-gssapi-service SOCKS5 proxy service name for gssapi --socks5-gssapi-nec Compatibility with NEC SOCKS5 server -Y/--speed-limit Stop transfer if below speed-limit for 'speed-time' secs -y/--speed-time Time needed to trig speed-limit abort. Defaults to 30 -2/--sslv2 Use SSLv2 (SSL) -3/--sslv3 Use SSLv3 (SSL) --stderr Where to redirect stderr. - means stdout --tcp-nodelay Use the TCP_NODELAY option -t/--telnet-option Set telnet option -z/--time-cond

The following short applescript gives a demonstration of curl in use, it downloads a user defined Protein Data Bank (PDB) file from RCSB. The first part of the script gets the POSIX path to the desktop as the download location, the dialog gets the users PDB code and then the URL is constructed and curl called using the do shell script command. It would be useful to have a little error checking to check files exist in the PDB and are not overwritten on the desktop but I'll leave that to you.

set path_to_desktop to path to desktop

set posix_path to POSIX path of path_to_desktop as text

--Download file to desktop
--display dialog posix_path

display dialog "Type in the PDB code e.g. 3K7F" default answer "" buttons {"OK", "Cancel"} default button 1

if the button returned of the result is "Cancel" then
  display dialog "Cancelled"
    set PDB_code to (text returned of the result)
end if

--do shell script "curl '' -o /Users/usename/Desktop/3K7F.pdb"

set weblink to "'" & PDB_code & "'"

set curl_command to "curl " & weblink & " -o " & posix_path & PDB_code & ".pdb"

do shell script curl_command