Unix commands for helping deal with very large files
I'm regularly handling very large files containing millions for chemical structures and whilst BBEdit is my usual tool for editing text files in practice it becomes rather cumbersome for really large files (> 2 GB). In these cases I've compiled a useful list of UNIX commands that make life easier.
The page is part of the Hints and Tutorials section and can be viewed here.
Whilst I use them when dealing with large chemical structure files they are equally useful when dealing with any large text or data files.
A suggestion from a reader. Sometimes rather than one large file download sites provide the data as a large number of individual files. We can keep track of the number of files using this simple command.
MacPro:~ Chris$ ls | wc -1 177248
If anyone has any additional suggestions please feel free to submit them.