A few people have reported issues using xterm under Catalina

cd documents
ls
ls: Operation not permitted

However if you try the same commands in the Terminal all works fine.

The solution was described here:-

solution was to give "Full Disk Access" to /bin/bash. This is a bit counter-intuitive, since I actually use /bin/tcsh for my shell (yeah, I know, I'm a luddite). I'm not sure whether bash is involved because my tcsh is a descendant of a bash process, or because the ls command is using bash in some way. But it is certainly true that I can toggle whether "ls ~/Documents" works by checking/unchecking bash in the Full Disk Access panel.

Dr. Burkhard Schmidt from Max-Planck-Institut provided an insightful explanation

Indeed giving Full Disk Access to /bin/bash is the correct solution. Why? Well, assuming the standard location for XQuartz, look at /Applications/Utilities/XQuartz.app/Contents/Info.plist. Under the key CFBundleExecutable you will find the value X11 which is /Applications/Utilities/XQuartz.app/Contents/MacOS/X11. The command ?file /Applications/Utilities/XQuartz.app/Contents/MacOS/X11? returns ?/Applications/Utilities/XQuartz.app/Contents/MacOS/X11: Bourne-Again shell script text executable, ASCII text?, so X11 is actually not a binary but a shell script. Open it with your favourite text editor. The first line reads ?#!/bin/bash?, so eventually /bin/bash is executed upon opening (double-clicking) XQuartz. All further X11 processes are child processes to it and inherit the Full Disk Access right.

This applies not only to Xterm but to all applications using X11 (thinking of grace, xfig, gimp and friends).

More info on Scientific Applications under Catalina here.