Deploying and managing web applications

Traditional software is downloaded, installed, then run. With web applications it is different. These are built, then pushed to a remote server and then the interface of that application is presented to the screen of the users via web-technologies like HTML, CSS & JavaScript, in the past sometimes assisted by Flash, these days using WASM. Besides that web applications need databases, storage, cache-systems, maybe a search solution, and sometimes more dedicated tools. From the end-user's perspective, it became a lot easier (although more restrictive) to access these tools, but getting it up and running got harder.

In the old days web software was deployed by uploading software via FTP, in a folder that was then read by a web server, and then presented to the user of the web-application. I've also deployed compiled .war files manually via a tomcat web-interface. Databases, storage, these were all pretty much managed by hand, sometimes even requiring physically adding a new drive or…

Block troublemakers using fail2ban

I don't mind running my own virtual servers. Fail2ban is a tool I've had running on my servers for years. It helps fencing of requests from ip-addresses that repeatedly misbehave when connecting to SSH and postfix. I never got to creating my own rules. I thought I had to write it in some arcane scripting language, but recently I learned it is pretty easy.

In this case I wanted to block 500 (internal server error) and 422 (Unprocessable Entity) errors. A server error once in a while is expected, but repeated server errors are suspicious. Common source of these errors are scripts that scan for things like SQL injections.

Examples given are for Debian.

/etc/fail2ban/filter.d/nginx-errors.conf

[Definition]

failregex = ^ -.*"(GET|POST|HEAD).*HTTP.*" (500|422)
port = http,https
ignoreregex =
backend = auto
logpath = /var/log/nginx/access.log
bantime = 600
maxretry = 10

And appending to /etc/fail2ban/jail.local…

Copying / syncing files over a local network with rsync

Just a short article to document for myself how to copy a large directory (e.g. a user-folder) over a local network. While (s)cp might work for smaller operations, rsync is my preferred tool as you can restart it when it breaks + in case you found an optimization, you can just abort and restart. Some things to take into account before I share the command:

• Do not mount a drive, just use ssh
• if you're sharing from macOS, make sure file sharing has access to the entire harddrive, otherwise some important folders will sync empty (e.g. Documents(!))
• Make sure you exclude files you don't need (a home folder typically contains many cache-files that you don't want to sync to a new machine
• Do not enable some form of compression (it waists cpu cycles when your network is fast enough)

So here is the command:

rsync -aWP --inplace --exclude-from=exclude-file.txt murb@someaddress:/Users/username/ .

Breakdown:

• -a is the archival option, and it is typically what y…

*nix: find the largest files/directories within a directory

Every now and then I’m searching for this little snippet in my notes using NotationalVelocity (or currently actually a fork):

du -hsx * | sort -rh | head -100

It’s a variation of a snippet I found somewhere, but hardly invested any time in understanding what it actually does. Let’s decompose, from head to taildu.

head

head -100

head simply limits the results to a maximum of 100 lines. Not much more to explain here

sort

sort sorts. by default it sorts the files by filename, but adding ‘-h’ to it allows it to sort by “human readable numbers” (e.g. 5M > 6K); if ‘-n’ would be added as option 6K would be > 5M. The ‘-r’ options reverses the sort wich is by default ascending.

du

du by defaults crawls a directory recursively for all files. passing '-s' tells it to sum the values of files within directories. the '-x' option is used to n…

How to do it: Using screen

A technical note to myself: One way of doing multiple things simultanenously on a server can be by setting up multiple connections via SSH, that's how I used to do things before. An alternative is to use a single connection and use the command screen on the remote server. Another good reason to use screen is if you have a long running process that you don't want to break just because your SSH connection flips on and off with your computer going in and out of stand-by.

This is for absolute beginners. If you don't know about screen, this is for you. If you are already familiar with screen, I probably won't be able to educate you :o

So what is Screen?

GNU Screen is a kind of window managment system for the terminal (you're ought to say terminal multiplexer) and has several advantages over using multiple SSH connections. Most importantly: the processes keep running when SSH d…