This is an article in a series on Deployment

Not everything can be run on a single machine (although much can, systems are powerful these days(!)). There are options to declare a full cloud configuration, albeit typically linked to a single cloud provider. You declare what servers need to be running what images, which additional services you need (managed storage, databases), network architecture and more. Well known options are Amazon’s CloudFormation (“limited” to manage AWS) and OpenTofu (multi cloud fork of the now “BSL” licensed Terraform).

Properties

See for possibilities Deploying Web applications:

• Style: Declarative
• OS virtuality: Typical
• Application virtuality: Direct to the virtual OS (often [Containered](https://murb.nl/articles/4…

This is an article in a series on Deployment

What if you can define not only your application OS declaratively, but also your apps, making it fully reproducible? That is what NixOS pioneered, but is now also offered by Guix system (where GUI doesn’t mean Graphical User Interface, but something along ‘Guile Unix’, referring to the definition language) and variants of Fedora: Atomic desktops (these are less suitable for server deployments).

Properties

See for possibilities Deploying Web applications:

• Style: Declerative
• OS virtuality: Out of scope
• Application virtuality: Direct to OS
• From where: Anywhere
• Scope: From OS to Application

This is an article in a series on Deployment.

This technically what I have been using as the basis of my ‘infrastructure’. On top of these images I would manage the OS semi manual (assisted with Capistrano) and deploy images using Capistrano, in separated user domains different, the application servers.

You can of course completely prepare the desired OS state as an image and deploy that.

Images avant la lettre? WAR

I made this in a separate category.

The next VM? WebAssembly’s VM

WebAssembly runs in a VM. Solomon Hykes, a co-founder of Docker, is quoted on WebAssembly’s Wikipedia page: “If WASM+WASI existed in 2008, we wouldn’t have needed to create Docker. That’s how important it is. WebAssembly on the server is the future of computing.” I’d consider it to be a promising development, but will not bet on this yet.

Properties

See …

This is an article in a series on Deployment.

Docker has popularised the concept of deploying containers. Containers should not be confused with images. VPS providers for example, run images in a virtual machine, on a much larger physical machine. Before Docker, when you would like to have some form of reproducible builds, you would build your own images to create controlled environments. These images were then spun up like virtual machines. Docker, however, is building on Linux kernel features such as namespaces and cgroups, hence not requiring a full OS as a lot of tasks are executed by the host-kernel, outside Docker, making it a more efficient way of deploying. Security wise, one might argue that hardware based virtualisation is safer, but when you’re controlling the machine something of reduced importance.

There are different ways to run containers: Docker, Podman, Kubernetes and more. You may even consider running containers using …

This is an article in a series on Deployment.

In my experience with Ansible, it was comparable to Capistrano. The major difference is that while Capistrano is just running a list of procedures consecutively; Ansible tries to act declarative, defining an end state rather than how to get there. Which is theoretically the nicest approach, but in the end Ansible performs tasks, which becomes apparent if you try to do something that isn’t supported by default, in these cases you’ll be writing writing scripts where you need to carefully think about processing a command and rolling it back. With many plugins there is a difference between theory and practice.

Properties

See for possibilities Deploying Web applications:

• Style: Declarative
• OS virtuality: Out of scope
• Application virtuality:…

This is an article in a series on Deployment.

Is it an image, is it a container? Bundling an entire application to be run isolated, perhaps a WAR file is a container avant la lettre. It allows JVM compiled applications to be run within a VM. This is not exclusive to applications written in Java, and might even include JRuby. Since it is atypical for Rails applications to be deployed this way, I am not considering it, but it might be an alternative for you.

Properties

See for possibilities Deploying Web applications:

• Style: Neither
• OS virtuality: Not applicable
• Application virtuality: VM
• From where: Local or CI build, deployment can be a manual step or part of the CI build
• Scope: Application

This is an article in a series on Deployment.

Copying files is essentially the basis of all deployment styles

Deploy with SCP (or rsync over SSH)

Deploying may not be that hard if you’re deploying something static. A simple copy or rsync command may well suffice.

This will work fine for static sites, but more dynamic sites often require more setup. Hence, you will often find some additional scripts that will take care of this. These scripts are imperative in nature.

Variant: deploy with .git

Small applications can be deployed with a true .git deployment: git push and remote pull, directly serving the git repository via a server.

.git based deploy

First time encounter I had with a .git based deployment approach was with Heroku (US based), a service that made deployment easy. You push (git terminology) your app, would configure your database, and it would figure out the ruby version…

This is an article in a series on Deployment

Capistrano is a solution that I’ve been relying on for years, and has served me well over these: easy rollbacks, easy to have files that are permanent. Capistrano fetches the latest version from git, and sets the right symlinks, and triggers additional commands in a predefined order such as update.

It is can be understood quite well, and its systematic deployment procedure takes care about a lot of things (building assets, running database migrations, restarting servers). It is also pretty easy to deploy the same code to multiple servers.

Properties

See for possibilities Deploying Web applications:

• Style: Imperative
• OS virtuality: Out of scope
• Application virtuality: Direct to OS
• From where: Local (but from CI is possible)
• Scope: Application …

This is the start of a series on deployment

In the old days web software was deployed by uploading software via FTP, in a folder that was then read by a web server, and then presented to the user of the web-application. I’ve also deployed compiled .war files manually via a tomcat web-interface. Databases, storage, these were all pretty much managed by hand, sometimes even requiring physically adding a new drive or server rack.

The problem with this approach, though conceptually simple, is that deploying manually is error prone. One forgets about files, configurations, put files in the wrong directory, or forget to make changes to the database in order to make the latest change work.

The first step is hence automation of the manual steps. I will discuss a few high level dimensions that can be used to discuss solutions. I’d like to stress these are in practice scales and approaches can be mixed.

Style: Imperative vs declarative deployment…

I don’t mind running my own virtual servers. Fail2ban is a tool I’ve had running on my servers for years. It helps fencing of requests from ip-addresses that repeatedly misbehave when connecting to SSH and postfix. I never got to creating my own rules. I thought I had to write it in some arcane scripting language, but recently I learned it is pretty easy.

In this case I wanted to block 500 (internal server error) and 422 (Unprocessable Entity) errors. A server error once in a while is expected, but repeated server errors are suspicious. Common source of these errors are scripts that scan for things like SQL injections.

Examples given are for Debian.

/etc/fail2ban/filter.d/nginx-errors.conf

[Definition]

failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*" (500|422)
port = http,https
ignoreregex =
backend = auto
logpath = /var/log/nginx/access.log
bantime = 600
maxretry = 10

And appending to `/etc/fail2ban/jail….

Just a short article to document for myself how to copy a large directory (e.g. a user-folder) over a local network. While (s)cp might work for smaller operations, rsync is my preferred tool as you can restart it when it breaks + in case you found an optimization, you can just abort and restart. Some things to take into account before I share the command:

• Do not mount a drive, just use ssh
• if you’re sharing from macOS, make sure file sharing has access to the entire harddrive, otherwise some important folders will sync empty (e.g. Documents(!))
• Make sure you exclude files you don’t need (a home folder typically contains many cache-files that you don’t want to sync to a new machine
• Do not enable some form of compression (it waists cpu cycles when your network is fast enough)

So here is the command:

rsync -aWP --inplace --exclude-from=exclude-file.txt murb@someaddress:/Users/username/ .

Breakdown:

• -a is the archival option, and it is typically what y…

Every now and then I’m searching for this little snippet in my notes using NotationalVelocity (or currently actually a fork):

du -hsx * | sort -rh | head -100

It’s a variation of a snippet I found somewhere, but hardly invested any time in understanding what it actually does. Let’s decompose, from head to taildu.

head

head -100

head simply limits the results to a maximum of 100 lines. Not much more to explain here

sort

sort sorts. by default it sorts the files by filename, but adding ‘-h’ to it allows it to sort by “human readable numbers” (e.g. 5M > 6K); if ‘-n’ would be added as option 6K would be > 5M. The ‘-r’ options reverses the sort wich is by default ascending.

du

du by defaults crawls a directory recursively for all files. passing ‘-s’ tells it to sum the values of files within directories. the ‘-x’ option is us…

A technical note to myself: One way of doing multiple things simultanenously on a server can be by setting up multiple connections via SSH, that’s how I used to do things before. An alternative is to use a single connection and use the command screen on the remote server. Another good reason to use screen is if you have a long running process that you don’t want to break just because your SSH connection flips on and off with your computer going in and out of stand-by.

This is for absolute beginners. If you don’t know about screen, this is for you. If you are already familiar with screen, I probably won’t be able to educate you :o

So what is Screen?

GNU Screen is a kind of window managment system for the terminal (you’re ought to say terminal multiplexer) and has several advantages over using multiple SSH connections. Most importantly: the processes keep running when SSH d…