development on HHRA

Adventures in Low Level Programming - Text play

Sat, 14 Dec 2019 00:00:00 +0000

This is part two of my adventures in low level programming series. In part one we got our computer to boot and then just sit there spinning it’s wheels. It doesn’t feel like much of an achievement at the moment, but in this installment we’ll start to see something a little more exciting.

Before we get stuck in, we’ll need a little more background though (boo! I know). Obviously at this stage we don’t want to be writing the text rendering code ourselves, nor do we want to be worrying about placement of the characters. Lukily BIOS has got our back at this stage in the boot process.

BIOS Interrupts

So we know that BIOS has got our back on this, but how on earth do we call it to action? That’s where our interrupts come in. Interrupts are a way of ‘interrupting’ whatever the CPU is doing at the time and tell it to temporarily run a different piece of code before coming back and continuing where it left off. Interrupts can be triggered both by code (as we’ll see in a second) and by latency-sensitive hardware such as network cards.

There is a table of interrupts which maps a number to the location of code in memory to deal with that interrupt. For example, 0x10 may point to a location 100 bytes into memory which is where the start of the code to deal with whatever interrupt 0x10 was starts.

Rather than having many many different interrupts, instead it’s common to combine multiple functions into groups and select between them in a switch like manner. For example, there may be one group related to screen functions (0x10) within which you may be able to call functions such as ‘print character’, ‘set cursor position’, or ‘Write graphics pixel’.

Registers

In order to understand how we switch between the options in a particular interrupt, we must also understand one of the most fundamental parts of programming at this low level. Registers! You can think of registers as variables in higher level programming.

Unfortunately, unlike variables in higher level programming, we’re usually limited to only four of them. The registers available on all x86 computers are: ax, bx, cx, dx. Each of these registers holds one word (two bytes) of data.

We can also choose to split each register into high and low bytes, effectively giving us 8 byte registers or 4 word registers. We reference each byte by swapping x for l (low byte) and h (high byte).

Let’s take a look at a quick example of what working with these registers looks like in practice. At this point we’re not yet ready to print out the contents of registers, so instead you’ll just need to trust me that they contain what I say they contain (sorry!).

MOV ax, 0x4534 # ax now contains 0x4534 or 100010100110100 in binary.
MOV bl, 0x45 # bx now contains 0x4500 or 100010100000000 in binary.
MOV bh, 0x34 # bx now contains 0x4534 or 100010100110100 in binary.

In each of the above examples, we’re using the MOV operation which is what we use to move data into, out of, and between our registers. It’s in the format MOV <destination>, <source> where source can be hard-coded numbers (as in our example), other registers, or even pointers to memory.

Printing a character

Now that we’ve got the necessary background out of the way we can begin on the exciting part - printing a single character to the screen! Bear with me though, I promise by the end of this part we’ll be able to print arbitrary strings out.

Beginning from where we left off in the last part, let’s try calling the bios printing routine by using the interrupt 0x10 and function code (that thing we discussed earlier about having multiple functions within a single interrupt) 0x0e to indicate teletype mode.

In this case, we put the function code in ah (the high byte of register a) and the ascii code for the letter we wish to print in al. We then call the 0x10 interrupt (screen functions) to actually execute the code.

mov ah, 0x0e
mov al, 'H'
int 0x10
block:
jmp block
times 510-($-$$) db 0
dw 0xaa55

When we compile and run the above using the same command as last time, you should see the same as before but with a ‘H’ now displayed.

We can begin printing whole words by repeating the mov al, H for each letter we wish to print, for example:

mov ah, 0x0e
mov al, 'H'
int 0x10
mov al, 'e'
int 0x10
mov al, 'l'
int 0x10
mov al, 'l'
int 0x10
mov al, 'o'
int 0x10
block:
jmp block
times 510-($-$$) db 0
dw 0xaa55

(Notice that as we don’t change the value of ah we only need to set it once and then just trigger the interrupt each time we change the value of al)

This method is very tedius however, so we really need to find a way to simplify this printing so we can place a series of characters somewhere in memory and then loop through and print that string of characters to the screen. We’ll need a little more background again to complete this however, so let’s dive in.

Comparing values

We’ll need some way to identify the end of a string. The usual way to do this in programming is to append a zero byte to the end. If you’ve ever programmed in C you’ll probably be familiar with the idea of a null terminated string. In fact, a common cause of bugs is forgetting to null terminate strings and have the print functions over-run into memory further down the line.

In order to detect this zero byte at the end of our string, we’ll need a way of comparing one value against another. Luckily x86 assembly has built in instructions for just this. These functions allow you to conditionally jump and come in many forms - the most common being je (jump if equal), jne (jump if not equal), jg (jump if greater), and jl (jump if less).

Let’s take a quick look at how that looks. In this example, we’ll also use the addition instruction to loop 5 times, printing . each time, and then exiting.

mov ah, 0x0e
mov bl, 0
printloop:
; Check if we've printed 5 dots yet.
cmp bl, 5
je block
; Print another dot.
mov al, '.'
int 0x10
; Increment our counter
add bl, 1
; Jump to the beginning of our loop
jmp printloop
block:
jmp block
times 510-($-$$) db 0
dw 0xaa55

Hopefully the above makes sense, although it looks complicated it’s just a combination of loops and printing which we’ve covered previously. You’ll notice some comments in there too, which begin with ;. We’ll take a quick look at our comparison code to make sure it’s clear what’s happening there.

 cmp bl, 5
je block

We do comparisons in two parts, first we tell the CPU what to compare (cmp bl, 5) which means we are comparing bl to 5. Notice at this point we don’t declare what we want to know (e.g. greater than, less than, etc…), only what we are comparing.

The next line is where we actually take action based on the outcome of the comparison. In this case, we jump to the block tag if bl equals 5. You can think of this as “jmp if equals”.

Reading from memory

The other piece of required background is being able to read from memory. For this part we won’t concern ourselves with writing to memory programatically, instead telling the assembler to pre-populate part of our code with a particular value. Let’s start with this then, and learn how to pre-populate a piece of memory with a particular string.

...
my_string:
db 'booting...',0
times 510-($-$$) db 0
dw 0xaa55

For brevity I’ve remove the code previously from the top of the file, represented by .... You’ll notice we start by using my_string: which looks very much like the labels we’ve used previously (block: and printloop:). In fact, as far as the assembler is concerned, there is no difference. All these labels allow you to do is reference a specific place in memory by name. It doesn’t care whether you are using the specific place in memory to jump execution to or move something into a register.

We then use the db that we explained in the last post to place the ascnasm bootloader.asmii characters ‘booting…’ and then a zero at the current point in the program. If we compile what we’ve got so far and take a look at the hexdump output, we can see our characters in the file:

$ nasm bootloader.asm
$ hexdump -C bootloader
00000000 b4 0e b3 00 80 fb 05 74 09 b0 2e cd 10 80 c3 01 |.......t...nasm bootloader.asm.....|
00000010 eb f2 eb fe 62 6f 6f 74 69 6e 67 2e 2e 2e 00 00 |....booting.....|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200

Because our whole compiled program gets loaded into memory, this also means that we’ve got out target string in the memory of the computer. Now we just need to actually read it! The code to print the first character of our string looks something like this

mov ah, 0x0e
mov al, [my_string]
int 0x10

We’ll create a new block named printstringdemo which we will be using to store the code for the printing we’re about to do. This can be placed just above the block:.

printstringdemo:
mov ah, 0x0e
mov al, [my_string]
int 0x10
jmp block
block:
jmp block

We can then modify our line above to jump to printstringdemo when done instead of block:

 ; Check if we've printed 5 dots yet.
cmp bl, 5
je printstrprintstringdemoing

If we compile and run that we should get five dots and then the letter ‘b’. However right now we don’t, this is due to a slight disconnect between our assembler and where our code is loaded into memory. When referencing memory using the [] operator assembly will, by default, reference memory relative to the beginning of our code. This is all fine and dandy assuming our code was loaded at the first byte of memory. As we know, however, the BIOS needs to store other items such as it’s interrupt table before our code.

It turns out, our code is usually loaded at 0x7c00 so referencing memory at address zero doesn’t in fact reference anything in our program. Instead, we need to reference 0x7c00 plus the offset from the start of our program. Luckily, rather than calculating this every time manually, we can put [org 0x7c00] at the very top of our program which will tell our assembler to calculate all references by adding 0x7c00 to the memory address.

Placing that line at the top of the file and compiling again does get us the expected outcome of five dots followed by a ‘b’. Now we’ve got to loop through and print the rest of the characters in our string. To do this, we can re-purpose the bx register and use it to store the current memory address that we need to print out, like so:

printstringdemo:
mov bx, my_string
mov al, [bx]
int 0x10
jmp block

So far our changes have made no functional difference, but they have given us a very useful tool. We can now increment the value in bx to get the second, third, fouth, and so on bytes of my_string. Let’s get the second byte now and print five dots followed by an ‘o’

printstringdemo:
mov bx, my_string
call printcharacter
jmp block
printcharacter:
; Move the value of the current value into bl for printing
mov al, [bx]
; If the value is 0 (indicating the end of the string)
cmp al, 0
; Then jump to printdone
je printdone
; Otherwise print the character
int 0x10
; And then add one to the current address
add bx, 0x01
; And loop
jmp printcharacter
printdone:
; Return to where we were in printstring:
ret

Putting it all together

Great! We’ve got all the background we need now, and are ready to create a nice usable routine for printing a string to the screen. To use this routine we’ll set bx to the memory location of the first byte of our null-terminated string. When we jump to the routine which will loop through the string one character at a time until we meet the zero byte indicating the end. It will then jump back out of the routine back to where it was called from so we can continue.

Let’s start by creating our block that will demonstrate how we want to use this routine (this should entirely replace the existing printstringemo function):

printstringdemo:
mov bx, my_string
call printstring
jmp block

We’ve got a new instruction here, call. Call allows us to jmp to a routine, but then remember where we were and jmp back to where we were. Very useful here where we may want to print something out, and then continue with other processing. We’ll see how we jump back in a second.

Now that we know exactly how we want to call our routine, let’s create the entrypoint.

printstring:
; Initialize interrupt to printing character
mov ah, 0x0e
; Jump to the character printing routine
jmp printcharacter

We now need to implement the character printing routine, which will keep looping until it sees a zero, at which point it will jump to a finishing routine.

printcharacter:
; Move the current character to print to al
mv al, [bx]
; Check if the current character to print is zero
cmp al, 0
; If it was zero (indicating end of string), jump to the finished routine
je printdone
; Print the character
int 0x10
; Increment the counter and loop
add bx, 0x01
jmp printcharacter

Finally, we need to return to where we were called from, we do this in the printdone block as such:

printdone:
ret

ret here is the instruction that tells us to jump back to the last call function.

Putting it all together now leaves us with a complete assembly file that looks like:

[org 0x7c00]
mov ah, 0x0e
mov bl, 0
printloop:
; Check if we've printed 5 dots yet.
cmp bl, 5
je printstringdemo
; Print another dot.
mov al, '.'
int 0x10
; Increment our counter
add bl, 1
; Jump to the beginning of our loop
jmp printloop
printstringdemo:
mov bx, my_string
call printstring
jmp block
printstring:
; Initialize interrupt to printing character
mov ah, 0x0e
; Jump to the character printing routine
jmp printcharacter
printcharacter:
; Move the current character to print to al
mov al, [bx]
; Check if the current character to print is zero
cmp al, 0
; If it was zero (indicating end of string), jump to the finished routine
je printdone
; Print the character
int 0x10
; Increment the counter and loop
add bx, 0x01
jmp printcharacter
printdone:
ret
block:
jmp block
my_string:
db 'booting...',0
times 510-($-$$) db 0
dw 0xaa55

Running which gives us five dots and then our string (“booting…”).

aaannnddd finally, that’s us done! Lots of work just to print a single string to the screen right? There’ll be a much shorter post next time (I promise! I need a break as much as you do) about the layout of our project where we begin to set ourselves up for expanding on what we’ve done so far.

Adventures in Low Level Programming

Sat, 12 Oct 2019 00:00:00 +0000

Ohh, this is going to be a fun one! And one I’ve played with in the past to varying degrees of success. What I’m trying to achieve here is simple, I want to boot a computer from nothing to running some C code which prints “Hello, World!” to the screen. The C code will be sitting on a FAT32 drive attached to the computer.

Getting an environment setup

First things first, we’re going to need to get a development environment up and running. We’ll use QEMU as our emulator to allow us to run our code in a nice user-friendly way. Make sure you can get to the point of typing qemu-system-x86_64 into the terminal and having it open up. On ubuntu that’s as simple as installing via apt.

# sudo apt install qemu

We’ll also need a few compilers. First an assembly compiler, you’ll need to be able to run nasm and get the version.

# nasm --version
NASM version 2.14

We’ll need gcc too. This is installed by default on many distributions. Again, you’ll need to be able to run gcc and get the version.

# gcc --version
gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Both of these can be achieved on ubuntu by installing the packages through the default repos.

# sudo apt install gcc nasm

A little bit of theory

Before we continue, it’s probably prudent to give a little bit of background to what we’re about to try to achieve. The format of a boot sector is actually fairly simple. It’s a set of data at the very start of a physical disk, usually 512 bytes long, that ends in a magic number 0xaa55. That magic number is what tells BIOS that this is indeed a boot sector and not just some random data on a storage disk.

Clearly, 512 bytes isn’t very much to work with, so we’ve got to work fast and efficiently to read from a filesystem further on in the disk where we can load whole kilobyte files(!).

Creating some boilerplate code

First things first, we’ve got to sort out that magic number. Let’s write the simplest assembly program we can that will just hang the computer. Importantly though, it’ll place the magic number at the end of the sector and fill in the rest with 0x0. This will continue to work as we add more and more assembly instructions meaning we never need to think about placing it in the right place.

block:
jmp block
times 510-($-$$) db 0
dw 0xaa55

Always read assembly top down, so lets take this line by line and analyze what’s happening.

block:

This is a label which we can use to navigate programmatically through our code. These are the basic building blocks of all control structures that we’ll use. Here we’re effectively creating the equivelent of an infinite loop in c:

while(true) {
 // do nothing
}

the next line of code finishes off this infinite loop by jumping immediately back to the label we put in place previously. This line is in two pieces, jmp which is what we call an unconditional jump. We’ll cover this in more detail later but there are other kinds of jump where we can say, for example, only jump if register x is larger than 10. This will give us a foundation for building ‘if’ statements.

 jmp block

The final two lines are the ones we need for the boilerplate. The first one fills all but the last two bytes in with 0x0

times 510-($-$$) db 0

Although it looks intimidating, when broken down this line isn’t quite as complicated as it looks. times is the ‘command’ that we’re calling. It effectively has two parameters, so you can think of it like: times <number-of-repetitions> <thing-to-repeat>. In this case we’re repeating 510-$ times. That doesn’t mean much, does it? Well… Let’s break this down even more. We know we want to put two bytes at the end of the file (our magic number), which is why we use 510 here and not 512.

So what we want to know is how many bytes we need to set to 0x0 to fill in space between where we are now and 2 bytes before the end of the file. This is where nasm really helps us as it will keep track of how many bytes through the file we are, that’s what the cryptic $ is. All the dollar does is (at compile time) represent the current position in the assembly file. So what we’re actually doing when we evaluate 510-$ is 510-<current-bytes-into-file> which will give us back the number of bytes we would need to fill with zeros to fill in all but the last 2 bytes of a 512-byte program.

Finally we have the thing we actually want to repeat. In this case db 0 which just says, I want to put a zero byte in the file at this position. Because it’s repeated multiple times we end up with multiple zeros.

All that’s left at this point is to actually stick the remaining two bytes at the end. We can use the dw command for this which stands for data word (previously we used the db command which stands for data byte). We then put the magic number in and we’re done.

dw 0xaa55

Compiling and inspecting our work

Let’s save the above assembly into a file called bootloader.asm and then run nasm bootloader.asm. If we take a look in our folder, we should now see a new file which is exactly 512 bytes long.

# ls -lahtr
total 24K
-rw-rw-r-- 1 hhra hhra 55 Oct 06 20:29 bootloader.asm
-rw-rw-r-- 1 hhra hhra 512 Oct 06 20:29 bootloader

Looking in the contents of the file with a piece of software called hexdump which allows us to print out the raw bytes of a file, we can see that the file is indeed 512 bytes long, and that it ends in our magic code!

# hexdump bootloader
0000000 feeb 0000 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
*
00001f0 0000 0000 0000 0000 0000 0000 0000 aa55
0000200

The way hexdump displays information can be a little weird at first. The first column isn’t actually the data in the file at all, it’s actually the offset from the start of the file. In this case the first line starts at 0000000 which is the beginning of the file. The rest of the first line shows us the first 16 bytes of the file in hexadecimal form. Most of it is zeros apart from the first two bytes which are our opcodes (we’ll get to this in a second). The second line then starts from 0000010 which, surprise surprise, is 16 represented in hex notation.

You’ll notice the file when run through hexdump doesn’t actually look like 512 bytes. That’s because hexdump is nice enough to remove duplicate blocks for us and replaces them with a *. The fourth line in the output then is the last set of bytes in the file.

00001f0 0000 0000 0000 0000 0000 0000 0000 aa55

The beginning of this line starts at the 496th byte in the file (00001f0 in decimal). There are 14 bytes of zeros displayed, bringing us to the 510th byte, and finally aa55 which completes our file at exactly 512 bytes. Absolutely perfect.

Let’s jump back for just a moment now to the beginning of that file where we have the only other two non-zero bytes in the whole thing.

0000000 feeb 0000 0000 0000 0000 0000 0000 0000

Ignoring the zeros on the end, we need to figure out what feeb stands for. We know they are x86 OP codes. Because of the endianeness of the system, they’re actually displayed in a weird order, the argument is coming before the command. The command we need to look up in that case is eb which corresponds to jmp - big freaking surprise. In particular, the command JMP rel8 which means that we want to jump to a position in the file, relative to the current command, specified using an 8bit value. This 8-bit value which is a signed value, in our example is fe in two’s compliment notiation this corresponds to -2. At this point in the file the current position of the program counter (which is what the jmp is relative to) is fe which is two bytes into the file, so jumping two bytes back takes us all the way to the start, completing our loop.

Feel free to experiment with adding further jmp’s and looking at how the soure code changes. In particuar how the number defining how far back we jump changes each time.

Running our code

Now that we fully understand what’s going on underneath, all that’s required now is to actually run the code and see what happens. I’m afraid it won’t be very exciting at this point, but just know that nothing happening is a good sign. If something goes wrong the machine will hard-reset. All it takes now is to attach the 512-byte file to our emulator as though it were a disk and run.

# qemu-system-x86_64 bootloader
WARNING: Image format was not specified for 'bootloader' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.

This should pop up a new window which looks like this and does absolutely nothing:

It doesn’t look like much, but this is a big milestone. we’re now executing our own code on a computer with no operating system at all. We’re still nowhere near our original goal, but we’ve now got a platform on which we can build. In the next installment we’ll get text rendering to the screen.

Environment Specific Settings Files with Drupal BLT

Fri, 13 Sep 2019 00:00:00 +0000

Despite having a big warning at the bottom of the docroot/sites/default/settings.php file telling you not to add additional settings to that file. The BLT documentation does little do say what the recommended way of conditionally including *.settings.php files depending on environment variables or some other decider.

Normally, you would do it by adding code to the effect of the below to docroot/sites/default/settings.php but since there is such a direct warning not to in the BLT version, I needed to find a better way:

if (getenv('RUNNING_IN_DOCKSAL') == 'yes') {
 include $app_root . '/' . $site_path . '/settings.docksal.php';
}

if (getenv('RUNNING_IN_QA') == 'yes') {
 include $app_root . '/' . $site_path . '/settings.qa.php';
}

There is a file called docroot/sites/default/settings/includes.settings.php which looks like it could be hopeful, however it’s clearly not initially setup or intended for this purpose. It has a list of settings files to include:

$additionalSettingsFiles = [
 // e.g,( DRUPAL_ROOT . "/sites/$site_dir/settings/foo.settings.php" )
];

and then a loop to include any in the array:

foreach ($additionalSettingsFiles as $settingsFile) {
 if (file_exists($settingsFile)) {
 require $settingsFile;
 }
}

In order to include custom settings files per environment, I ended up adding a few lines between those blocks of code to dynamically add entries to that array depending on the environment.

// Load local docksal settings to override database connection details.
if (getenv('RUNNING_IN_DOCKSAL') == 'yes') {
 $additionalSettingsFiles[] = DRUPAL_ROOT . "/sites/$site_dir/settings/settings.docksal.php";
}

// Load QA specific settings.
if (getenv('RUNNING_IN_QA') == 'yes') {
 $additionalSettingsFiles[] = DRUPAL_ROOT . "/sites/$site_dir/settings/settings.qa.php";
}

However, I wasn’t entirely sure whether this was the best way or not, so I asked someone at Acquia what they thought. He said:

You could extend the logic as described to include an environment specific configuration file if you had lots of settings for each environment and wanted to store them in separate files - as long as the files aren’t picked up by BLT scan for settings.

Which pretty much confirms this is the most-correct way of including environment-specific settings files with BLT. The comment “as long as the files aren’t picked up by BLT scan for settings” is worth bearing in mind. This is referring to a piece of text in the Acquia Documentation “Acquia BLT globs the docroot/sites/settings directory to find all files matching a *.settings.php format”. This, of course, means that if you’re not careful with the naming of the files you may end up with them being included at all times, which isn’t what you want.

For this reason, I have made sure to name my files settings.<environment>.php which keeps them safe from that blob. You could also place the settings files in docroot/sites/default instead of docroot/sites/default/settings which would keep them out of harms way too.