Thread Concurrency – with One CPU?

[From the last episode: We looked in more detail at the characteristics of threads.]

Last week we ended with a question: we’re talking about threads running at the same time, in parallel, but… if you have only one CPUStands for "central processing unit." Basically, it's a microprocessor - the main one in the computer. Things get a bit more complicated because, these days, there may be more than one microprocessor. But you can safely think of all of them together as the CPU., how would that even work?

That’s totally not a dumb question. CPUs run one program or threadA piece of a program that can execute at the same time that some other piece of that same program can execute. at a time. So, if you have only one CPU – and most – likely all – IoTThe Internet of Things. A broad term covering many different applications where "things"  are interconnected through the internet. devices would have only one – then only one thread will be in process at any given time. So what’s with this “at the same time” thing?

Pseudo-Concurrency

In reality, it only seems like the threads operate concurrently; in fact, indeed, only one runs at a time. In some specialized cases, you may have the operating systemSoftware – or firmware – that handles the low-level aspects of a computer or IoT device. It gets direct access to all the resources. Software asks the OS for access to those resources. deciding which thread to run based on some kind of policy – like “round robin,” where each thread gets a certain amount of time and then hands over to the next one, with each thread getting an equal turn.

But that’s not how it works more generally. The ability to do this is based on one simple fact: some things take a longer time to do than others. Especially if you’re getting data from DRAMStands for "dynamic random access memory." This is temporary working memory in a computer. When the goes off, the memory contents are lost. It's not super fast, but it's very cheap, so there's lots of it. someplace (and it’s not in the cacheA place to park data coming from slow memory so that you can access it quickly. It's built with SRAM memory, but it's organized so that you can figure out which parts haven't been used in a while so that they can be replaced with new cached items. Also, if you make a change to cached data, that change has to make its way back into the original slow storage.). That can take a while (by computer standards – it’s a blink of an eye by human standards). So what happens while you’re waiting for that data?

If you’re simply running a program with no threads, then you sit around waiting for that data. But if there are other threads waiting to execute, then, while you’re waiting, you could swap in another thread and have it run for a while. Each thread is likely to “stallIdeally, a CPU executes instruction after instruction, with no breaks. But there are times when there may be a delay – getting new data from DRAM or having to reload the pipeline due to a branch, for example. When this happens, we say that the system briefly “stalled.”” here and there, waiting for some longer operation to finish. So, why not make good use of that time?

Keeping Things in Context

Of course, it’s not quite as simple as all that. For any running program or thread, you have what’s called a “context.” That context is all of the stuff in the CPU that’s related to what’s running. What things are those?

• Every processorA computer chip that does computing work for a computer. It may do general work (like in your home computer) or it may do specialized work (like some of the processors in your smartphone). has a bunch of “registers” in hardwareIn this context, "hardware" refers to functions in an IoT device that are built into a silicon chip or some other dedicated component. It's distinct from "software," which refers to instructions running on a processor.. Those are for super-fast storage – even faster than SRAMStands for "static random access memory." This is also temporary memory in a computer. It's very fast compared to other kinds of memory, but it's also very expensive and burns a lot of energy, so you don't have nearly so much.. It’s where you keep things you are working with right now, kept nearby for easy, fast access. It’s nothing you or I worry about; the compilerA software development tool that takes a high-level program (source code) and translates it into a low-level machine-language program (object code). takes care of it. But it’s important data related to what’s running.
• There’s a thing called a stackRelated to communications: A way of organizing parts of a complicated process (like communications) so that any task relies on tasks below it and feeds the tasks above it. Related to computing: A place in memory where you store “where was I?” information when you go from, say, one function into another. Before starting a new function, you store where you were in the old one so that, when the new function ends and you’re back in the old one, you can figure out where you were and continue on., and it’s a feature of memory (just like heap memoryA block of working memory reserved for use by programs as they request an allocation. The programs request and release memory from the heap as they go, and where specific data will end up in the heap is unpredictable. is). There are many things that go onto the stack, and they’re mostly related to keeping track of where you were before doing something else. For example, if you’re in one functionA small portion of a program that’s set aside and given a name. It helps isolate key code in an easy-to-understand way, and it makes changes easier and less error prone. It’s a really important way to make programs easier to understand. May also be called a subroutine. and that function calls another function, then you put various things related to where you were in the original function up on the stack, and then you run the new function. Once the new function completes, you want to go back to where you left off in the original function. You get back there by pulling that stuff back off the stack.

There may be other stuff, but those are the most important two elements of the context. When one thread stalls and you want to swap in another one, you have to do what’s called a context switchRefers to a CPU changing from one thread to another, with all of the related information from one thread being stored and replaced by the same type of information for the new thread.. And there are a couple of ways this might happen.

Software vs. Hardware Threading

If the thread management is done by softwareIn this context, "software" refers to functions in an IoT device that are implemented by running instructions through some kind of processor. It's distinct from "hardware," where functions are built into a silicon chip or some other component., then you simply take all of the context stuff and store it somewhere close by. You then get the (presumably already stored) context of the new thread you want to run and put it in place. Kind of like the stack, it’s storing the “where was I?” info. Or perhaps it’s kind of like bookmarking your place in a book before you go looking somewhere else in that same book (like a cookbook, perhaps).

Now, there are also CPUs that have built-in hardware support for multi-threading. In that case, they have multiple copies of the register set, for instance. So they don’t have to move so much stuff around; they just switchA switch helps direct network traffic to the right destination. At a high level, it's very similar to a router. Technically, switches are used to create local subnetworks; routers connect subnetworks together. from one to another. And, if they have multiple stacks, then that doesn’t need saving either. That way you can switch contexts much more quickly.

Why does that speed matter? Well, saving everything also takes some time. So you have to be stalled for quite a while for it to be worth going through all the context-saving hassle with software. With hardware threading, you can take advantage of shorter stalls to get more things done.

Some perspective here, however. Super-simple IoT devices probably won’t even have threads to start with. Ones that do probably use software. It’s the beefier CPUs that handle this in hardware, likely with support for 2 or 4 threads. You might find them in higher-end IoT devices; they’re much more likely in the cloudA generic phrase referring to large numbers of computers located somewhere far away and accessed over the internet. For the IoT, computing may be local, done in the same system or building, or in the cloud, with data shipped up to the cloud and then the result shipped back down..

Now… all of this changes if you have more than one CPU. We’ll start to look into that next.