*Many complex systems exhibit a phenomenon known as a tipping point. A tipping point is a non-linear event, when a small quantitative change triggers a significant qualitative change that is often irreversible. These moments of rapid and dramatic change play a prominent role in diverse fields of study, such as physics, sociology, economics, epidemiology, as well as software engineering.  Knowing about tipping points, understanding where and when they occur, helps organizations and individuals to anticipate and mitigate the significant consequences for teams and the applications they work on.*

As history repeatedly demonstrates, the complexity of software continues to grow as software becomes more broadly and deeply integrated into every aspect of our lives. Companies are quickly recognizing the need to provide a good developer experience to manage that complexity in order to boost productivity and increase developer satisfaction. While the need for a good developer experience is widely recognized, there is no consensus on its definition.

However, a common sentiment among software developers is that a project’s developer experience starts out delightful but eventually and suddenly becomes dreadful.

Let’s take a Django web application as an example: After you set up a Django project and generate and configure the initial application, development follows a straight path. First, you create models, controllers, views, and unit tests. Django does the heavy lifting to create an admin interface as well as database migrations when your models change. As the application evolves, you add more models, more controllers, more views, and more unit tests. Django makes updates to the admin interface and the database migrations.

We could describe the Django developer experience as linear: While the application grows quantitatively (more models, controllers, views, etc), it remains the same qualitatively (the architecture of the application is still expressed as models, controllers, views, etc). In other words, even though your application gets bigger, your developer experience does not change. Hence, the application grows linearly.

Until you hit the tipping point! Just one more requirement, just one more feature, a small quantitative change that triggers a significant qualitative change, the last straw that breaks the camel’s back, the last drop that overflows the barrel.

There are many factors that push us over the tipping point, but some are consistently at play: Implementing a multi-step process that spans multiple database transactions or orchestrates multiple services, a process that outlives a single request response lifecycle, or time-based actions.

For example, Django, like most web frameworks, supports only single-step processes well. Per request, the Django framework executes one request handler, which is a plain function. In turn, that request handler invokes one database transaction. However, how do we approach a task that does not solely execute one database transaction or one service call, or a task that outlives request response pair? How do we implement a task that requires two, five, or ten transactions that orchestrates two, five, or ten services or which runs for minutes, hours, days, weeks, months, or years?

Every project I’ve worked on hits a tipping point. We’ve all had the meeting where the tipping point becomes apparent. Sometimes we were junior engineers, and quietly listened. Other times we acted as senior engineers, and hotly debated terms such as orchestration, choreography, commands, events, job queues, job scheduling, or outbox patterns. Our arguments flooded the room, but regardless of our role, the imminent turn for the worse quickly became apparent. We had arrived at the tipping point!

At this point, Django no longer feels like my ally. Django feels like my adversary. Instead of empowering me, Django holds me back, I find myself needing to work around restrictions and limitations. Databases, queues, and timers are the duct tape holding my application together. Sagas, commands, and events are the crutches on which I drag myself through the day.

![Non-Linear Developer Experience](//images.ctfassets.net/0uuz8ydxyd9p/2gfRtq4DYuNLEIyvqfRVz8/add1a45b30106e37a1a2ffa1bcb95078/Non-Linear_DX.jpg)

I have two options: I can embrace the tipping point as inevitable or, I can reject the tipping point as inevitable.

By embracing the tipping point, I can either declare defeat now or later. If I am willing to declare defeat now, I begin building my application as if I’ve reached the tipping point already. From day one, I implement my application in terms of sagas, commands, and events, I orchestrate and choreograph, I enqueue and dequeue in anticipation that eventually I will have to go down that path anyways. Well, if my developer experience is dreadful from day one, at least I can’t be missing what I once had. If I am willing to declare defeat later, I begin my application as if no tipping point looms on the horizon. I soothe myself with euphemisms, I tell myself that if I don’t have to re-architect and re-implement my applications later, I am making a mistake now. KISS, keep it simple, stupid, YAGNI, you ain’t gonna need it, I can find comfort in an ecosystem armed with acronyms assuring me all of this is fine.

However, it’s not. If I embrace the tipping point, whether I declare defeat now or later, eventually quality, productivity, and satisfaction will suffer. Engineers will be writing scores of code to “plug into” hunting and fixing heisenbugs in the edge cases of the application, barely able to predict or reason about the consequences of releasing that bug fix. Unexpectedly, I will “lose” the best engineers on the team, they will no longer be doing creative work on the application, instead, the best engineers will now be doing creative work on libraries, frameworks, and platforms.

Nobody will be delighted.

Instead, I should reject the tipping point as inevitable. At this point in time, when most software runs in the cloud and on the edge, integrates with multiple and various 3rd party APIs, executes processes that run from a few seconds to many days, I should demand libraries, frameworks, and platforms that offer a linear developer experience, one that I will not outgrow as my project scales.

One that starts delightful and stays delightful.

![Linear Developer Experience](//images.ctfassets.net/0uuz8ydxyd9p/4mkj8V8wXwOyoF4KvTCpzM/a27c536aa1e0d9816030658873f2ff15/Linear_DX.jpg)

Tipping points can be a devastating event causing a sudden and consequential decline in the experience of your team, leading to a decline in quality, productivity, and satisfaction. By knowing about tipping points and understanding where and when they occur you are able to anticipate, mitigate and outright avoid their consequences.

Don’t let a tipping point derail your project. Be proactive and be aware of your developer experience. Adopt libraries, frameworks, and platforms that start delightful and stay delightful.

Look for a Linear Developer Experience


Linear & Non-Linear

The Linear Developer Experience

Principal Engineer

Dominik Tornow

dominik-tornow

#  How we built it: Building Scheduled Workflows in Temporal
Temporal’s new [Scheduler](https://docs.temporal.io/workflows#schedule) feature introduces an improved experience for developers with a schedule-specific API, finer-grained configurations, improved visibility, and the ability to have overlapping runs. For an overview of its features and use, Keith Tenzer, one of our Solutions Architects, wrote [a great blog post](https://keithtenzer.com/temporal/temporal_schedules/).

As this article is a contribution to our *How We Built It* series, I want to focus on the *Why?* and *How?* of our new Scheduler feature. As Keith covers in his post, there was a previous simple schedule feature in Temporal, but it had its drawbacks:

- Schedule couldn’t be stopped without affecting the running Workflow
- Workflow couldn’t be terminated without affecting schedule
- Schedule couldn’t be paused or have its configuration updated
- Couldn’t support overlapping runs
- Started in an unqueriable state
- No good visibility into running schedules

To improve upon these drawbacks, we knew we had a few options. 

We could build a separate scheduler system in the server using low-level components like mutable state and timer queues. This would be our most efficient option, but also the most work intensive. Ultimately, we didn’t go with this option. But, as a nod to its potential, we still designed the API with this implementation in mind, should the opportunity to switch present itself down the road. 

Another option was to build more developer-friendly functionality on top of the existing CronSchedule implementation. But, as this would mean a Schedule and a Schedule Workflow are still conflated, the problem of overlapping runs wouldn’t be solved. The inability to support overlapping runs put this option to rest.

The third, and prevailing, option was to build out the new Scheduler as a Workflow. The logic of the Scheduler process would look a lot like a Workflow even if we didn’t write it as one. And, if we did write it as one, we get to reuse a lot of functionality already in the server. Ultimately, this ability to reuse existing functionality proved a strong motivator and we moved forward with building a Workflow-based scheduler.

## Building a Workflow-based scheduler 

Deciding to build the Scheduler as a Workflow certainly wasn’t the last design choice we had to make. The next big decision we had to make was where to house the Workflow implementation. 

Do we implement the scheduler as part of the SDKs? What about as a separate library that users can link to in their workers? Or do we implement it directly on the Server workers?

SDK inclusion and shipping it as a library would mean re-implementing the scheduler Workflow in multiple languages (Go and Java don’t use our [Core SDK](https://temporal.io/blog/why-rust-powers-core-sdk)). And, in the case of implementing it as a library, users would have to worry about an additional library in their build process and potentially suffer a degraded experience from build complications.

All this considered, we decided to build the scheduler directly into Temporal Server. With this option, users wouldn’t have to install or add any additional components—they'd just make calls to an API. From a product perspective, implementing the API on the Server also gives us the flexibility to switch to a more efficient implementation later on.

### Challenges

Implementing the scheduler as a Workflow presented some unique engineering challenges. 

One of the first challenges we ran into was with Namespaces. Our existing internal Workflows run on the Worker role, in a Namespace called `temporal-system`. This is a local, not-replicated Namespace. But, our new scheduler Workflows correspond to entities (“Schedules”), which live in a user Namespace. So we asked ourselves: can we make it work if the Schedules entity lives in a user Namespace, but the Workflow that implements it lives in a system Namespace? 

We decided against trying to make this fit, primarily because of concerns around replication and migration. We support replicating a user Namespace to another Cluster and migrating active copies to new Clusters—as I mentioned above, the system Namespace is not replicated, so this would require building a lot of new replication functionality. But, if we were to run the scheduler Workflows in the user Namespace, we pretty much get replication for free.

So what did we do? 

We run the scheduler Workflows in the user Namespace. But what about the SDK worker in the Worker role? It only listens on a task queue in the system namespace. For now, we did the simplest thing possible: we just run more SDK workers! For each user Namespace, we run at least one SDK worker on one of the server Workers. There’s a new component that keeps the running workers in sync with registered Namespaces.

This may sound excessive but the SDK worker is relatively light, so hundreds of them in one process is not a problem. In the future, we plan to optimize this by allowing one SDK worker to handle multiple Namespaces and task queues at once (this may be the subject of a future post).

#### Hiding

As things stood, with the scheduler Workflows in the user Namespace, if a user were to list all Workflows, they would see these scheduler Workflows as well. We consider these an internal implementation detail and didn’t want them coming up in user queries. So we had to figure out how to “hide” these scheduler Workflows from user interactions. 

One option would be to simply exclude these Workflows from visibility. However, we need visibility to implement “list schedules”. We could push them to a separate visibility index, but our current visibility implementation only works with a single index, so this would require a lot of work. 

Then we thought, maybe we can just filter them out of the results of visibility queries that we get back from the database? But this led to pagination woes: a request for a page of 100 user Workflows might return only 95, or worse, 0, if only scheduler Workflows were found. But we were on the right track.

In the end, we decided to use a simple trick: a search attribute. We marked these Workflows with a specific search attribute, and added a clause on that attribute to all queries (unless a query mentions it specifically). To keep things general, we called this new level of grouping within a Namespace a "Namespace division." 

Seeing some benefits, we might later extend this Namespace division functionality to other elements of Temporal.

#### Signal vs. query

I noted above that two of the big improvements with this new scheduler is that it is stateful and can now be interacted with via a synchronous [gRPC](https://grpc.io/) API. So, under the hood, calls like `create schedule` and `delete schedule` are really `start workflow` and `terminate workflow`. What about `pause schedule`? 

Since commands like `pause` and `update` are changing the state of the Workflow to the effect that its behavior changes, they are implemented with **signals**. But, what about something like `describe schedule`? This got a little tricky. Part of the output of `describe schedule` is a set of the currently-running Workflows that were started by the scheduler. The scheduler knows what Workflows it started, but does it know which are still running? For the scheduler’s state to be updated, it would need to be notified when the Workflow is complete. But, the only things that can wake up a Workflow are a timer (or timeout), a signal, an Activity completion, or a Child Workflow completion.

The Workflows that are started by the scheduler are not Child Workflows; they're independent (this is because the scheduler Workflow is an implementation detail that we don’t want to expose and because Child Workflow completion notification doesn’t work across continue-as-new anyway). This means that the only reasonable way to get notified if a scheduled Workflow completes is to have an Activity that listens and returns when the Workflow completes. 

So based on this, we can query the Workflow for "describe", but the set of running Workflows it returns might include ones that have completed already but whose state has not yet updated. How did we avoid this?

Eventually, we’ll use a new feature that allows synchronous Workflow updates, but this is still in development. So for now, the “describe” implementation in the frontend service checks that all the “running Workflows” are still running. If any are completed, it sends the scheduler a signal, to tell it to wake up and refresh the state of all its running Workflows, and then it does the query again.

## Conclusion

Building our new scheduler as a Workflow was a great challenge and ultimately allowed us to build a feature that I really believe is going to improve the developer experience for Temporal users. 

If you want to learn more about using the new scheduler in your applications, [check out our docs](https://docs.temporal.io/) for SDK-specific information. 


clem-onojeghuo-T7gi cyrkdg-unsplash

Temporal’s new Scheduler feature introduces an improved experience for developers with a schedule-specific API, and the ability to have overlapping runs.

How We Build It: Building Scheduled Workflows in Temporal

Engineering

David Reiss

David-Reiss

Do you get nervous when you hear the term "distributed systems?" Does your mind immediately jump to computer science courses you struggled with or didn't take? If you've been building applications for the web or mobile devices, you probably understand more than you think about distributed systems.

If you've built an app that talks to a database, you've worked with a distributed system. If you've built a JavaScript client that talks to a backend application, you've created a distributed system. If you've used a third-party API to process payments, you understand important concepts about distributed systems. Whenever you use a website, you're using a distributed system.

A distributed system is where multiple computers or computing devices communicate over a network, working together to accomplish a task. The complexity comes in when you start considering how all those devices, like browsers, phones, servers, and smart devices, work together and the problems you'll face when things go wrong.

Let's look at one of the most basic examples: fetching data from another computer using an API call. Open Notify has a public API that shows you [who's in space right now](http://open-notify.org/Open-Notify-API/People-In-Space/). To display this information on a web page, you'll use a small amount of code to make a request to this API, get the results, and then filter the results to display them the way you'd like. 

You've dealt with problems like this more times than you can remember. Getting the results back is the easy part, and your experience tells you that many things can go wrong. What happens if this API:

* Is unreachable because there are network issues?
* Is unavailable because their service is down for maintenance?
* No longer recognizes your authorization key?
* Returns data in a different format than you expect because they made a change on their end?
* Times out or even fails under heavy load?

These are just some things you must think about when working with distributed systems. You've most likely thought about–and even dealt with–all of these scenarios and more. You know you'll have to handle authentication rejections. You'll probably need to handle a network timeout or a rejected connection and retry the connection a few times. Most importantly, you know that there is code in your system that you control, and there's code that you use that someone else wrote, and your code has to handle situations where their code doesn't respond the way you expected.

Like most concepts in software development, things get more complex when you go beyond the "golden path," where everything works and you encounter no problems. This is where the complexity in distributed systems lies.

During the past ten years, the process we use to develop applications has changed dramatically. Ten years ago, your average web application consisted of a large backend application that talked with a database. Occasionally, some JavaScript would make requests to those backends, but to the developer, this was all one large system, and the web framework they used abstracted a lot of the complexity away for them. But as business needs changed, applications grew in complexity, and developers began splitting their applications into smaller ones. Monoliths became groups of microservices, and developers had to start thinking about how these separate systems worked together. If you're a software developer who began your career in the past five years, you've already seen a lot of distributed systems concepts through your daily work.

For example, the scenario where you fetch data from an external system is a fundamental concept in distributed systems called a [Remote Procedure Call](https://www.w3.org/History/1992/nfs_dxcern_mirror/rpc/doc/Introduction/WhatIs.html), or RPC. It's where your program calls a function on another system. Over the years, you've seen all sorts of approaches to RPC, from SOAP to REST to gRPC. Just like there are design patterns for object-oriented programming, there are design patterns for handling various distributed systems scenarios.

Here are some common distributed systems patterns you may have encountered:

- **Replicated Load-Balanced Services (RLBS)**, or **[Load Balancing](https://en.wikipedia.org/wiki/Load_balancing_(computing))**, where you have duplicate services, and you use a load balancer to send requests to those services. This is one of the first distributed systems patterns many developers encounter because it's a common way to handle heavy traffic. For example, a company may deploy a web application to 10 servers worldwide, and load balancers route traffic to servers based on traffic, geography, or resource usage.
- **[Command and Query Responsibility Segregation (CQRS)](https://martinfowler.com/bliki/CQRS.html)**, where you separate read and write operations into their own models, using commands to write data and queries to locate and fetch the data. Applications with high transaction volume use this pattern, but it's also a good fit for managing immutable application state. The popular Redux JavaScript library [takes inspiration from this pattern](https://redux.js.org/understanding/thinking-in-redux/motivation).
- **[Two-Phase Commit](https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html)**, which is a pattern used to consistently update multiple data sources. In a two-phase commit, each data source is a node in a cluster. You first prepare all the nodes to receive the update, using one of the nodes to coordinate the transaction. Each node lets the coordinator know it's ready and available to receive and process the commit. Once every node is ready, you execute the transaction on all the nodes. Two-Phase Commit is a good strategy when you're concerned with the integrity of the data rather than resource usage.
- **[Saga](https://microservices.io/patterns/data/saga.html)**, where you have a series of local transactions and an event bus. Each local transaction sends a message or event to the event bus, which hands the process off to the next transaction. If a transaction fails, you execute a different set of transactions to roll things back. A corporate travel site where a customer can book a flight and a hotel at the same time might use the Saga pattern. If a customer reserves a flight but then decides to cancel because there are no available hotels, you'll need to cancel their flight reservation, which means rolling back the transaction for the flight.

As you read through these examples, you probably thought about situations where you encountered these patterns in the codebases you worked on. You may have used frameworks or products that used these patterns without realizing it. If you've written code to handle a transaction that involves a couple of separate systems, you may have used the Saga pattern. If you've worked on a web application that gets a lot of traffic, you may have dealt with load balancers and had to think about how to handle session data for your users reliably.

Modern applications *are* distributed systems, and you're working with them every day, adding features, fixing bugs, and improving the customer experience. At the same time, you're growing your knowledge about distributed systems as you solve these challenging problems.

You are a distributed systems developer. And now that you realize it, you should look at how Temporal takes care of many of the distributed systems patterns for you. It allows you to code at a new, higher level of abstraction, where you don't have to concern yourself with reimplementing these patterns. With Temporal, you get a reliable, fault-tolerant, and scalable system out of the box, and you can focus on just coding your business logic. Start with the [Temporal in 7 Minutes](https://www.youtube.com/watch?v=2HjnQlnA5eY) video, and then join the [Temporal Community](https://temporal.io/community).


distributed-systems-engineer

Do you get nervous when you hear the term "distributed systems?" If you've been building applications for the web or mobile devices, you probably understand more than you think about distributed systems.

You're already a distributed systems developer; it's just that nobody bothered to tell you

Director of Education

Brian Hogan

brian-hogan

This blog post gives some insight into Temporal's strategy for securing our cloud environment. It also calls attention to an unexpected facet of AWS access policies encountered along the way.

![temporal-access-control](//images.ctfassets.net/0uuz8ydxyd9p/140WQZN7K5OfvepujlfimK/5b495b9b2d7e1d6c8521f1565803c11f/temporal-access-control.png)

Finally, I'll come around to what I discovered—what I thought I wanted isn't what I needed—and describe what I need.

## The Challenges

Our long-term goal is to develop an access-control mechanism that can be flexible, scalable, and secure. After we reviewed various strategies, a [Biba-style model](https://en.wikipedia.org/wiki/Biba_Model) of segmenting our AWS accounts seems to be the best fit for us. Each account forms a hard ^1^ "perimeter" and provides an easy way to set, measure, and maintain the maximum "blast radius" of each account. If we assume an account has been breached, what is the maximum extent of damage an attacker can inflict? 

Setting up numerous smaller accounts can constrain possible damage, but this approach makes our management infrastructure a more compelling target. Configuring only one management account to allow those with access to assume roles into all other accounts greatly expands the blast radius in the event of a breach. **Clearly, this is not ideal.**

We want to provide strong isolation guarantees to our customers as well as ourselves. An engineer paged at 3:00 AM needs the confidence to fix the problem at hand—without the fear of causing more widespread problems. To provide these strong isolation guarantees, we plan to have a small number of trusted core accounts.

Think of these as Ring 0: interacting with the infrastructure of these accounts in any way should be a big event.^2 The next ring of accounts, Ring 1, will only listen to commands from Ring 0 and issue commands to Ring 2. Similarly, our core accounts—those Ring 0 and Ring 1 accounts—should not be able to read the data in those outer rings.

It's easy enough to describe this approach, but promises are meaningful only if you follow through with them! The first step for me was to codify this concept into our AWS service control policies (SCPs). SCPs are extremely powerful tools because they apply to all principals within an AWS account, including the all-powerful root user. Effective SCP layout and implementation is critical to a well-secured AWS environment.

## Digression 

Before SCPs became available, all AWS Identity Access Management (IAM) functioned via [discretionary access control](https://en.wikipedia.org/wiki/Discretionary_access_control). There was no way to restrict the actions of an AWS account. Controlling the permissions of subsections of an AWS account has been possible since the release of AWS IAM, but these were entirely the user's responsibility to put in place—and it was impossible to mandate that some actions always be denied.^3 With privilege escalation readily usable—for either legitimate or malicious purposes—within an account, it remains unwise to rely on a single AWS account.

When SCPs were released, AWS IAM gained the ability to set [mandatory access controls](https://en.wikipedia.org/wiki/Mandatory_access_control). An SCP applied to anything and everything in the AWS account, including the root user. It doesn't matter if the permissions are allowed within the account—SCPs are more powerful.

To escalate permissions within an AWS account, an attacker must first break into another AWS account and take actions to enable their attack on the initial target. AWS CloudTrail logs an event any time a user encounters an "Access denied" message due to SCPs, and CloudTrail can also log actions such as moving accounts between OUs and adding, removing, or modifying SCPs for an OU or account. In short, breaking into an AWS account this way requires a series of steps that are much noisier–and therefore more easily monitored.

## How Mandatory are SCPs?

Let's set the stage.

We have a group of AWS accounts. Some accounts need the ability to assume a role into a separate group of accounts to perform infrastructure/maintenance/control tasks.

We have the ability to **mandate** what an account is able to do. All that I need to do is mandate that our accounts operate in this manner. We will have our rings of AWS accounts, and AWS will handle the enforcement of what a principal can do. This is how AWS recommends SCPs be used, as guardrails around an account.^4

I wasn't worried about the positive case: granting the permissions of one account to assume-role into another. Instead, I wanted to test the negative case: disallowing any principal except for a particular role in an account in the correct ring to assume-role.^5

I reached for the tool that I assumed would be the right one: an SCP with conditionals.

```json
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Deny",
 "Action": [
 "sts:AssumeRole"
 ],
 "Resource": [
 "arn:aws:iam::575555303850:role/TestAssumedRole"
 ],
 "Condition": {
 "StringNotEqualsIgnoreCase": {
 "aws:PrincipalTag/AssumeRole": [
 "OK"
 ]
 }
 }
 }
 ]
}
```

My understanding was that this SCP would disallow the role in this account (creatively named `TestAssumedRole`) from being assumed, unless the Principal performing the AssumeRole API call had a key/value tag pairing of "AssumeRole/OK". The initial test was successful. **But when I removed the tag from the role doing the assumption ... it still worked.**

Surprising, right? Well, maybe not. IAM is complicated. My initial assumption was that I had somehow screwed up the Condition section. This is a common occurrence. If you haven't messed up an IAM policy yet, you will. My next step was to apply an SCP that denied _all_ API calls.

If you're struggling with permissions in AWS, check CloudTrail. From the CloudTrail record, it's possible to infer what conditionals might be useful or what underlying API calls need to be allowed or disallowed.

```json
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "DenyAll",
 "Effect": "Deny",
 "Action": [
 "*"
 ],
 "Resource": [
 "*"
 ]
 }
 ]
}
```

You had one job, SCP—don't let anything happen. Yet, this happened:

```bash
aws sts assume-role --role-arn 'arn:aws:iam::575555303850:role/TestAssumedRole' --role-session-name 'nope'
```

Credentials came back. Those credentials can't do anything, _but_ they still work.

```json
{
 "Credentials": {
 "AccessKeyId": "ASIAYMAOE7GVFJQZXUXH",
 "SecretAccessKey": "F1L82dBoYUbk9V...G1S9Yve6+GgbEiG3z5L",
 "SessionToken": "IQoJb3JpZ2luX2V...is2NDpHyKlAa",
 "Expiration": "2022-03-31T05:45:04+00:00"
 },
 "AssumedRoleUser": {
 "AssumedRoleId": "AROAYMAOE7GVP5EH66UUY:nope",
 "Arn": "arn:aws:sts::575555303850:assumed-role/TestAssumedRole/nope"
 }
}
```

Even with an SCP that denies _everything_, the AssumeRole API call succeeds and returns valid credentials. This is visible in the CloudTrail logs for the account; amongst a sea of Permission Denied errors is one successful API call: the one to AssumeRole.

![cloud-tail-success-noip](//images.ctfassets.net/0uuz8ydxyd9p/54Z9Xw0mjopGE88Bllb30f/5cabeae2289960134ba20fcc393673bf/cloud-tail-success-noip.png)

I contacted AWS Security. After some initial confusion, I was helped to understand where IAM in practice and my understanding of IAM differed.6 My mental model was that SCPs applied to an account, and even the root user cannot override an SCP's grants. This is true, but put on a lawyer-hat and read that again. "SCPs applied to an account" is true in a very, very literal sense.7 Another lesson: Not everything that shows up in CloudTrail has passed through the permission matrix that decides whether an action is successful.

To be extremely clear, it is a good thing that AssumeRole calls made against an AWS account are logged, regardless of source. It's a subtle nuance worth knowing.

Now, let's go back to the limitation of SCPs.

Initially, the SCP didn't seem to be working. As it turns out, this is by design. Technically, everything is working as it should. This surprised me, and I set out writing this post to help other people know about the times when SCPs don't apply.

You see, an SCP is applied to all principals in the account it is applied to. An SCP does not impact principals outside of the account. The initial, tagged, role exists in an account unencumbered by an SCP and therefore had no restrictions on assuming roles. Meanwhile, the credentials returned by the assume-role API call _were_ encumbered by the SCP and thus could not do anything.

With this understanding, it makes sense. An IAM role in a different account is not encumbered by this SCP and therefore will not have the `sts:AssumeRole` call blocked. But when a principal that is subject to the SCP tries to take an action, it _will_ be blocked.

This presents a problem for us. Although we can specify that our inner accounts can communicate to only the next ring of accounts, we have limited options for specifying which outer-ring accounts can accept commands from inner-ring accounts.8

There are currently no easy workarounds. The Assume Role Policy Document (ARPD) can be constantly updated to maintain a listing of AWS accounts allowed to assume the role the ARPD is attached to, but modification to the ARPD comes from _inside_ the account. We are back to the early days of IAM, when mandatory access controls were unavailable. It is possible to constantly, exhaustingly, manage this by assuming roles into an account and updating the ARPD attached to various IAM roles, but there is no dynamic way to prevent an account from listening to anyone except those who are trusted. Although it is certainly possible to use an SCP to prevent updates to an IAM role by applying a Deny policy, that is only a partial solution—and it still doesn't solve the problem. It can even generate more problems because the permissions needed to prevent malicious ARPD updates are also needed to create service-linked roles, maintain existing IAM roles, or even to remove unknown accounts trusted by the ARPD.

## The Actual Problem

Even if I spent the time and energy to build (and test) a system to keep everything updated, and it was automated, and I could define/tag various accounts and specify who is allowed to access them, and AWS IAM enforced that for me, I'll still have a problem: vendors.

Really, what I want is an official way to know who owns an AWS account. Scott Piper runs the ["well known AWS" text file](https://github.com/duo-labs/cloudmapper/blob/main/vendor_accounts.yaml), but I could just edit it to claim an account number.9 There's no formal verification here. Further, absence of an entry in this file is no indication either way of the trustworthiness of the account. AWS should be providing that capability so that vendors can give out trusted account numbers for me to verify before establishing that IAM role can assume an IAM role in my account.

There are implications even beyond simple IAM roles. For example: How can a Temporal customer know, if they want that from a trustworthy source. The actual item today is called the "Assume Role Policy Document," and that is very literally true. But we (as a community) have been calling it a Trust Relationship or Trust Policy for so long we've blurred the lines too much.10 Having an ARPD is fine, but I as an operator of the system need to:

1. Plan the update process
2. Update all those documents for all my accounts
 * Move an account out of an OU that prevents IAM modifications by SCP
 * Make the change
 * Verify the change
 * Move the account back into the production OU
3. Monitor for drift or attempts to cause drift
4. Update list for the steps I forgot the last time
 * Fix inevitable bugs in the process
5. Repeat

That's hard work, but I'm not afraid of hard work. However, what I really need is a way to verify that an account number is actually who I think it is, instead of trusting documentation. **AWS should provide a verified account number service.**

## Conclusion

My end goal is the same as when I started: to define the trust relationship from one account to another. But currently, all I have is an ARPD, and it is up to me to manually calculate that trust relationship. I would like AWS to handle some of that for me, to make it easier for me to know who I can trust (and then apply those trust relationships across all of Temporal's accounts!).

-----

 1Relatively speaking.

 2Sets off pagers, enforce a "no lone zone", etc. The sort of place you stay on the well-defined path and don't go directly accessing the AWS APIs.

 3[Subjunctive mood](https://en.wikipedia.org/wiki/Subjunctive_mood)

 4[How to use service control policies to set permission guardrails across accounts in your AWS Organization](https://aws.amazon.com/blogs/security/how-to-use-service-control-policies-to-set-permission-guardrails-across-accounts-in-your-aws-organization/)

 5There's a joke in here somewhere about security people and testing the "unhappy path," the development of which is left as an exercise for the reader.

 6[Vulnerability Reporting](https://pages.awscloud.com/GLOBAL_GC_vulnerability-reporting_2021127_7014z000000rnU1.html) form. I can't imagine the volume of inbound flow this receives; I cut the initial responders some slack, especially because even I didn't understand how IAM worked.

 7Please remove your lawyer-hat now.

 8The boundaries AWS recently released are SCP-based and therefore work only on principals that exist in an account; they cannot do anything about the outside world wanting to come in.

 9Remember, AWS account numbers are not secrets!

 10The official AWS documentation for [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) call it a "trust policy", but the actual API call only calls it an "[assume role policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/update-assume-role-policy.html)." Amusingly, even the documentation admits this confusion: "This is typically referred to as the 'role trust policy'."

temporal-access-control

This blog post gives some insight into Temporal's strategy for securing our cloud environment. It also calls attention to an unexpected facet of AWS access policies encountered along the way.

A Deep Dive into Temporal's Access Control Strategy in AWS

Staff Security Engineer

Brandon Sherman

brandon-sherman

![m0ck8jsfmdaashl3c77q](//images.ctfassets.net/0uuz8ydxyd9p/4GDbnMi9GFdlNW3KRD6oOJ/1a6b0d0654920ce76b0214f05ff4b9f7/m0ck8jsfmdaashl3c77q.png)

At first glance, I made the same assumption that I see many other people making - that Temporal executes their workflow code. As I looked deeper, I learned that the execution model of Temporal is actually more interesting, and more powerful. I figured I should write a post describing the execution model at a high level. Hopefully, this will help give people a starting mental model for Temporal’s core architecture.

I call it: **Inversion of Execution**.

## Disclaimer.
I am not saying that Temporal was first to implement this kind of execution model. It wasn't. There were earlier products that used similar ideas. I'll leave the genealogy research for an aspiring PhD student to perform. Somebody probably wrote a paper laying all this out back in the 1970s.

## Architecture
Temporal is a workflow engine. That made me immediately think about how I would deploy my workflow (application) code to it, how it ensures security, performance isolation, versioning, and all other concerns that accompany running somebody else's code. I see many people making the same set of assumptions and having similar concerns, in the beginning of their Temporal journey.

In reality, **Temporal doesn't execute application code at all**. It "only" orchestrates execution of your code in order to drive workflows to completion. Application code leverages Temporal via the client SDKs in such a way that it's not even immediately apparent where the Server gets into the picture.

In other words, there is an **Inversion** of the standard **Execution** model we are used to.

That's how the analogy with Inversion of Control came to my mind. Instead of injecting dependencies, Temporal "injects" steps of execution (known as tasks) into application code. Temporal can run (and, if necessary, rerun) those steps to overcome intermittent failures of outgoing calls and to recover from infrastructure failures. There are two kinds of tasks in Temporal: Workflow Tasks and Activity Tasks. They have different purposes and use separate persistent queues for reliability and scalability.

This is the topology of a typical Temporal-based system:

![uiniawopp0cgrloiy777](//images.ctfassets.net/0uuz8ydxyd9p/PXNpGHUJXn5rYY1VBGn0e/e1d23ff3917cbafbff83cdf460b67327/uiniawopp0cgrloiy777.png)

Temporal Server is usually run as a cluster of servers for reliability and scalability. The Server is backed by a database for persisting state.

None of your application code ever runs on the server. Instead, application logic runs on the client side of the picture. It is also usually organized as a cluster of Worker Processes, for scalable execution.

The umbilical cord connecting client to the server is the Client Runtime piece (a.k.a. SDK). Don’t underestimate it based on the name. Unlike typical SDKs that are mostly language-specific wrappers around a set of server APIs, Temporal SDKs do much much more than that. They contain the complex machinery that hides all interactions with the server and handle the non-trivial process of reconstructing client state after a failure. The goal is to free the business logic from all of that complexity, and enable the application code to be concise and easy to write.

## Bank Transfer Example
Let's look in more detail at how Temporal actually executes workflows, using the canonical example of transferring a sum of money from one account to another. While a real financial transaction may involve many steps, we will keep it simple and just talk about performing one debit and one credit operation. That should be enough for showing the mechanics of Temporal.

A workflow like this is typically started by a client process, for example, a web frontend, with a single gRPC call to the server through the SDK.

Go:
```golang
we, err := client.ExecuteWorkflow(context, workflowOptions, money.TransferFunds, params)
```

Java:
```java
WorkflowClient.start(transferWorkflow::transfer, from, to, reference, amountCents);
```
If this call succeeds, the money transfer workflow has been accepted by Temporal Server. Even if the client process were to crash immediately after that, Temporal will drive execution of the workflow to eventual completion or failure.

Under the covers, Temporal Server already persisted a record of the workflow with its arguments, options, and unique execution ID. As part of that transaction, it puts a Workflow Task into the corresponding Workflow Task Queue. This task contains a `WorkflowExecutionStarted` event as a first event in the workflow’s history.

Workflow implementation code (in our example `transferWorkflow::transfer`) is compiled into a Workflow Worker. When a Workflow Worker process starts, it connects to Temporal Server via Temporal Client Runtime (a.k.a. Temporal SDK) and starts long-polling the Workflow Task Queue for tasks to execute. That's how it receives the Workflow Task with a `WorkflowExecutionStarted` event inside from the server.

Note that the Workflow Worker didn't even have to be up and connected to the Server when the workflow was submitted by the client. It could connect later or experience a temporary outage. It doesn't matter; once the application worker connects, it will pull the task from the queue and start executing it.

![71e0saj3qeavgwf3phsw](//images.ctfassets.net/0uuz8ydxyd9p/7xHzChajKbTVGqmT2JY4Q3/91d711f360ce6b7358a581b7c64a086c/71e0saj3qeavgwf3phsw.png)

A Workflow Worker (there's usually more than one worker process running for redundancy and scalability) receives the initial Workflow Task that contains a `WorkflowExecutionStarted` event as the first event of the workflow history.  This event includes a `workflowType.name` field that indicates which workflow type Client Runtime needs to invoke. Runtime performs a lookup in the map of registered workflow types and calls the corresponding application function that starts actual execution of the workflow logic.

Workflow code defines a sequence of steps, not necessarily linear, that need to be executed for a workflow to complete. Some of these steps are actions that involve communication with external systems. In our example they are Withdraw and Deposit functions on the two given accounts. Such functions are referred to in Temporal as Activities. They are the second most important concept in Temporal after Workflows.
Workflow code invokes an Activity by calling `workflow.ExecuteActivity` function in case of Go:

```golang
future := workflow.ExecuteActivity(context, money.DebitFunds, params)
```
or by invoking Activity stubs in case of Java:
```java
@Override
public void transfer(
    String fromAccountId, String toAccountId, String referenceId, int amountCents) {
  account.withdraw(fromAccountId, referenceId, amountCents);
  account.deposit(toAccountId, referenceId, amountCents);
}
```
Execution of Activities is tracked and orchestrated by Temporal Server, very similar to how Workflow Tasks are handled. When `account.withdraw()` is called in our example, Client Runtime sends a command to the server behind the scenes - a request to execute an Activity of the given type with the provided arguments. This request gets recorded in the Workflow History as an `ActivityTaskScheduled` event, and the corresponding Activity Task gets added to the Activity Task Queue. These updates are performed transactionally, so that the Workflow History and Task Queues are always in sync, and no task can be lost.

The Activity Task then gets picked up and executed by one of the Activity Workers long-polling the Activity Task Queue. Activity Workers are conceptually similar to Workflow Workers, and are often combined into a single Worker process. Even different Workflow and Activity types can be isolated into their own Worker processes, for security, performance, or other concerns.

The important part here to stress is that **Temporal Server records the intent to execute application steps before their actual execution and then records the results they produced after they complete**. This makes it possible to resume execution of application logic after any failure, from the step that previously failed. These recorded application steps, Workflow and Activity Tasks, are then passed to Application Workers for execution. Hence, the notion of **Inversion of Execution** - an application process is told by Temporal Server (via the Client Runtime) what tasks to execute when.

## Benefits
This model of execution brings a number of benefits.

### 1. Reliability
Each step of a workflow execution is recorded first with all its inputs, so that the workflow can resume from where it left off after any failure or even a complete shutdown of the system. When an Application Worker starts after a failure or a planned update and receives a task to resume execution of a workflow from a particular step (next after the last successfully executed), the Client Runtime deterministically replays execution of the Workflow Tasks that already succeeded, so that it can continue as if there was no failure or restart. Interestingly enough, the application code doesn’t even know if it’s executing a Task that previously failed or running it for the first time or replaying a task that already succeeded. That's why the Temporal programming model is often referred to as Fault-Oblivious - because application code indeed can stay mostly unaware of recoverable failures.

### 2. Security
The separation of concerns provided by the Inversion of Execution approach where Temporal only handles orchestration of execution without actually executing application code, makes it possible for applications to run with zero trust of Temporal. Every payload (arguments and results) of Workflows and Activities can be encrypted on the Application Worker side (via the  [Data Converter](https://docs.temporal.io/concepts/what-is-a-data-converter) feature), so that Temporal Server has no way of knowing what application is doing and what data it is operating on. This clean separation makes security and compliance reviews much easier to conduct because the zero trust story is very compelling.

### 3. Scalability
Execution of each workflow is independent and logically isolated from all other workflows in the system. Temporal is only responsible for recording execution steps and communicating with application workers. Because of that, Temporal Servers are relatively easy to scale out by adding more servers and redistributing responsibilities for different workflow ID ranges (shards) between them. Of course, storage needs to be able to scale up or, better, out accordingly.

Scalability is made easier because Temporal is not responsible for hosting and executing application code, and only needs to scale with the rate of execution tasks being generated and recorded. Scalability of application code (workers) is handled separately, and such clear separation makes it easier to find out and eliminate bottlenecks.

## Is this a queueing system?
If you think that this looks like a task queuing system, I wouldn’t say you are wrong. Temporal can be viewed as a specialized queue of execution. In a way. But if you ever implemented a high-throughput reliable message processing system that used queues, I'm sure you’ve had to solve a number of non-trivial problems in order to achieve good performance with strong consistency. Retries and backoff in case of failures, deduplication of reprocessed messages, how to handle a failing or slow message that is blocking a queue partition, dead-letter queues and violation of ordering guarantees, etc. These are hard problems to solve.

Temporal provides an opinionated set of solutions to these problems, and raises the level of abstraction, so that application developers don't have to think about them. Inversion of Execution is one of the core opinionated decisions, and I hope this post helps you see why.


m0ck8jsfmdaashl3c77q

If you think that this looks like a task queuing system, I wouldn’t say you are wrong. Temporal can be viewed as a specialized queue of execution.

Inversion of Execution

Sergey Bykov

sergey bykov

Several companies offer compliance automation platforms—[Vanta](https://www.vanta.com/), [Strike Graph](https://www.strikegraph.com/), [Drata](https://drata.com/), and others. Depending on who you ask, these platforms are either the best thing since sliced bread or they are selling snake oil. The truth, from my experience, is somewhere in the middle, with many of the pros and cons of these platforms shifting depending on your company's experience level.

To be clear, System and Organization Controls (SOC) 2 makes up most of my experience when it comes to compliance, so my current view is through that lens. But I would argue that the goal of any compliance effort—SOC 2, HIPAA, or ISO 27001—is the same: establishing customer trust. (See the following section, "Why compliance frameworks are important," for my take on this.)

## Why compliance frameworks are important

If you already understand this, feel free to skip this section. But if you are new to compliance and are trying to figure why you should even care (and how it translates to customer trust), this explanation might help.

A technology company can have a great story around security, integrity, availability, privacy, and confidentiality without ever thinking about compliance.

Is it likely to happen organically or be maintainable as the business scales, or adjusts to meet market demands? Probably not.

That would require significant levels of contribution, coordination, and discipline from many experienced individuals, routinely, even in a small (20–50 person) company. Personnel, products, processes, and user needs constantly change. It is possible, but nearly improbable to maintain and scale best practices in such a dynamic environment without some sort of framework to fall back on.

### Compliance frameworks

A compliance framework gives organizations something to fall back to. Let's take SOC2 for example, which stands for Service Organization Control 2 framework. The SOC 2 framework is just a list of criteria, established by the technology service industry, that represents best practices and guiding principles.

The idea is that an organization can map their current processes to the list of criteria to validate that what they are doing is ideal, or to find gaps that could leave them exposed to some sort of risk (think data breach, or a malicious attack on a system's availability). To ensure that mapping is done in an objective fashion, you pay a third-party auditor to review your organization's processes and issue a report on their findings.

### Controls

A Control (particularly with SOC 2) is a statement that captures the essence of what an organization does—the behavior, if you will, of some aspect of the organization. The scope of a Control can be broad or extremely granular. For example, a broad Control might be one that declares that the "organization conducts a formal risk assessment every 6 months," while a granular Control might declare that "a vulnerability scan is performed on the code base for every change that is requested to be merged into the main branch of the core SaaS repository."

An auditor then:

* Looks at the set of Controls that the organization has declared
* Maps them to the compliance framework criteria
* Identifies any gaps of coverage
* Analyzes evidence to determine whether the organization is adhering to their Control Statements

### The report

The auditor then issues a report. In their report, they include their assessment of the Controls and evidence, as well as a "system description" of the organization that describes the company and its products in a fairly high degree of detail.

The report that the auditor issues is—more often than not—an extremely comprehensive analysis and honest view of the organization and how it operates.

### Conclusion

**In recent years, a compliance report (such as SOC 2) has become a cornerstone factor in whether a customer decides to place their trust in the organization**. In a lot of cases, many businesses won't even consider a B2B relationship if the other business is not compliant within a framework

That is why you should care. If your organization is compliant within a framework, it is following best practices to the best of its abilities, and thus your customers can feel good about placing their trust with you.

## So what role do these platforms play in helping you do that?

Well, each one has a slightly different angle. Almost all of them promote themselves as a way to "simplify" the compliance process by providing a set of tools to help you organize your resources and automate compliance processes, particularly around an audit. Personally, I think you should be wary of any of these platforms that say they can make things "simple" for you, because that is silly. This stuff is complex. What I _do_ value are platforms that can help you understand how your internal operations map to compliance framework criteria. That is the art, the heart, and the best-kept secret of this whole thing that is so often glazed over.

Consider this well-known adage:

> Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime.

Most of these platforms teach only when they have to, as a means of getting you to pay to use their tools. They don't want to teach, they want you to keep coming back to them. The truth is, after you learn how to parse the language of compliance framework criteria and how to map real-world operations to it, you don't need these platforms.

Now, to be fair, learning this is not super easy.

I recently asked a SOC 2 expert, [Martin Cozzi](https://www.marana.io/), how he learned. He said it was the same way I was learning: through experience. (He has been through more than 50 audits.) He knows of no course that teaches this.

To me, what makes a compliance audit easy (instead of a slogging and demoralizing haze) is not whether you have fancy automation tools, a fancy web UI, or an all-in-one auditing platform. What matters is whether your company has embraced the framework against which you want to be compliant and understands how your operations fit into it. That's it. You either know what you are doing, or you don't. And the unfortunate truth is that many of these platforms take advantage of those who don't know what they are doing and sell them a set of tools that are effective only if you know what you are doing.

What these platforms could (and should) be doing is teaching.

## Keys to a successful compliance program

I have now been through two SOC 2 audits with Temporal Technologies. Throughout each of them I acted as the compliance lead, a sort of project manager. My primary responsibilities were to organize materials, gather evidence for audits, schedule meetings, and generally create a structure to operate in. I was also the auditor's point of contact with our company, funneling and prioritizing communications, and translating requirements into action items for our team. I kept other key players up to date. And if necessary, I pushed them when deadlines closed in.

I have worked with both [Strike Graph](https://www.strikegraph.com/), one of the compliance platforms, and [Martin Cozzi](https://www.marana.io/), an independent consultant.

From these experiences, here are my five key things that shaped our successful compliance program.

### 1. Retain the help of an expert to get ramped up

Compliance programs offer little room for trial and error, and they have too many pitfalls to overcome through self study. So, I definitely recommend getting someone with experience to help.

For startups and other small companies that do not have in-house experience yet, platforms like Strike Graph (for SOC 2 at least) are actually not a bad way to go. They can help you get going for a lot less money than a dedicated consultant. As an example: Strike Graph maintains a team of customer success agents who can steer you in the right direction by providing a lot of context and boilerplate policies.

But just remember they are not in the business of understanding your business, and therefore they most likely will not be able to help with precise custom Control Statements that scale for you. Additionally, they may not be able to help accurately translate the expectations of your auditing partner into action steps.

And using a compliance platform means that you have yet another external, third-party system to sign in to and learn how to use. Managing Control Statements and evidence outside of normal business tools often means "out of sight, out of mind" and greatly reduces visibility into compliance processes internally.

If you can afford a personalized consultant, such as Martin Cozzi, I would highly recommend that route.

Consultants like Martin take the time to understand your business in conjunction with the expectations of the auditing firm, to craft a compliance strategy that is best suited to your company. This also means that they can translate auditor expectations into actionable steps for you. They can help write custom Control Statements that accurately reflect what you are doing and map them to compliance criteria. And my favorite part: Consultants like Martin can show you how easy it is to manage compliance materials by using the project management tools that _you already use_. If you don't use project management tools, all you need is a spreadsheet. When you are oriented in the compliance world, you don't need a fancy UI and tools to keep track of the state of your compliance efforts.

### 2. Do what you say you are doing

When you write a Control Statement, it must be one that is accurate and true. You want your implementation of that Control to be done in a way where you can prove that you did those things during a period of time.

For example, if you have a Control Statement that says "The engineering team is notified whenever there is a release to production", then (A) that should actually be happening, and (B) you should be able to prove that.

But really think about this statement for moment, because it forces you to define the following: - What is the "engineering team"? Who is in that scope? - What is our method or mechanism of notification? - What is considered a "release to production"?

By ironing all of that out and ensuring that you have a history of evidence to gather, this Control Statement (and the associated actions you take) can scale with the company for a very long time.

### 3. Internally manage Control Statements and resources

This one piggybacks a bit on the first key to success and draws a line in the sand about deciding whether to use a compliance platform. I feel strongly about it because I have seen just how much more engagement occurs across the company when Control Statements and resources are managed using "normal" and "routine" tools.

For example, at Temporal Technologies we use [Notion](https://www.notion.so/) for just about all of our internal documentation and project management. With just a little bit of effort, we were able to track our compliance efforts by using the same project management tools in Notion that we already use for Engineering and Product efforts.

As it turns out, this was a pretty big deal, because everyone in the company has access to Notion. Everyone can see the Control Statements, what criteria those statements map to, the policies that we wrote for processes to adhere to, the progress of any given compliance effort, and so on. This level of transparency also means that we can easily see how any internal task rolls up under a compliance-related effort, or has been generated by it. Essentially, this approach puts the compliance program right at the heart of our operating mindset, which is so critical for the long term.

### 4. Use a reliable auditing partner

Many companies offer auditing services, but finding the right one for your company can be a challenge. That's because each company does things a little differently.

We have worked with two different auditors so far, and for us we found our experience with [Schneider Downs](https://www.schneiderdowns.com/) to be really good. Martin recommended them, and he has worked with them for a few years.

The pros that we really liked were:
- They were highly communicative.
- They had previous experience with developer-facing SaaS companies.
- They had a polished system for evidence submission that gave us clear visibility into the status of our audit.

I definitely cannot say that this is the case for the other companies we talked to and worked with previously.

### 5. Spread out the workload

While spreading out the workload helps to prevent any single person from getting overwhelmed, it is also about knowing the roles that individuals play in the compliance effort. We found that any given person generally falls into one of three roles.

#### The Department Head

This is a very important role. The Department Head's responsibility is to make sure that people are doing the things we say we are doing. The Department Head must have a good sense of Control Statements that fall into their area. They are the "buck stops here" person in terms of driving their department to think and care about the compliance process.

In our last period we had four Department Heads who were heavily involved in the process:
- HR
- Engineering
- Security
- Product

#### The Project Manager

As I mentioned before, the Project Manager has the responsibility of generally creating a structure to operate in.
- Organize materials, gather evidence for audits, and schedule meetings.
- Be the auditor's point of contact with the company, funnel and prioritize communications, and translate requirements into action items for the team.
- Keep other key players up to date and push on them when deadlines close in.

As the Project Manager for Temporal's SOC 2 efforts, I created and maintained a section in Notion for all of our policies, Control Statements, and evidence aggregation. I established a routine way of escalating requests for evidence and scheduling decision-making sessions. I also worked very closely with Martin, picking his brain, to provide internal guidance when needed.

#### The Evidence Gatherer

The Evidence Gatherer is someone who captures evidence based on a request for the auditing process.

An Evidence Gatherer must be someone who is familiar with the domain to know what to gather and how to gather it, working with both the Project Manager and the Department Heads to do that.

In our last audit, we had to submit well over a hundred pieces of evidence, most of them unique, and some large in scope. We had at least a dozen Evidence Gatherers, including Department Heads, and me as the Project Manager.

## Summary

So with all that said, if I had to reduce my sentiment to a single sentence it would be this:

> Embrace and learn your compliance framework; do not outsource the responsibilities of understanding it.

Does that mean you shouldn't ever use one of the compliance platforms with a shiny UI and automation tools? No, it just means that if you choose to use one, make sure it is integrated into the heart of your business and its operating processes. Try to extract from them every ounce of knowledge you can. Just like an individual needs to be their own advocate when it comes to their health and wealth, a company needs to be its own advocate when it comes to understanding its operating processes in the context of any given compliance framework.

If you know what you are doing, a compliance platform could help you. If you don't, it could become just another complex layer blocking you from figuring it all out. A good compliance program doesn't need fancy tools and a UI. It just needs internal accountability.


maxim-berg-ZESbrsLBPPk-unsplash

 I would argue that the goal of any compliance effort—SOC 2, HIPAA, or ISO 27001—is the same: establishing customer trust.

Good compliance programs don't require a fancy platform

SDE Technical Writer

Cully Wakelin

cully wakelin


Gremlin is at the forefront of the fast-rising Chaos Engineering movement. Chaos Engineering involves running thoughtful, planned experiments that teach us how our systems behave in the face of failure, and Temporal provides a testable, scalable solution to handle failures. Our co-founders Maxim Fateev and Samar Abbas recently joined Gremlin's podcast [Break Things on Purpose](https://www.gremlin.com/podcast/) for a rare double appearance to talk Temporal.
[Listen to it here](https://www.listennotes.com/podcasts/break-things-on/maxim-fateev-and-samar-abbas-C1p42zDwLZX/)!

What follows is a lightly edited transcript for readability.

## Origin Story of Temporal

***So, you started with Cadence, which was an internal framework at Uber, and decided to strike out on your own and build Temporal. What caused you to strike out on your own?*** 

We built Cadence from the beginning as an open-source project and grew by bottoms-up adoption within Uber. **It grew from zero to over a hundred use cases within three years**. But because it was an open-source project from the beginning, **after a year or two we started to see companies like HashiCorp, Box, Coinbase, Checkr, adopt us**. And there are a lot of others, it’s just that not all of them are public. 

We believe this technology is very widely applicable, so we needed a separate entity, like a company, to actually drive the technology forward for the whole world. For example, Uber would never create a cloud offering, and everyone wanted us to host Temporal. So one thing led to another and we ended up leaving Uber and starting our own company. And that was the main reasoning — **we wanted to make this technology successful for everybody in the whole world**, not just within Uber. 

Another benefit of starting Temporal was that **we had actually accumulated a pretty large technical debt when running Cadence**, just because we ran Cadence without single backwards-incompatible change since we first put it in production. Even after four years we were still on the same cluster with the same initial users. So, we had to do everything in backwards-compatible manner. 

At Temporal, we could rethink our design decisions, and we spent almost a year just working on [Temporal v1.0](https://docs.temporal.io/blog/temporal-v1-announcement/) and doing a lot of fixes and tons of features which we couldn’t do otherwise. Because that was our only chance to do backwards-incompatible change. 

## Lessons from Uber: Reliability on Rails

One of Uber's values was to **provide rides as reliable as running water**. And that translated into interesting system requirements for engineers. Most of the time, what ended up happening is product teams at Uber were spending a large amount of time building this resiliency and reliability into their applications, rather than going after building real features that the users of the platform cares about. This is the problem that we were trying to solve back at Uber, where let us give you that reliability baked into the platform.

**Why does every engineer needs to be a distributed systems engineer to deal with all sorts of failure conditions?** We want application teams to be more focused on building amazing applications, which makes a lot of sense for the Uber platform in general. And this is the value proposition that we were going after with Cadence. It hit a nerve with all the developers out there, especially within Uber. 

One funny story is that one of the early use cases moved onto Cadence was because of a multi-day outage in one of the core parts of that system, and **the way they mitigated the outage is they rewrote that entire system on top of Cadence in a day**, and were able to port over that entire running system in production. And that’s how they mitigated that outage. That was the developer experience that we were striving for.

## Orchestration vs Choreography

***What’s different about using Temporal compared to distributed systems built with simple messaging like Kafka?***

A lot of systems are built with **choreography**: 

- You have a bunch of callbacks which listen on queues, then update certain data sources’ databases, and then put messages into the queues back.
- In a real system you also need to have durable timers, so you either build your own timer service, so you just poll your databases for messages to be in certain state to account for time. Getting this right is non-trivial.

This choreography in theory kind of works, but in practice is usually a mess:

- Your system now has a bunch of independent callbacks and you essentially have a **very complex state machine.**
- **Business requirements are broken** into a thousand little pieces which need to work together, and all the other requirements about retries and so on are actually pretty hard to get right.
- On top of that, you have very **poor visibility** into your system — if something goes wrong, good luck finding the problem.

**Orchestration** means that you implement your business logic in one place and then you just call into these downstream services to implement the business logic: 

- We know how to do that for short requests. Let’s say you get a request, your service makes the five downstream API calls, does something with those calls, then returns data. If this transactions takes, let’s say, a second, it's pretty easy to do, and you don’t care about reliability that much if it fails in the middle.
- But as soon as any of those calls can fail for three minutes, or, a downstream call can take ten hours, you now have to break this nice piece of code into 50 callbacks, queues, and state tables in database, and so on.

**Temporal helps you keep that code as is**. The main abstraction, which is non-obvious to people is that it makes your process **fully fault-tolerant, including stack variables**, threads, and so on. 

- So, if you make a call that takes five hours, you’re still blocked in exactly the same line of code. And in five hours, this line of code returns and then continues.
- If you’re calling `sleep` for one month, you’re blocked on this `sleep` line of code for one month, and then it just returns to the next line of code.

Obviously, there is some magic there, in the sense that we need to be able to persist and replay the state of your Workflow in exactly the same state, but this is exactly what Temporal provides out of the box. **You write code as if failure doesn’t exist** because if your process fails, we just reconstruct the exactly the same state in a different process, and it’s not even visible to you. I sometimes call it **fault-oblivious programming** because your program is not even aware that a fault happened. That is main idea: fault-tolerant code that is guaranteed to finish execution.

## Temporal Features and Use Cases

There is a lot more under the hood:

- **Task Queues**: We don’t invoke services directly, usually we invoke them from queues. But these queues are hidden in a sense, because when you call some API in Workflow code, it gets called asynchronously. But in your code, it’s not visible. It’s just a normal RPC call, but behind the scenes, it’s all asynchronous, has **infinite retries, exponential backoff, heartbeating for long-running tasks**, and so on.
- **Local State**. Workflows are stateful because you can keep state in variables and you don’t need to talk to a database.

Imagine you want to implement the customer loyalty program for airline, so you need to implement a point system. So, you listen to external events, and every time your flight finished, and you will get an event to increment that. In a normal system, you need to get a database, queues, and so on. In our world, you would just increment local variables. And then when this counter reaches a hundred, for example, you will call some downstream service and say, “Okay, promote that person to the next tier.” And **you could write this type of application in 15, 20 minutes** on your desktop because all of the rest is taken care of by Temporal.

It assumes that you can have hundreds of millions of these objects simultaneously running because you have a lot of customers—and we do that because we built it at Uber, which had hundreds of millions of customers—and run this reliably because you don’t want to lose data. It’s practically your financial data. This is why Temporal is very good for financial transactions, and a lot of companies like Coinbase, for example, uses us for their financial transactions because we provide much better guarantees than alternative solutions.

## Designing For Temporal

***If I wanted to implement Temporal within my application, what do I need to keep in mind to ensure that I have a good experience with it? Any tips?***

### Redesign End-to-End

Maxim: Temporal does best when you also rethink how your application is structured. It’s kind of a new category of software, so you cannot just take your current design and directly port it over. One time, engineers from a payment system came to us and said, “We have a downstream dependency to a bank and in their SLA says they can be down for three days. Can we just start workflow on every message if the system is down, and keep retrying for three days?” Well, technically, answer is yes, but we eventually found out that they had this huge pipeline of queues, with enormous complexity, and they had hard time maintaining all that. So **we ended up redesigning the whole pipeline as just one workflow,** and instead of just doing it piece by piece, and it helped them tremendously. It completely simplified the way their application was designed and they removed a lot of code. You can do a small proof of concept piecemeal, but if you adopt Temporal end-to-end, you will get 10x benefits.

### Durability

Samar: Think about how important durability is for your application. Do you have some state which needs to live beyond a single request/response? If the answer is yes, then I think Temporal is an awesome technology to help you deal with that complexity. It's better than the traditional way of building those applications, using databases, queues, retry mechanisms, and durable timers for retrying for three days. With Temporal, instead of building a sub-system which deals with all this retrying behavior, you can literally **just specify your retry policy** when you schedule an activity. So think holistically about your system, think about the statefulness and how important that state is. People have accepted that building this class of application is inherently complex, but Temporal is an amazing technology which makes this much more developer-friendly.


mo-U3Kst7MY4Ok-unsplash

Our co-founders Maxim Fateev and Samar Abbas recently joined Gremlin's podcast Break Things on Purpose for a rare double appearance to talk Temporal. Listen to it here!

Maxim and Samar on Gremlin's Build Things Podcast

Developer Relations

swyx

Latest Release at Time of Writing: V1.12.1

> Note: please remember to keep your Temporal version up to date!

**TL;DR;** 

- Cadence support EOL date remains **September 29, 2021**
- Node.js final stages before beta
- UI update, nearing early alpha
- Visibility improvements now available
- We are giving two talks at Kubecon, have a booth and throwing an afterparty
- Temporal meetups now have a set pace, next week very special guest

Hey everyone, it's been a while and I hope things are well. There has been immense growth in our community since our last update which couldn't make us happier. Over the last few months we've stayed pretty quiet and have been heads down and focused on improving the product. This post will cover some highlights of that work.

## Cadence support EOL reminder

It has now been 6 months since, [we announced the official EOL date for Cadence support](https://docs.temporal.io/blog/cadence-eol-support). That date remains **September 29, 2021**, less than two weeks away. This date marks the 1 year anniversary of Temporal V1 release. We cannot overstate how much appreciation we have for the Cadence project and team. We believe this is the right move which will allow Temporal to focus on the future, instead of worrying about the past. 

If you are still running on Cadence and depend on our team or the support they provide in any way, we highly recommend migrating before the EOL date.

## Kubecon

Kubecon is coming and 2021 will be the first year where Temporal has an official presence. We are engaged with Kubecon in a few ways:

1. Our own [Dominik Tornow](https://www.linkedin.com/in/dtornow/) is giving a talk with our partners at [Chronosphere](https://chronosphere.io/) about Chronospheres Temporal usage in conjunction with Kubernetes. The talk focuses on the difficulties of dealing with stateful operations within Kubernetes operators, and how Temporal can supplement Kubernetes to provide a good stateful and stateless experience. 
2. Our awesome Developer Advocate [Tihomir Surdilovic](https://www.linkedin.com/in/tihomir-surdilovic-5641361/) is the owner and author of the [CNCF Serverless Workflow Spec](https://serverlessworkflow.io/) which he has talk about. The twist is that this is still a Temporal talk. Tihomir recently spent the time to write a [Temporal sample](https://github.com/temporalio/samples-java/tree/master/src/main/java/io/temporal/samples/dsl) which can run the minimum specification for Serverless Workflows. His talk will partially focus on why Temporal is the best foundation for almost any DSL solution.
3. We have a booth this year in the startup hall. Quite a few members of our team will be there to answer questions and engage with attendees. Please stop by for some free swag and Temporal related conversation. 
4. We are now CNCF Sponsors and welcome ideas and invitations to talk about how Temporal can work with other CNCF ecosystem projects.
5. Last but not least, we will be hosting an afterparty for Kubecon. Details below:

    **Location**: JW Marriott / The Mixing Room patio

    **Date**: Wednesday, October 13th

    **Time**: 5pm-7pm

    **Covid guidelines**: Masks only required inside. No masks required on outdoor patio.

    **To enter**: Attendees will walk through hotel Lobby towards LA LIVE Campus and enter from outside to access patio

We're excited to have in-person presence at such a great conference. We really hope to see you there!

## Visibility

In the last update, we gave an overview of the exciting changes to our Visibility API that were nearing completion. Immense progress was made on this front and the improvements are available in the latest releases. The primary goal of the redesign was to simplify the configuration and usage of search attributes, as opposed to expanding the functionality. Our Visibility API already provides an immense amount of value, it's just that very few used it due to the barrier of entry. We believe these updates change that storym making our Visibility functionality a joy to use. Here are some specific highlights:

* Improved query language for advanced visibility: ExecutionStatus is of string type, IS NULL support for missing search attributes.
* New system search attributes for advanced visibility: HistoryLength, StateTransisionCount, ExecutionDuration.
* Cleaner Elasticsearch index schema.
* Improved operational support: more metrics, better logs.

## UI update

*This section was guest written by our Front-End Architect, [Steve Kinney](https://www.linkedin.com/in/stevekinney/)*

**TL;DR;** “We went from nothing to almost a full something.”

In the beginning of July, we started work on a new user interface distinct to Temporal. The new UI addresses many of the complaints that customers have about the current interface from Cadance and also sets us up to deliver features unique to Temporal Cloud. We outlined the MVP set of features and initial architecture. We built many of the core features in under three months and are preparing to go into an internal alpha in the next few weeks. The UI and Cloud teams have collaborated and drafted out an initial roadmap for delivering self-serve admin features in Q4 and the UI is on track to deliver those features.

## TypeScript SDK

*This section was guest written by our Head of Developer Experience, Shawn Wang*

The TypeScript SDK is nearly ready, on track for a public beta launch in October/November. 

- We have pushed through a final API redesign based on community feedback. The latest API is inspired by React, with Workflows comparable to React Components.
- Major missing features have been implemented, including a new `patched` API for migrating Workflows (replacing the legacy versioning process), Child Workflows, Signals, and Continue As New.
- We are laying the groundwork for a major public push. Next.js, the most popular React framework, now has [a Temporal example](https://github.com/vercel/next.js/tree/canary/examples/with-temporal) (thanks to Loren), Val is on the blog and swyx is ramping up mentions on [podcasts](https://www.youtube.com/watch?v=H3h1WICelqs) and upcoming [conferences](https://reactnewyork.com/).
- Besides our list of existing Temporal users like Descript, we are already getting inbound interest on the SDK from new-to-Temporal startups from Vercel to Stately.

## Meetup

We've landed on a consistent pace for our Meetup events and the format and content improves with each session. In the last few sessions, we've been fortunate enough to have members of our community present alongside our team members. This makes us super happy as we see Meetups as a platform for the community more than for our company. If you have any interest in presenting something Temporal related at a future meetup, please reach out (ryland@temporal.io, swyx@temporal.io).

If you haven't already signed up, [here is the Luma link](http://temporal.io/meetup) that will keep you updated for each of these monthly events. Worry not, if you miss a session they can be watched later on our Youtube channel. Here is a [link to the playlist](https://www.youtube.com/watch?v=gWjtjRavqWU&list=PLl9kRkvFJrlSM-s-s-8WbgjN-R3Y4L7WG).

We have a session next Tuesday 09/28/2021 at 9am PST. For those of you who are eagerly waiting, here is a sneak peak of the agenda (subject to change):

1. Talk about Server Workflow Spec and Temporal. This session will be especially great for those who can't watch the Kubecon talk Tiho gives.
2. An update on the NodeJS SDK and the outline for our release plans.
3. A very special guest who has quite the interesting presentation about how they use Temporal. 

## Conclusion

In the last months we have seen huge growth on the community, team and product front. We are so grateful for the amazing community that makes this possible, the awesome team which has formed around the product and all the other people who support our journey. 

As always, feel free to reach out with questions, comments or critical feedback via email, Slack or our community forum.

Email: [ryland@temporal.io](mailto:ryland@temporal.io)

Slack: [temporalio.slack.com](http://temporalio.slack.com/)

Forum: [https://community.temporal.io/](https://community.temporal.io/)

Join the Cloud waitlist!: [temporal.io/subscribe](https://temporal.io/cloud)

Find an awesome job where you can use Temporal. Work for great companies like Wellhive and Descript: https://temporal.io/careers#external-jobs


shubham-dhage-8ZGZlksF8mo-unsplash

Temporal Transparency Update #12

Head of Product

Ryland Goldstein

ryland goldstein

shubham-dhage-C8YfO3YStVo-unsplash

Our CEO Maxim Fateev recently joined the OSS Startups Podcast with Robby Robson of Cowboy VC & Tim Chen of Essence VC to talk Temporal.

Maxim Fateev on the OSS Startups Podcast

Latest Release at Time of Writing: V1.10.5

**TL;DR;** 

- Cadence EOL date remains **September 29, 2021**
- Improvements and whimsy for the Temporal landing page
- Cloud adoption continues to grow
- Node.js is imminent. Represents a fundamentally better experience for Temporal
- Custom search attributes redesign

Hey community, things are going very well in the Temporal universe and we've seen huge jumps in open source and cloud adoption. There aren't any fundamentally new announcements in this update, but it should shed light on a lot of high impact efforts that are in flight.

## Cadence support EOL reminder

A few months ago, [we announced the official EOL date for Cadence support](/blog/cadence-eol-support). That date remains **September 29, 2021** which also marks the 1 year anniversary of Temporal V1 release. Cadence will always be an important part of the Temporal story, we just need to close that chapter out so we can continue fundamentally improving the way applications are built and run. 

If you are still running on Cadence and depend on our team or the support they provide in any way, we highly recommend migrating before the EOL date.

## [Temporal.io](http://temporal.io) updates

When we redesigned the landing page for [Temporal.io](http://temporal.io) the team and company was in a much different place than we are now. Back in September we were fully focused on the V1 release and then immediately switched focus to the funding announcement after the release was out. Hiring was by far the biggest challenge for the company at the time, so with our very limited time we decided to optimize the landing page for hiring. Most of our efforts since then have gone into improving the docs experience.

We are now reprioritizing the landing page to better speak to developers, and apply design polish and whimsy (Konami code anyone?). You can see some of our [design explorations on CodePen](https://codepen.io/collection/oEEVov). Finally, we are opening up the funnnel slightly to let more people [register interest in Temporal Cloud](https://docs.temporal.io/#cloud).

## CLI Improvements

`tctl` is a powerful tool that makes it easy to debug your Temporal applications and even the Temporal server itself. While `tctl` offers a lot of functionality, finding and using that functionality is often a challenge. We've know for a while that the CLI experience needed some love, but we couldn't rationalize prioritizing it throughout the last year. 

Fortunately, we are now in a better place to revisit the original decisions made in the CLI. [Ruslan](https://github.com/feedmeapples) recently decided he would spearhead a collaborative effort to make `tctl` more predictable, friendly and easier to reason about. A ton of scoping has already been done and the changes are currently in the proposal stage. We are now in a place where **we need your input!** These changes will affect everybody and we want to make sure the direction is representative of what our users want. There are a number of open RFCs [in our proposal repo](https://github.com/temporalio/proposals/pulls). Please don't be shy, every piece of feedback in valuable. Here are a few proposals which are worth highlighting. 

- [https://github.com/temporalio/proposals/pull/31](https://github.com/temporalio/proposals/pull/31) - consistent formatting for output (tables, json, cards)
- [https://github.com/temporalio/proposals/pull/26](https://github.com/temporalio/proposals/pull/26) - ability to pipe output to well known tools like `less` and `more`
- [https://github.com/temporalio/proposals/pull/35](https://github.com/temporalio/proposals/pull/35) - customizable timestamp formatting
- [https://github.com/temporalio/proposals/pull/27](https://github.com/temporalio/proposals/pull/27) - (merged) customizable defaults using config file

## Cloud

It's now been almost half a year since we onboarded our first design partner onto the cloud. Since then we've continued to onboard and iterate with our partners which has driven hundreds of improvements to the offering itself. To make things more concrete, here are some high level stats about the cloud:

- 10+ paying customers
- 5 production customers
- 5 deals in flight with publicly traded companies
- 500+ on the waitlist

We are also fortunate to have public cloud customers. Each of these companies is world class at what they do and drive us to make the best developer experience possible.

![cloud-customers-0064e6f3e7b8084229e79ffb86fdf24d](//images.ctfassets.net/0uuz8ydxyd9p/50czefCVoLfGjQrVpYmQ9x/9053f2c4a3902e09ba2113f740054f87/cloud-customers-0064e6f3e7b8084229e79ffb86fdf24d.png)

If you are interested in the cloud, we are still being selective with onboarding but are taking new partners. ~~Please fill out our newly updated cloud survey to help us understand if your use case would be a good fit.~~ [Survey closed—thanks for your feedback!]

## Node.js

Temporal's ultimate goal is to make application development fundamentally more accessible to developers of all experience levels and backgrounds. While the current product delivers on that goal well for Golang and Java developers, the majority of developers today are writing JavaScript and Python. It became clear to us that until we support the core languages that developers rely on, our ability to make application development more accessible was very limited. In other words, in order to make application development more accessible for the broader ecosystem we first need to make Temporal easier to adopt and learn about. Needless to say, accessibility was top of mind when we started thinking about Node.js support.

The Node.js SDK is unlike any other SDK as determinism is guaranteed at the workflow level using V8 isolates. It's also the first SDK built on top of our next generation Rust core. By guaranteeing determinism at the workflow level, we eliminate an entire class of problems that exist with our other SDKs. This significantly reduces the mental overhead of writing and debugging workflows.

The SDK is currently in alpha ([source](https://github.com/temporalio/sdk-typescript), [docs](https://learn.temporal.io/getting_started/typescript/hello_world_in_typescript/)) but even today it provides a completely unique and incredibly simple Temporal experience. We believe it represents the fundamental direction and level of accessibility Temporal is aiming for in the future. That being said, we are desperately in need of additional feedback, so if you have opinions on JavaScript and/or Temporal please share it with us (and join the #nodejs-sdk channel in Slack). Together we can drastically lower the barrier of entry for building connected applications.

## Visibility

Visibility is one of the most attractive features Temporal offers. It's rightfully attractive as it gives you unrivaled insights into running Temporal applications. Unfortunately the majority of our users are only using a small part of our visibility story. For those who are unaware, there are really two levels of visibility usage within Temporal. The first level works out of the box and automatically creates and updates visibility indices for workflows. This default experience only provides basic indices like `StartTime`, `ExeuctionStatus` etc. 

The second level is what we refer to as "advanced visibility" and enables the use of custom searchable attributes. Custom searchable attributes are user defined indices which can be attached when workflow is started and upserted directly from Temporal workflows. Custom searchable attributes drastically change the way you operate and debug your Temporal application. Unfortunately, the experience of using them has been far from ideal. 

Due to some very hard work by [Alex](https://github.com/alexshtin) on the engineering team, the custom search attributes experience is finally in a place we can be proud of. Instead of requiring users to manually edit Elasticsearch schemas and wade through dynamic config, all you have to do is run a few tctl commands:

```
tctl admin cluster add-search-attributes --name ProductId --type Keyword
tctl admin cluster get-search-attributes
tctl cluster get-search-attributes
```

Our hope is that this opens up the gates to an entire set of users who were previously interested in custom attributes, but couldn't stomach the process of using them. We plan to continue improving the visibility API, but we hope you agree that this is a great start.

## Open office hours

Our first open office hours of 2021 took place last month. We were super excited by the turnout and the amazing questions that users asked during the session. 

import { ResponsivePlayer } from '../src/components'

<ResponsivePlayer url='https://www.youtube.com/watch?v=Qz0LURUyf-w' />

**Our next office hours is Tuesday July 20th, 9am PT**. You can [sign up here to get an invite straight to your calendar](https://lu.ma/temporal).

We have some exciting presentations planned, including our first customer presentation! If there are specific topics or types of content you would like to see, please email `swyx@temporal.io` or [add to this forum thread](https://community.temporal.io/t/what-sessions-do-you-want-to-see-at-the-next-open-office-hours/243/12).

## Conclusion

We're incredibly happy with how the company, team and community have grown in the last year. Our goal is to continue ensuring that Temporal is the best answer for any distributed application, whether that is on the cloud or self-hosted. 

As always, feel free to reach out with questions, comments or critical feedback via email, Slack or our community forum.

Email: [ryland@temporal.io](mailto:ryland@temporal.io)

Slack: [temporalio.slack.com](http://temporalio.slack.com/)

Forum: [https://community.temporal.io/](https://community.temporal.io/)

nkululeko-jonas-fSeKcdzeZTE-unsplash

Things are going very well in the Temporal universe and we've seen huge jumps in open source and cloud adoption.

Temporal Transparency Update #11

A few weeks ago, around the end of March 2021, I set my focus on finally refreshing the information around Temporal's core concepts. Refreshing the core concepts documentation area had become a major priority in my mind. The area had largely gone untouched in the last couple of years, and user feedback suggested it consisted of far too many industry terms laden with heavy connotations and lacking in practical purpose and use case.

I figured the best place to start was with Workflows. As I tried to wrap my head around the best approach to introduce this concept to a new user, I fell back on a first principle to writing documentation, asking myself "what does it do?"

And this is when a wave of realization crashed over me. Like a surfer working their way past the break line, I had been bobbing about in between sets and another had just rolled in.

![71xf](//images.ctfassets.net/0uuz8ydxyd9p/4Y3rR0IDiFs8Nx11kJw8So/7bd28c5a0c59d4bd0202e79aa369375a/71xf.gif)

I had to take a breath, duck under the wall of white water and try and find my angle of approach, suddenly aware that if we didn't get past the break line soon, we may be stuck here for awhile.

The realization was that the answer to my question was already fairly well defined across any number of resources; A "workflow" is a sequence of processes, from initiation to completion, through which a piece of work passes. A Temporal Workflow is just that, hence the name. Within the context of Temporal, it is _how_ that happens, and _why_ it happens in the way that is does that is the secret sauce.

However, one must learn not only of Temporal's overall strategy, but also language specific implementation details simultaneously to get a taste of that secret sauce. At the time of writing this, we supported three implementation languages with a fourth on the way, with each language presenting its own unique way of defining and writing Workflows. My instinct was that something fundamental was off in the current approach to presenting this information.

I needed to think. So, I went for a run.

![giphy](//images.ctfassets.net/0uuz8ydxyd9p/3sha5EIUIIYZxWgSbVyC9v/bb1321d0164279e5af766517fbaa3d9b/giphy.gif)

With my legs pumping freshly oxygenated blood through my brain, my heart, and my veins, I set my mind to the task at hand. I tried to come at the problem from a new user's perspective. This is how I discovered a problem that I hadn't fully noticed before.

## User personas

The thing is, not all users are after the same information. Unlike many other technologies, we don't offer a single service or API endpoint. Temporal's boundaries are high and wide, and there are at least four unique user groups accessing the documentation.

1. **Infrastructure and operations engineers** looking to deploy and operate their own Temporal Server cluster. 
2. **Developers and software engineers** looking to start building applications. 
3. **Operators and SREs** looking to troubleshoot, debug, or generally monitor the system's health. 
4. **Business decision makers** evaluating the technology as a whole or considering the use of our cloud service (still in beta).

A single person may bridge multiple personas, but while embodying one, they won't want to be bogged down by information directed at the other. This particular aspect of our users convinced me that in order to meet our user's information needs, separating content based on our user's personas was the first step towards a fundamentally sound base to build off of.

Building documentation is an iterative process. And, up until that point, we had iterated our documentation such that this is what our landing page looked like:
![documentation-landing-page-march-2021-3f3b685033a971415421aff326847d74](//images.ctfassets.net/0uuz8ydxyd9p/3WLBuGFWggouGS0O6HSo7E/72c2919cee8595a4209294d49da84a9d/documentation-landing-page-march-2021-3f3b685033a971415421aff326847d74.png)

It's not entirely bad. But, as you can see, it doesn't provide a very clear path for a new user with a particular agenda. Someone might spend a good deal of effort wading through the full spectrum of information to potentially find what they are looking for. Thus, I decided, information architecture was a key variable in this equation.

## Now you're speaking my language

Now that I was convinced that changes were needed to our information architecture to address our user's personas more appropriately, I revisited the problem of our core concepts.

At this stage, I was able to make the following connections:

1. From an application development perspective, everything boils down to about six core concepts: 
  * Workflows 
  * Activities 
  * Workers 
  * Task Queues 
  * Signals 
  * Queries 
2. The developer/software engineer persona is the one who is primarily concerned with internalizing and fully understanding these concepts. 
3. One can only fully describe any of these concepts within the context of a particular language. 
4. A developer is almost certainly only going to be interested in learning in a single language.

Look at how a Workflow is defined across each language:

__Go__

```go
func SimpleWorkflow(ctx workflow.Context, someArg string) (string, error) {
  // Do something
  if err != nil {
    return "", err
  }
  return "success", nil
}
```

__Java__

```java
@WorkflowInterface
public interface SimpleWorkflow {

    @WorkflowMethod
    String simpleWorkflowMethod(String someArg);

}

public static class SimpleWorkflowImpl implements SimpleWorkflow {

   @Override
   public String simpleWorkflowMethod(String someArg) {
       // Do the stuff
       return something
   }
}
```

__PHP__

```php
#[WorkflowInterface]
interface SimpleWorkflowInterface {

    #[WorkflowMethod]
    #[ReturnType("string")]
    public function simpleWorkflowMethod(string $someArg);
}

class SimpleWorkflow implements SimpleWorkflowInterface
{

    public function __construct()
    {
        // Do stuff
    }

    public function simpleWorkflowMethod(string $someArg): \Generator
    {
        // Do stuff
        return something;
    }
}
```

While, one could tabulate between languages within a concept like above, as you get into the details, it becomes much more complicated. Additionally, not all features are available across each language, like [Sessions in Go](https://docs.temporal.io/go/how-to-create-a-worker-session-in-go) for example. And, because the implementation details and terminology can be so different from language to language, the obvious solution presented itself:

1. Each language should be visually separated from the others as its own topic area. 
2. We must present each of these core concepts within the context of each language specifically.

At the same time, we couldn't just abandon a high-level concepts section, because from a practical perspective we need to be able to offer a linkable landing page for each concept that is still independent of a specific language. Imagine trying to share Temporal with someone but only having a language specific link to provide. That would create a bit of a conundrum for those trying to share our page links.

Thus, our core concepts section would become a sort directory for each of the languages, directing developers to language specific concept pages.

## April 2021 changes

In April, the Temporal Product Team set about iterating towards these solutions. And we are excited with the results!

We redesigned our documentation landing page, [docs.temporal.io](https://docs.temporal.io/):

![documentation-landing-page-april-2021-247dc60fd9dd166231186373052f5221](//images.ctfassets.net/0uuz8ydxyd9p/5Hz3DPWaAV60S3lQnNyt6D/b3534bf89999c877a5c6790cc96a31f7/documentation-landing-page-april-2021-247dc60fd9dd166231186373052f5221.png)

We implemented a new SDK landing page, [docs.temporal.io/application-development](https://docs.temporal.io/application-development):

![documenation-app-dev-landing-page-april-2021-a3ef403470ec42a74619336ba4f42e33](//images.ctfassets.net/0uuz8ydxyd9p/7D25S4EvUspLjvb449q7Wl/d8f107db27e1e52d345b4f6eebb8aafb/documenation-app-dev-landing-page-april-2021-a3ef403470ec42a74619336ba4f42e33.png)

And we created a much more immersive experience for the user personas that we identified above:

- Go developers for example, [docs.temporal.io/go](https://docs.temporal.io/go):

![go-topic-area-94c5114f548ece5e00e22a45dc264d9b](//images.ctfassets.net/0uuz8ydxyd9p/uquKj98SZGENFjZewlGoV/e5b73f693869932533555154b2b09f46/go-topic-area-94c5114f548ece5e00e22a45dc264d9b.png)

- Infra engineers for example, [docs.temporal.io/server](https://docs.temporal.io/clusters):

![server-topic-area-275c0ba9022071fad8b84453eadcff8e](//images.ctfassets.net/0uuz8ydxyd9p/7lkPjYV1V55NCk5qzF6qDt/39767cf174053717331547eee26bd940/server-topic-area-275c0ba9022071fad8b84453eadcff8e.png)

And we started converting the core concepts into a directory for new users learning about Temporal:

![concepts-topic-area-4a817c7368ddecf6b5ffe0daba99925f](//images.ctfassets.net/0uuz8ydxyd9p/6RHyNleEc7VMlbu3OA8qvu/728ac7d3aa88ac02a59ab653eed40acc/concepts-topic-area-4a817c7368ddecf6b5ffe0daba99925f.png)

## Still day one

We have finally arrived at a place where there is an information architecture that fundamentally matches our user's personas. With this change in focus, our hope is that learning about the how and why of a particular concept becomes a much more rewarding and informative experience.

We know we have a LOT more work to do. We still lack full coverage of the concepts within each language, and the navigation and search experience can be made more intuitive. If you have ideas for how to improve them, [email us](mailto:docs@temporal.io)!


Refreshing the core concepts documentation area had become a major priority in my mind; that subset of docs had largely gone untouched in the last couple of years.

How we structured docs for user personas

![rlw56fpp0hjgd73xmq8h](//images.ctfassets.net/0uuz8ydxyd9p/5jPbcCyMag5DmVPWP4cEsI/634ceacaf2c90be4e21e5fe5fe78d847/rlw56fpp0hjgd73xmq8h.png)

I wanted to write this post ever since I saw David Fowler's tweet and the discussion it triggered.
![qt8irq2n337p5tsf7u4i](//images.ctfassets.net/0uuz8ydxyd9p/Yd9ZekQcJ9Q4FCeIFlhpu/6cb27eccc422b69839bc5768d459fbcb/qt8irq2n337p5tsf7u4i.png)

While it may have taken me quite some time to get around to answering it, 2020 wasn't an ordinary year by any measure.

I took my first steps into the Actor Model space more than a decade ago, when I started working on the Orleans project. I have been and will continue to be an enthusiast of actors. We published a few papers and I gave a number of talks about them. However, over time I gradually stopped using the term 'actors' even when explaining the properties and benefits of the Actor Model. This post is an attempt to explain why.

## Minefield of Conflations

When we open-sourced Orleans in January of 2015, I was surprised by the amount of debate it generated on the seemingly trivial topic. The debate was about whether or not Orleans was in fact a faithful implementation of the Actor Model and if grains were actors at all. Even though we published the tech report "[Orleans: Distributed Virtual Actors for Programmability and Scalability](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf)" nine months beforehand, it didn't seem to help much in those discussions.

It honestly took us more than a year to reach a point when Virtual Actors of Orleans were generally recognized as a legitimate interpretation of the Actor Model. It was not just another interpretation, but one that has its unique benefits, especially for high-scale applications like cloud services. I was shocked by the uphill battle it took to get there. Over time, I came to realize that a big reason for these debates is the fact that actors are inherently a minefield of conflations.

### Conflation #1: Distributed & Local

Whenever somebody would say "we are using actors", I learned to first ask, "is it for a distributed system or a single-process?" This question was necessary because many developers use actors as a concurrency mechanism, leveraging their "processes one message at a time" property. The Orleans team was coming from the C#/.NET background where there was already a strong support for concurrency and asynchrony, with features like Promises (Tasks) and await. So from our vantage point, there was little reason to use actors just for basic concurrency. However, in languages with less native support for concurrency (ie: Java), local actors (used within a single process) continue to be a useful mechanism for concurrency and asynchrony.

Both local and distributed actors adhere to the same three rules of the Actor Model definition — in response to a message, an actor can:

- Send messages to other actors
- Create actors
- Change its behavior for next message

However, distributed actors that live in a cluster of servers exist in a very different and more hostile environment; an environment of network messages, latencies, failure modes, and uncertainty about their state.

Despite the commonality of the core three rules that apply to both, I would argue there's very little else in common between local and distributed actors, especially when it comes to application design, tradeoffs, failure modes, and major aspects of how actors supporting runtimes are implemented.

### Conflation #2: Supervision Trees & Actors

Erlang was the first popular implementation of the Actor Model. Arguably, it was Erlang that is responsible for bringing actors into the mainstream and pioneering a number of important design choices. One of them was the idea of supervisors which are actors that are responsible for handling failures of other actors by recreating or restarting them, etc. Supervisors are usually used in hierarchies, known as supervision trees. These trees make it easy to reset a system of interconnected actors into a known state after a failure.

Akka, being a faithful adaptation of Erlang ideas to the JVM world, also implemented supervision trees as the key failure handling mechanism. When your goal is to build a resilient system that cleanly resets chunks of its state in response to a failure this makes a ton of sense. The subtlety of the fact that supervision trees of Erlang and Akka are just a way to implement actors was lost on many people. In their minds, supervisors and supervision trees became part of the Actor Model itself.

It took us a lot of effort to explain why we chose a different approach (Virtual Actors) to handling failures in Orleans. The Virtual Actor method of automatic lifecycle management by the runtime doesn't use supervisors and has its benefits, especially in many cloud scenarios. Keep in mind that the supervision tree approach may be superior in other cases, such as where you have a hierarchy of actors and need the ability to reset it. The point is that "actors" ≠ "supervision trees", and it's a tax having to explain it to new people coming from the Erlang/Akka background.

### Conflation #3: Message Passing & One-Way Messages

In the world of traditional actors, it is more common to send one-way messages without expecting an immediate response. More than that, the request-response (RPC) pattern is considered dangerous. Actor developers are told to use it with extra care because the calling actor will be blocked until a response is received.

In Orleans, we chose the opposite default, with asynchronous RPC being the primary way of invoking actors. Each such RPC call has a built-in timeout. That removes the need for developers to worry about their actor getting blocked forever. Actors can also be marked as reentrant, so that they aren't blocked from processing other calls at all while awaiting for a response.

Multiple asynchronous RPC calls can be made by an actor concurrently, e.g. to fan out to a number of other actors. The elegance of async/await makes merging of the resulting promises in a desired way and awaiting a joint Promise for the whole fan-out operation a trivial pattern.

One-way messages are also supported in Orleans, but they are not the primary pattern because in most cases developers want to know at least if a call successfully arrived to the caller or failed or timed out.

This is yet another fundamental area with a significant "explanation tax", incurred by the different choices other implementations of the Actor Model have made. I suspect that if we had not used the term "actor" in defining Orleans from the beginning, we wouldn't have spent so much effort explaining ourselves. The async/await pattern for efficiently managing asynchrony had been established in the .NET ecosystem a long time ago, and there's no expectation of supervision trees in that developer community.

### Conflation #4: State transition & Become

This is a smaller issue. However, I've had several conversations with people who insisted on a specific interpretation of the third rule of actors (that they can change their behavior of processing subsequent messages). They interpreted it to mean that there must be an explicit way to tell an actor to become something different. The claim was that if your actors don't support an explicit feature like that, they are not real actors.

In my opinion, this rule simply means that an actor can change its internal state, whether it's a formal state machine or a boolean/enum flag that will define how the actor should process another call. For example, Digital Twins are a mainstream pattern to model program entities that shadow physical IoT devices in order to reflect their state and to communicate with them. Actors are an obvious fit to implement Digital Twins.

When a Digital Twin actor receives a "turn device off" command, it is very natural for the actor to flip an internal state variable that reflects the "off" state. In that state, the actor ignores or rejects all commands except for a "turn device on", which flips that variable back to "on".

## Elephant in the Room: Actors & Models

![hgwatx1rr8zxfamkbnx0](//images.ctfassets.net/0uuz8ydxyd9p/1syhpybY0F0plVt6S2p6tD/1c25d586a5e38b23bd680fd7a209fef2/hgwatx1rr8zxfamkbnx0.png)

I've seen a number of presentations about actors that start with a meme slide showing some famous actor's photo. This is because the vast majority of developers have never heard of the Actor Model. This would be ok if the term "actor" carried some intuitive connotation for them. In my experience, it does not. Even worse, when presenting about Orleans, the minority of the audience that knew about actors often had the above listed conflations in mind. It was a no-win situation, for both parts of the audience. Every time I presented, I had to spend energy and time pushing that boulder up the hill. At some point I stopped doing that by avoiding talking about actors altogether.

Instead, I started describing grains in Orleans as objects that live somewhere within a cluster of servers. These objects have stable identities and are always available for an invocation. Objects are a widely understood concept. It is easy to build on the concept by adding that each such object has a unique identifier of your choosing, hides (encapsulates) its state, and is only accessible via asynchronous method calls defined as part of an interface. Object, interface, method call — these are no new concepts to grasp. You just have to imagine objects working transparently across machine boundaries in the combined memory and compute space of a cluster. This approach was more effective, catering to a wide range of audiences, from academic to experienced cloud developers to "I want to learn about building scalable applications" developers.

I'm happy that we chose to call Orleans actors "grains", not "actors". "Grain" is not a perfect term by any means, but at least it conveys the general idea of a rather small piece of an application. I would argue it is much better than "actor".

[The landing page of Orleans documentation](https://dotnet.github.io/orleans/docs/index.html) nowadays only mentions actors once — in reference to our [Orleans: Distributed Virtual Actors for Programmability and Scalability](https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmability-and-scalability/) 2014 paper. This is a result of our conscious effort of reducing the cognitive load on people that come to the page to learn about Orleans.

[Reuben Bond](https://twitter.com/reubenbond) recently started describing grains as Cloud Native Objects. Again, not a perfect term. But I like it because it tries to convey the benefits of the model and where it is most applicable. [Roger Johansson](https://twitter.com/RogerAlsing) even suggested a CNOB acronym for it. 🙂

![cjtxg7phtl0vu8yfk6jh](//images.ctfassets.net/0uuz8ydxyd9p/5SFzXJbQ2jbqbVlkabF6mw/98b754d257d148b3229f2c20a11168bc/cjtxg7phtl0vu8yfk6jh.png)

## Conclusion

I am a big fan of the Actor Model as a simple and clean model of computations. In particular, it is an excellent fit for building distributed systems, whether it be on premises or in the cloud. I am forever grateful to Carl Hewitt, Peter Bishop, and Richard Steiger for their original insight, and to many subsequent followers that pushed those ideas forward. Implementations of the Actor Model and the ideas it pioneered power many high-scale and mission-critical systems today.

At the same time, I'm convinced now that the name of the term "actor" was a rather unfortunate choice. It took me years to gradually arrive to this realization. In my view, the very word "actor" continues to be a major barrier for adoption of the Actor Model ideas for the broader population of developers. I cannot formally prove it. This is just my speculation, of course. I listed several other contributing factors that in my opinion add to this confusion.

Remember that old saying about two hard things in computer science: naming things and cache invalidation? I believe it is part of the answer to the question, "Why aren't actor frameworks more popular?

In my opinion, the Cloudflare folks made a pragmatic choice to call their (for all practical purposes) virtual actors "durable objects." Once again not perfect, but much more developer friendly than "actors." I like Reuben's idea of calling grains Cloud Native Objects. It helps people quickly get a high-level intuitive understanding of what it is and decide if it's relevant to them.

If there's a better term, I'm open to your ideas. Just not "actors", please.

omar-flores-lQT bOWtysE-unsplash

While it may have taken me quite some time to get around to answering it, 2020 wasn't an ordinary year by any measure.

The Curse of the A-word


Perhaps you've heard that here at Temporal, we're working on new SDKs to support more languages. We are engineers on the SDK team and wanted to elaborate on our challenges and how we're meeting them.

Temporal faces some unique challenges with respect to our client side libraries, namely:

* The client side logic isn't just a thin wrapper over some HTTP/gRPC/etc calls. The SDK needs to handle the reconciliation of events and their related state changes, which may be generated at different times by different actors in the distributed system that is a Temporal deployment. More on this below.
* We want to support as many languages as possible, while avoiding duplicating the complex logic in each language. On top of that, we need to present an idiomatic-to-their-language interface to our users.
* We expect instances of the SDK (workflow and activity workers) to often be long lived, and hence the SDK must be extremely reliable.

In this post, we'll dive into the points mentioned above. We will also explain why we chose to write the [Core SDK](https://github.com/temporalio/sdk-core in Rust to help meet these goals. Note that some familiarity with Temporal's programming model will be helpful. [Read more here](https://docs.temporal.io/temporal). A Temporal SDK provides the APIs in your language of choice needed to author [Workflows](https://docs.temporal.io/workflows) and [Activities](https://docs.temporal.io/activities), as well as the behind-the-scenes logic required to drive them. They allow you to write durable, long lived business logic without worrying about burdensome retries or other temporary failure concerns. This is all done in a way that feels natural to your language of choice.

### What's complex about it?
From a 10,000 foot view, a Temporal worker follows this algorithm when running a workflow:

1. Long poll the server for workflow tasks (for example, Server says "I need you to run the user's workflow code").
2. Apply the event history contained in the task to a collection of state machines associated with the workflow.
3. Run the user's workflow code, appropriately providing values from history for the results of Activities, firing Timers and Signals, etc., until the workflow code eventually blocks on something that isn't in history yet, or exits the workflow function.
4. Reply to the server, possibly telling it about some new commands (the things we blocked on, for example: "I want to start this timer") that have been generated by the user's code. Otherwise, notify that the workflow is completed.
5. goto 1

As you might imagine, steps 2 and 3 are pretty complex, especially step 2. This complexity arises from the huge number of combinations of actions that need to be taken as a result of workflow history being fed into these state machines. In turn, they determine what happens in the user's workflow code, and what needs to be sent to the server.

The [state machine for timers](https://github.com/temporalio/sdk-core/blob/f0d7d99bf513a60ecff1f059cb062da4dd78a0ec/core/src/workflow/machines/timer_state_machine.rs), for example, encodes the logic that determines when we tell the server that it needs to track a new timer or when one should be cancelled. It also determines if a timer should be blocked or not in the user's workflow code. It's one of the simplest machines, and there are about 16 of them we need to implement.

What may not be immediately clear is there's nothing language-specific about these state machines. Conceptually, they translate actions taken in your workflow code to commands that must be sent to the Temporal server. In the other direction, they translate workflow history into new information exposed to your workflow. This "translation" is the same regardless of what language your workflow is written in. In fact, there's no reason why semantically identical workflows written in different languages running on different workers couldn't handle each other's histories - though you probably wouldn't want to do this.

Yet, as it stands, each of our existing SDKs re-implement this difficult logic. Clearly, we don't want to repeat this for each language. We need some kind of **Core SDK** that all other language SDKs can be built upon.

### Towards a shared core
It's clear we could substantially accelerate the development of new SDKs and increase the maintainability of existing ones by building a shared common core library used by the language-specific SDKs.

We knew any design would need to meet the following requirements:

* Clean integration with other languages
* Good ergonomics for the end user (avoid imposing new operational requirements)
* High performance
* Maintainable

Those requirements are pretty restrictive. To expand on the operational requirements: it's desirable from a packaging and performance perspective to be able to live in the same process as the language-specific SDK. For an end user that means they can simply deploy one binary which will run their worker, rather than needing to deploy the core SDK separately.

To implement the core and meet the requirements, we need a. We [need a hero](https://youtu.be/bWcASV2sey0?t=56) to rise to the challenge! Enter... Rust.

### Why Rust?
There's a lot of good reasons to pick Rust; some of which could fill up entire separate blog posts. The same reasons it's gaining in popularity so quickly these days apply to why we chose it. "Fearless concurrency", "performance and safety", a quality type system, etc. These reasons check the "high performance" and "maintainable" boxes.

When it comes to connecting multiple languages to a shared library, the traditional choice is often C, but Rust makes for a safer and more modern alternative. Having language SDKs directly link to the Rust core meets our end-user-ease goal and keeps overhead low.
There are other ways we could've tackled the problem (for example by running another process that communicates with the language-specific SDK over some kind of IPC), but those options probably fail our ease-of-use goal. If desirable, we can always run out-of-process later because we use Protobufs to represent data passed between the Core and language-specific SDKs. This technique also reduces the amount of duplicate code we would have to write for each new language.

Rust also very easily compiles to WASM, which unlocks some very interesting possibilities for us that we'll likely discuss in a future blog post.

There's one other reason that matters a lot to me personally: Rust is fun to write. It's not often (though certainly has been) said, but I think it's a huge part of the reason the language has grown in popularity so quickly. It presents a challenging but rewarding mental model that, once internalized, provides some deeply rewarding "a-ha!" moments while also bolstering you with that "if it compiles, it probably works" confidence.

### What's next?

In the future, we plan to support a larger selection of languages than we do currently, all based on top of the common core. We can expect Core to ensure a stable, well-tested basis for new languages SDKs where they all can benefit from the reliability and performance it provides.

Keep your eyes peeled for the first alpha release of our [Node.js SDK](https://github.com/temporalio/sdk-typescript/) which will be coming quite soon. It is built on top of the Rust core that we've been developing in tandem. We'll announce that release here on the blog as well as other communication channels.

Lastly, we plan to port our existing Go SDK to it as well. We're excited to grow the number of languages supported by Temporal, and bring you rock-solid reliability while doing it!


ricardo-gomez-angel-9AdeEdYB2yk-unsplash

In this post, we'll explain why we chose to write the Core SDK in Rust to help meet these goals. Note that some familiarity with Temporal's programming model will be helpful.

Why Rust powers Temporal’s new Core SDK

Spencer Judge

spencer judge

This is an annotated summary of a talk given at the Systems @ Scale 2021 conference hosted by Facebook.