Amazon DynamoDB MCP Server can really help in this process. You can start working through the schema using the MCP server or alternatively you could use it as a learning tool by making your own design first and then seeing whether the tool would have done the same design. The feedback you get is quite detailed, so it will really help to understand why the tool has taken certain decisions.

Setting Up Amazon DynamoDB MCP Server

There are several ways to run the MCP server using agentic tools. I used it with AWS Q CLI and the installation is straightforward. The configuration is simply added to the settings files of the AWS Q, and after that it is able to use the tool whenever you open a new AWS Q chat in the terminal:

Working Through the Requirements

The process starts by describing the type of application you are building and the business context. The tool will ask you questions, and based on your answer,s it will keep completing the requirements on the list that is shown in the terminal, as well as the complex requirements in the two files it is saving in your current folder: dynamodb_requirements.md and dynamodb_data_model.md

Collecting Access Patterns

After having an overview of the type of application we are building and the basic requirements, it is time to list the exact access patterns. The tool does this by asking questions about the volume and exact functionality that is required:

The tool will work through adding access patterns, and it will also suggest you ones that you might have forgotten, such as return, getting low stock alerts, or admin functions like user management in the context of the inventory management application.

Finalising the Data Model Design

After collecting all of the access patterns, the tool is ready to finalise the data model design. A detailed description is saved in the markdown document, and it will also give you a summary on the terminal. It will also explain the key design decisions it has taken:

One of the best things is that you are also able to ask it questions about the design. Below is an example of a question, and the answer is quite detailed — only parts of it are shown below

The example answer regarding simple table design also covered the problem a single-table design would create in this scenario, such as sales reports needing to use FilterExpression and what kind of requirements might have made the tool choose a single-table design, such as if brand changes were affecting product displays immediately. You can ask more in-depth questions about any of the aspects and you would get more detailed explanation with examples of situations where for example a single table design would be suitable.

After the initial database design, you are able to continue the discussion and make changes. After I had finalised the initial database modelling, I asked the tool to add modelling for an OCR (Optical Character Recognition) process with a ‘human in the loop’ implementation. We had a short back-and-forth conversation about the functionality that I wanted to achieve by having a system that will send label images to AWS Textract, after which a Lambda function will check against the database whether the brand and model name exists or whether it is something that has already been misread and corrected before. Otherwise we would need to find the closest match (using for example string similarity algorithm as it is not something that DynamoDB supports natively) which is then sent to the frontend for the human to check the correctness. Essentially this would be a system that will improve its correctness over time.

When the tool was aware of the schema and how the process would work, it was able to explain the steps that would be taken as part of the workflow:

Apart from helping with the database modelling, the tool can also help you implement it and it can also help you manage the database (in the context of CRUD actions and schema-level interactions), as long as you have your AWS credentials set up. I personally feel the database modelling is the greatest advantage that it offers as that is usually the most demanding aspect.

After I had finalised the database design, I did also take advantage of the tool’s awareness of the schema and made it create some of the queries for me using my preferred SDK. I was then able to simply copy and paste the queries directly into my code. In summary, I found the tool very useful and will definitely use it to get started with my next DynamoDB project.

It is worth noting, that the tool’s recommendations are advisory and a human review, load testing and validating assumptions remains crucial. It could be that the model will for example propose GSIs that looks optimal for access patterns but under high write loads could create hot partitions. LLM-driven explanations can also occasionally state implicit assumptions (e.g. expected query frequencies) and you need to verify that these actually match your real usage.

Kiro has two different ‘modes’: spec mode and vibe mode. The vibe mode is how most developers would understand it - direct implementation based on a prompt focusing on conversational style and without a formal structure. However, what makes Kiro unique is the spec mode. During a spec session, a formal workflow is followed using a structured approach. This makes it easier to plan and coordinate tasks, resulting in a paper trail for future reference and team collaboration. The whole flow of spec-driven development is something new and different, and I have really enjoyed the organised and logical way of programming, which is very different from the at times chaotic vibe coding where you are unsure of what is happening and how to fix things that have gone wrong.

I have been testing Kiro with a couple of existing projects, where the expectation for Kiro is to start by scanning the project and trying to understand the requirements and the structure of it. Kiro creates spec files that are like documentation for your own little project manager.

To illustrate, I'll use a simple serverless project that utilizes AWS services. Kiro is versatile and compatible with any cloud provider, but AWS was simply what I used for this particular project. The project had a functioning Lambda function code that interacts with Amazon Bedrock and some database tables, but there was no infrastructure as code apart from an empty CloudFormation folder. Generally, the project was unorganised and uncompleted and my first prompt requested Kiro to improve the repository structure and also add the infrastructure-as-code capability for the project. Based on this prompt, Kiro started to create documentation in the specs folder based on it’s standard design - requirements - tasks -flow. I will describe below the flow in more details in the context of this specific example.

Design

The purpose of the design.md document is to describe what Kiro is aiming to achieve with the steps that it is planning to take. In my specific example, the document contained the following details:

• Current state of the architecture (for example, agent.py located at the repository root)

• Target state of the architecture represented as a directory tree

• Infrastructure as code - listing how the files will be structured and where this like development environment parameters will be stored. The design document also listed all the AWS resourced that the IaC is meant to create

• Documentation updates that would be done on the README.md after the steps have been completed

• Data models that would be used in the DynamoDB tables

• It also describes thing like error handling, testing strategy and implementation considerations

Once Kiro has completed the design document it will ask you to verify whether you are happy with it before moving on to the next step. At this point, you are able to ask it to make modifications if want to make a change or if you want to exclude something (for example, testing) and tackle it as a separately later on.

Requirements

Once you have confirmed that you are happy with the design file, Kiro will move on to create the requirements.md file. In this document, it will summarise the requirements to user stories, which for my specific project were, for example, the following:

Each of these requirements will have an acceptance criteria listed in the form of when - then - shall -statements:

Tasks

After you have confirmed that you are happy with the requirements file, the fun part begins. Kiro is going to create a tasks.md file. This file will list all the different tasks that need to be taken in order to reach the previously described design that follow the requirements. In my example, the first task was to create new directory structure and move source files. This task included not only creating and moving the files, but also updating any relative import paths if needed. The task file also clearly describes which requirements this specific task is going to fulfil.

The nice things is that you are able to manually start the tasks directly from the file, and you will always have an easy overview of which tasks have already been completed and which are still waiting for completion. After the task have been completed, you can easily review the changes that have been made:

This allows you to easily follow execution step-by-step and request changes as you go if anything doesn’t look the way you wanted it to.

Further specs

After completing the tasks from my original prompt, I asked Kiro to complete a few additional features. The first request was to add an AWS API gateway in front of the Lambda function, and the second was to create a simple frontend application to provide a user-friendly interface. For both of these requests, Kiro created new spec directories with their own design.md, requirements.md and tasks.md files. This separation of concerns makes it easy to keep track of what is happening and to have clear documentation for the future:

Hooks

Hooks is a feature that I have until now tested only with very basic examples, but there are plenty of use cases where it would be very valuable. Hooks are essentially event listeners that, when triggered by specific IDE events or user actions, automatically launch a full agent execution session. Kiro creates a hook folder, which contains a file for each hook.

The hook I used in this project was to update hook the documentation. It listened to the Python source files, CloudFormation templates and requirements.txt changes to automatically update the README.md document. The prompt used for the agent execution session when triggered, is the following:

‘The source code or infrastructure has been modified. Please review the changes and update the README.md file to reflect any new functionality, API changes, infrastructure changes, dependencies, or usage instructions. Pay special attention to CloudFormation template changes and deployment instructions. Ensure the documentation stays current with the codebase.‘

The nice thing about this hook is that it doesn’t only update the documentation when it has completed tasks, but the hook will make updates also if you, as a developer, have manually made changes to any of the files. Having said that, the above doesn’t take into consideration the fact that the hook will consume agentic interactions and there will be price considerations in terms of how often you want your hooks to run, so this would need to be balanced.

Conclusions

My experience of Kiro has mainly focused on the main ‘design-requirements-tasks’ -flow, and I haven’t yet had a chance to test other features such as integration with MCP servers or steering files, and I have only done some initial exploration with the hooks. With such a new product there are, of course, sometimes been small issues and bugs, but generally, I have absolutely enjoyed working with the design-requirements-tasks flow. I wonder if it also depends on the personal working and documentation style whether someone would find this way of working compelling, but I find the organisation and structure really helpful even with a personal project and in a teamwork setting, it would bring added benefits for coordination. It makes it so much easier when you can clearly follow what AI is doing and make small adjustments as needed. And having a clear documentation for future reference is the second main advantage - imagine coming back to the project after a while and having clear feature spec directories that contain tasks lists to give you an easy overview of the status.

As the API is currently in beta and only some of the resources are available, I started by checking what is there that I could work with. The Oak Academy website has video lessons for different maths units, and each of the lessons has an exit quiz. The exit quizzes could be an interesting material to work with, as you could use them to create more similar quizzes that would then test the pupil, for example, at the end of the year across the whole year’s curriculum to see if there are some units or lessons that they are not confident with and to help them revise.

The exit quizzes contain usually six questions each, and most of the questions have an image attached to them in this type of format:

I started by fetching the exit quiz materials for year 1 and 2 and saved them in Amazon S3 as PDF files. Next, I would configure the Amazon Nova models to use these materials as starting point for creating similar content. The idea was to use one Nova model to extract information from the PDF and create content for a new quiz question based on it. Another Nova model would then be used to create an image based on the image description:

Describing the quizzes and creating content with Amazon Nova Lite

In order to use the quizzes that I now have saved as PDF files, I would need to transform them in a format the model can utilise. For this task, I used Amazon Nova Lite, which is a low-cost multimodal model that can process image, video and text input.

I wanted the model to extract some information from the PDF - the lesson name, the question and answer options. I also wanted it to determine what the correct answer to the question is, as well as create a detailed description of the image that is associated with the question. In addition, I wanted it to create a new, similar question-answer pair and a description for an image that could be associated to that new question. The model returned the data in this kind of format:

{
  "lesson_name": "Add and subtract 1 to and from a 2-digit number crossing the tens boundary",
  "original_question": "Which decade is missing on the number line? Tick 1 correct answer the fifties the sixties the forties",
  "original_answer_options": [
    "the fifties",
    "the sixties",
    "the forties"
  ],
  "original_correct_answer": "the sixties",
  "original_imageDescription": "The image shows a number line with numbers 49 and 50 marked. There is a gap between 50 and 60, indicating a missing decade. The options provided are 'the fifties', 'the sixties', and 'the forties'.",
  "new_question": "Which number is missing on the number line? Tick 1 correct answer 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60",
  "new_answer_options": [
    "45",
    "46",
    "47",
    "50"
  ],
  "new_correct_answer": "45",
  "new_imageDescription": "The image shows a number line with numbers 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, and 60 marked. There is a gap between 44 and 46, indicating a missing number. The options provided are '45', '46', '47', and '50'."
}

The model was good at extracting different questions from the document, determining correct answers, and describing the relevant images. Also, the newly created questions seemed to have similar style and were aimed at a similar age group, at least based on the limited tests that I was able to complete within a restricted timeframe. The prompt had to be modified a few times based on different issues I came across, such as the model getting confused by blank spaces in some of the questions. Further tests would most likely reveal more such issues that would help define the prompt further.

Creating a new image with Amazon Nova Canvas

From the process with Amazon Nova Lite I now had a new question - an answer to it and a description for a new image that could be used. The new image could now be created with Amazon Nova Canvas, which is an image generation model that can be used to create images from image/text. I first tried creating images based on purely the text prompt that I had from the previous step, combined with a general prompt that described what the purpose of the image is and what style should be followed. The images were of nice quality, but the difficulty was getting the style right. It seemed that even after trying several types of prompts, the model was unwilling to create the type of images that would be suitable for children’s maths exercises but rather wanted to create more realistic images. For example, if the prompt asked for a clear image of a certain number of apples next to each other that a child could easily count, the created image showed the apples in a more artistic and realistic display so that some apples were only partially visible. Even bigger issue was accuracy - if the prompt asked for an image with three apples, the produced images sometimes contained two or four apples instead and the results didn’t seem very reliable.

I then tried adding the original image as a conditioning image to the prompt - in this example to get the model to create a number line like in the first example:

I thought the conditioning image might help getting the style right and helping the model to understand exactly I need. Style-wise, this worked as expected and the created image was very close to the style of the original image. For example, when the prompt requested a number line and the conditioning image showed a number line, the created image had some kind of number line as well - whereas without the conditioning image the whole concept of the number line was often lost. However, the number line didn’t look otherwise right. I further experimented by changing the control strength. When the strength was closer to 1, I could get images that were very close to the original image (and didn’t follow the prompt requesting the changes to match the new question & answer pair at all). By reducing the strength closer to zero, the results became more interesting and didn’t follow the original image exactly but still displayed, for example, a number line as requested. But as it is clearly visible, the numbers themselves were completely mixed up, and these images wouldn’t work for their intended purpose:

In summary, the workflow of using the existing quizzes as a foundation of creating more similar question-answer pairs worked quite well, apart from the image creating inaccurate numerals. If it was possible to create accurate images, this workflow could be used in creating a large number of questions across the curriculum. As the questions were not simply copies of each other with different numbers, but the model rather had some creativity, the revision quizzes could remain interesting for the children. I will re-try this idea at some point when the models are capable of higher numeral accuracy in image creation.

Looking for a way to implement this, made me first look into the DynamoDB Geohash library. I wanted to understand whether this package would work with my existing schema where I was already planning to store all of the items in a single DynamoDB table.

With dynamodb-geo-npm

I started by creating a sample project where I would create a new table using the library to see how exactly it works. The table that was created had this schema:

The geohash attribute contains the full geohash, whereas the hashKey attribute contains the first 5 characters of the geohash to ensure the items are evenly distributed across partitions.

In addition, the library created a local secondary index and that is where the actual search based on the geohash will happen:

Creating queries using the library is very simple, you simply define the coordinates of your centre point and the radius in meters around it where you want your search locations to be located.

The limitations is that the library doesn’t support composite primary keys. So let’s say you have different type of items, such as schools and libraries. Now, if you wanted to search only for the libraries within certain location, you would ideally create some kind of composite key that allows you to separate these type of items as required by your access patterns. When you are not able to have that level of separation, you end up getting both types of items in the search results. So in order for this to work, you would need to have a separate DynamoDB table for schools and another one for libraries or alternatively apply filters in the application layer after retrieving the results.

Without dynamodb-geo-npm

Based on the limitations, I wanted to try how to implement this in a way that would work with the existing schema where all items are stored within one table following the principle of single-table design. This required a couple of steps that had been handled behind the scenes when using the library.

The first step was to use the ngeohash library to create a geohash for each item that is saved in the database. For that you need the coordinates of the location and the library will then create a geohash from these coordinates. This geohash will be a string containing letters and numbers. Previously when testing with dynamodb-geo, the geohashes were actually number-only hashes as the library internally converted the format from the alphanumeric hash. I then saved the geohash in the existing database and created a global secondary index that would allow me to search based on the item type (library or school etc) and the geohash:

It is important to have the geohash as the sort key, so that you are able to later use the ‘begins with’ type queries.

In order to query from this tale, the first step is to determine a ‘bounding box’ around the centre point. This means that you have a geopoint and then determine a radius such as 2000m around it. Using the geolib library it was easy to get a bounding box coordinates around a centre point:

As the bounding box is a ‘box’ it will end up having some coordinates that are actually outside of the defined radius as described above. These locations can be filtered out later if only the locations within a certain distance are required.

The next step was to determine what the geohashes based on these coordinates are. This can be done using the ngeohash library. The library will simply return you an array of geohashes based on the defined coordinates and the length of geohash that you want. The length of geohash determines its precision and the size of area it covers. So that is something that you will have to decide based on your exact scenario. For example for the 2000m radius a 6 character geohash might be the best option as using only 5 characters would give you too wide cells whereas using 7 characters would be too fine-grained and end up requiring more queries than necessary. The concept might seem confusing at first but testing with a few different variations will make it easier to see what the options are and what makes sense. Based on the coordinates and length of desired geohash, the ngeohash library will return you an array of geohash cells:

These cells are now again wider than the actual bounding box coordinates were, just based on the way these cells are calculated. So you will end up having again more results that don’t belong inside the exact search radius. So again, these need to be filtered out in the end if you require results that are only within the defined radius.

Now when you have a list of hashes, you would need to make a query for each of these hashes. These can be done using the GSI. As the GSI sort key was created using the full geohash, now when searching for items we need to use the ‘begins with’ option as we are searching items within a certain radius rather than the exact geolocation as geohashes have a hierarchical property where prefixes represent larger areas:

It is worth mentioning that although this system will require several database queries, the same would apply if using the dynamodb-geo library as it would also do several database queries behind the scenes (according to the documentation typically 8 queries are executed per radius search).

As a result, you would have a list of library locations that are located within those geohashes. Each item would have it’s coordinates and based on those coordinates it is possible to do further organising such as finding out which location is closest to the exact coordinates of the search point and calculate what the distance is by using the haversine formula. And as mentioned before, do to the way the search was done, there would be some items in the results that don’t fit in within the exact radius, so those would need to be filtered out if that’s your requirement.

Conclusion

Creating a geohash based search without using the dynamodb-geo library is not a very complicated thing to do and the steps are quite straightforward. But of course deciding which option to go with is a more complicated decision and there are many trade-offs to consider. The library would be optimised and if you are working with your own schema, the latency and cost will depend on the way you have designed your database and access patterns. The main aspects impacting the performance of your design would the geohash precision (length), number of queries that are needed and the post-processing overhead depending how much you need to further process the items after the database queries have been completed. So as usual with DynamoDB, this would require careful considerations and monitoring.

I have wanted to learn more about GitOps and found the perfect opportunity to do that with a hands-on approach at the GitOps for Terraform MiniCamp. In this post, I'll walk you through the technical requirements of the project—what tools, resources, and setups are needed to implement the fully functional GitOps pipeline to deploy AWS infrastructure using Terraform—before delving into the implementation details in future posts.

In this project, I will create a GitHub Actions CI/CD pipeline that will run through certain jobs and deploy the AWS infrastructure. I have summarized the steps of the pipeline and the way it will be triggered and connected to other services in the diagram below. This is, of course, my current understanding based on the requirements, and it might be that when I start the actual implementation, I will notice some parts are not accurate and this diagram will be modified.

Backend resources

The first step in the project will be to create the backend infrastructure using Cloudformation. This will be done separately, not as part of the Terraform infrastructure, as it will be used to store the Terraform state file itself. There will be a DynamoDB table that will have only one item—LockID. This item is simply used to make sure that several workflows can’t make changes to the Terraform infrastructure simultaneously, which would lead to conflicts. When our workflow is modifying the infrastructure, there will be a LockID in the DynamoDB table, which would prevent any other workflows from making changes until we have finished. The actual Terraform state file will be stored in an S3 bucket.

Additionally, the OIDC role that will be needed in the later stages to access AWS will be created with Cloudformation. All of these are resources that will remain unchanged during later work and will lay the foundation for this project.

GitHub Actions workflow

The first step I want to automate after making modifications to my code is a step that should run even before being able to commit any code changes. This step is called a pre-commit hook. The hook will run terraform fmt, which has the main purpose of ensuring that the Terraform code is formatted properly following style guidelines.

The next step will be to push my code to a feature branch in my GitHub repository. The main branch will be protected and the correct process will be to push the code always to a feature branch first, which will then trigger the GitHub actions workflow. The workflow starts running through several different steps. These will include

• TSLint (analyze Terraform code for best practices, syntax issues, and possible errors)

• Terraform fmt (same as the pre-commit hook, just to make sure)

• Terraform validate (making sure that the Terraform configuration is valid according to the Terraform syntax)

• Terraform plan

• Infracost

In the last step, a tool called Infracost is used to estimate the cost of the Terraform infrastructure. At this point, also a tool called Open Policy Agent (OPA) will also be integrated into the workflow. The idea is to enforce a policy that fails if the estimated cost of infrastructure exceeds a certain threshold. This is a way of automatically enforcing cost controls in the Terraform workflow.

Dispatch step

We want to make sure that there is some kind of human interaction before the actual resources are deployed on AWS and there are several ways of doing that. The easiest way will be to add a dispatch step, which will ensure that the workflow requires manual approval before moving on to the deployment.

When the approval for deployment has been given, the workflow will need a way of authenticating itself to access your AWS account to provision resources. The most secure way of doing this is using OIDC—OpenID connect. OIDC will utilize the IAM role that has already been created using Cloudformation. GitHub actions will assume that IAM role, which will give it short-lived credentials to access AWS. These credentials will be destroyed afterwards and cannot be re-used.

Deployment

Once the workflow has access to AWS, it will first make sure that there is no LockID currently in the DynamoDB table. It will then move on to provision the resources that have been defined in the Terraform code. In this project, it would be a simple EC2 instance. In addition to that, some resources can be provisioned to monitor our infrastructure. AWS Config and EventBridge can be set up to run a scheduled drift detection, which would notify us if our Terraform state file doesn’t match the deployed infrastructure. Other services such as Lambda and CloudWatch events could be set up to run scheduled port accessibility checks for the EC2 instance.

After the required changes to the infrastructure have been made, the new state will be stored in the state file in the S3 bucket and the DynamoDB state lock will be released.

After successful deployment, it is time to merge the feature branch code to the main branch. In the approach that we are taking, the infrastructure is the ‘source of truth’, meaning that the deployment will happen first and until the merge, the main branch will actually have code that doesn’t anymore match the current infrastructure. The alternative approach would be to merge the code to the main branch first, which would keep the main branch strictly as the ‘source of truth’. This would come with challenges, such as having to roll back code if there turns out to be an issue with the deployment and for this reason, validating everything before merging, is often the preferred workflow.

Next steps

There are several ‘bonus challenges’ that I would like to add to the project as soon as I have implemented the above-described parts. One of them would be to deploy to multiple environments (stage, prod). It would also be very useful to automatically open an issue for the repository if the schedule check notices that the infrastructure has drifted. Another nice addition would be to configure the GitHub actions workflow to ignore non-terraform changes, meaning the workflow would only start running if there have been changes to the Terraform code. There might be other extension ideas once I start working through the implementation.

As I was working in a team with two amazing AI developers, this seemed like a very exciting opportunity to get my first experience building a solution that utilizes AI. I'm not going into a detailed explanation of the AI solution in this article as it is not my area of speciality, instead of I am focusing on my experience as a a fullstack developer who took part in designing, developing and testing the app as well as integrating some of the AI endpoints to our frontend application. We had additionally also a frontend developer in the team, so we had a balanced team with different skills and interests and that was a great starting point.

Designing the app

We started our project by brainstorming what kind of functionalities the app should have for the user and what kind of building blocks would be required. Do we need

• a database where to save data --> yes, we want to save user data and data of completed simulations

• a backend --> yes, to interact with the database

• authentication --> yes, an easy implementation with Firebase

We would need a React app, NodeJS backend, MongoDB database and the AI element, which our application would access via Fast API. It was decided that we would connect the React frontend directly to the Fast API as for the purposes of this hackathon it would be easier for us to integrate the endpoints directly into the React app. The architecture of our application is described in the below diagram:

Implementation

As we had team members with various specialities in our team, we started working on all the different components simultaneously. We started creating the frontend components based on Figma design, building the NodeJS backend that interacts with MongoDB and verifies authentication as well as building the AI backend.

Each system was tested separately - for example by testing the NodeJS backend with Postman, we made sure that the database interactions worked as expected and Firebase authentication tokens were validated. Similarly, the AI API was tested using similar tools to make sure that it is accepting the API requests and returning the kind of information we expect.

When everything was working, it was just a matter of integrating everything. Well, they should fit together like pieces of a puzzle as we already planned everything, right? It's never that straightforward, but always a lot of fun and a great learning experience.

Integrating frontend, backend and AI backend

Integrating the frontend to the NodeJS backend was straightly straightforward as our application is dealing with simple CRUD actions. Integrating the frontend application to the AI API was something I didn't have previous experience with. In all fairness, it was not that different from making 'traditional' API calls, especially as our AI API was built with a simple architecture for the purposes of this hackathon.

We didn't have a way of managing context in the LLM application, but instead, we sent a lot of data (all previous conversation history) with every single API call. This needed careful management at the frontend for the data that should always be passed to the backend. This would of course be an inefficient way of running a production application, but worked well for our prototype.

Lessons learnt

For me, the main benefit of this project was getting a better understanding of how AI APIs are built. I got a lot of insight into the challenges of collecting appropriate data and had a chance to see how adjusting the prompt can fully change the responses we get from the AI.

I also learnt how beneficial it is to do early integrations during development, even if it is just a part of the applications or one API endpoint. For example, we hadn't considered how there would be a delay in the responses we get from the AI, which meant we didn't have the time to change the UI to make it more obvious to the user that they have to wait. Had we tested it earlier, we could have still applied some UI changes that could have improved the user experience by making the waiting period more transparent and less frustrating.

Overall, it was a great hackathon where I learned a lot and it was very rewarding to see how we managed to build an application where the user can interact with AI within such a short timeline.

Further technical details (including the AI implementation) can be found in the GitHub repository.

Before you begin

This tutorial utilizes several tools and requires familiarity with basic tools like Git, as well as a basic understanding of frontend development, and knowledge about AWS services and infrastructure as code. The main purpose of this tutorial is to build a small project where all of these technologies are connected. If any of these tools is new to you and feels challenging, it is a great chance to go and read some documentation and then gain some practical experience by following this tutorial.

This project has a simple folder structure, where the main component is the Terraform code that is used to create the AWS resources needed to run the serverless backend. We will also have folders to store the Lambda code and frontend code in the same repository:

React

The React project is going to be simple to limit the length of this tutorial, but you could of course easily extend the frontend application. We will use a React UI library called Mantine to create the contact form. Mantine has several pre-built components and styles and we can create a simple form easily.

The easiest way to get started with Mantine is by using one of their ready Vite templates. You can read more about it here and find the Vite template here. Vite is a front-end tool that can be used to create React apps. The template includes all React and Mantine UI dependencies, so by using this template you don't need to do any further installations and it is a quick way to get started.

After you have used the template to create a new repository and created a local project from it, you can install dependencies (npm install) and run the project (npm run dev). The home page contains a mock page and to keep this project really simple, we will add our contact form directly on the homepage.

Mantine UI has a ready form that we can use, we only need to add some logic to make it work. To use Mantine form, you first need to run the following installation npm install @mantine/form and after that, you are ready to replace the code in your Home.page.tsx -file with the following code:

import { useForm } from '@mantine/form';
import {
  TextInput,
  Textarea,
  SimpleGrid,
  Group,
  Title,
  Button
} from '@mantine/core';

export function HomePage() {
  const form = useForm({
    initialValues: {
      name: '',
      email: '',
      subject: '',
      message: '',
    },
    validate: {
      name: (value) => value.trim().length < 2,
      email: (value) => !/^\S+@\S+$/.test(value),
      subject: (value) => value.trim().length === 0,
    },
  });

  interface FormValues {
    name: string;
    email: string;
    //subject: string;
    message: string;
  }

  const handleSubmit = async (values: FormValues) => {
    try {
      // Replace with your actual API endpoint URL
      const apiUrl = 'https://12345.execute-api.eu-west-2.amazonaws.com/test'; 

      const response = await fetch(apiUrl, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify(values),
      });

      if (response.ok) {
        // Request successful, do something here
        console.log('Email sent successfully!');
      } else {
        // Request failed, handle errors here
        console.error('Error sending email.');
      }
    } catch (error) {
      console.error('An error occurred:', error);
    }
  };

  return (
    <form onSubmit={form.onSubmit(handleSubmit)}>
      <Title
        order={2}
        size="h1"
        style={{ fontFamily: 'Greycliff CF, var(--mantine-font-family)' }}
        fw={900}
        ta="center"
      >
        Get in touch
      </Title>

      <SimpleGrid cols={{ base: 1, sm: 2 }} mt="xl">
        <TextInput
          label="Name"
          placeholder="Your name"
          name="name"
          variant="filled"
          {...form.getInputProps('name')}
        />
        <TextInput
          label="Email"
          placeholder="Your email"
          name="email"
          variant="filled"
          {...form.getInputProps('email')}
        />
      </SimpleGrid>

      <TextInput
        label="Subject"
        placeholder="Subject"
        mt="md"
        name="subject"
        variant="filled"
        {...form.getInputProps('subject')}
      />
      <Textarea
        mt="md"
        label="Message"
        placeholder="Your message"
        maxRows={10}
        minRows={5}
        autosize
        name="message"
        variant="filled"
        {...form.getInputProps('message')}
      />

      <Group justify="center" mt="xl">
        <Button type="submit" size="md">Send message</Button>
      </Group>
    </form>
  );
}

Your project now has a simple contact form that will validate the form fields:

Submitting the form will try to send the data to the apiUrl defined above, which won't work yet as we haven't implemented the serverless backend. While this tutorial walks you through setting up the front-end app locally, we'll design it to seamlessly connect with the serverless backend once it's deployed.

AWS

The creation of AWS resources requires you to have an AWS account. You also need to create credentials for your AWS account to be used with Terraform in the next section. Additionally, there are a couple of steps we are going to be doing directly on the AWS console.

As the goal of this project is to create a contact form that will send the contents to an email account, we will need an email address for this and this email account needs to be verified by AWS. The easiest way to do this is in the Amazon SES (Simple Email Service) console. Simply click 'verified identities' - 'create identity' and follow the instructions. The identity status of your email must be 'verified' for this project to work:

Another setup step in the console is creating an S3 bucket that is used for storing the Lambda code. This could be done using AWS CLI or by creating a script, but the easiest way for now is to do this directly in the console. You can create a bucket manually and make a note of the name of the bucket, as we are going to need it for our Terraform code. Now, if you followed the instructions in the beginning, you have an aws-lambda folder where you can create a file called index.mjs. Below is the Lambda code you can add to the file:

import { SESClient, SendEmailCommand } from "@aws-sdk/client-ses";
var ses = new SESClient({ region: "eu-west-2" }); // Change here your region

export const handler = async (event) => {
  const eventData = JSON.parse(event.body);

  const email = eventData.email;
  const name = eventData.name;
  const message = eventData.message;


  const emailBody = `Hello,\n\nYou have received a new message via the 
    contact form on your website. Below are the details of the message:
    \n\n**Sender Information:**\n- Name: ${name}\n- Email: ${email}\n\n
    **Message:**\n${message}\n\nPlease respond to this message at your 
    earliest convenience.`


  const command = new SendEmailCommand({
    Destination: {
     //Change here your destination email address
      ToAddresses: ['email@example.com'],
    },
    Message: {
      Body: {
        Text: { Data: emailBody },
      },

      Subject: { Data: "New Message from Contact Form" },
    },
    Source: 'email@example.com', //Add here your source email address
  });


  try {
    let response = await ses.send(command);

    response = {
      statusCode: 200,
      headers: {
        //Change here the URL where your frontend app is running:
        "Access-Control-Allow-Origin": "http://localhost:5173", 
        "Access-Control-Allow-Headers": "Content-Type",
        "Access-Control-Allow-Methods": "POST, OPTIONS"
      },
    };

    return response;


  } catch (error) {
    console.error('Error:', error);


    return {
      statusCode: 500,
      body: JSON.stringify({ message: 'Internal server error' }),
    };
  }
};

As you can see, there are a couple of things you need to modify in this code. First of all, change the region to your preferred AWS region. You also need to change the email address to the one you have added and verified through the AWS console (you can use the same email address as 'to' and 'from' email to keep things simple). For the CORS settings, you also need to specify the URL address where your front-end application will be running. This setting is crucial to prevent unauthorized access to your API, as it ensures that only authorized domains can send requests to your backend

To store the code in the S3 bucket, you first need to manually zip the file as having the file in a zipped format is a requirement for Lambda deployment. Name the zipped folder as lambda.zip as that's how it will be referred to later in the code. Now upload this zipped file to the S3 bucket you have created previously. Your Lambda code will now be ready in the AWS cloud to be used by your Terraform code when we start creating the resources.

Terraform

The infrastructure of our project will be provisioned using Terraform. This tutorial assumes you have created an account for Terraform Cloud, installed Terraform CLI and created an organization and workspace on your Terraform Cloud account. Furthermore, as we are creating AWS resources with Terraform, we need access to the AWS account to be able to do this. Terraform can be used for local or remote execution and the local execution would require local configuration. However, in this tutorial, we are using Terraform with remote execution, which means you need to add your AWS credentials as environment variables for the Terraform Cloud workspace:

At this point, you should have all the setup ready to be able to start adding the code for the resources and start running Terraform commands from your account.

We will now start adding code to describe the AWS resources. We start with Lambda, as we in the previous steps uploaded the actual code for the function itself. In the modules/serverless-backend-aws folder, create a new file called resource-lambda.tf:

resource "aws_lambda_function" "serverless-contact-form-lambda" {
  function_name = "ServerlessContactForm"

  # change the name of the S3 bucket to the one you have 
  # created through the console
  s3_bucket = "serverless-contact-form-lambda"
  s3_key    = "lambda.zip"

  handler = "index.handler"
  runtime = "nodejs18.x"

  role = "${aws_iam_role.lambda_exec.arn}"
}

resource "aws_iam_role" "lambda_exec" {
  name = "serverless_contact_form"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
# inline policy in order to access SES
 inline_policy {
    name = "SESPermissionsPolicy"
    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail",
        "ses:SendRawEmail"
      ],
      "Resource": "*"
    }
  ]
}
EOF
  }
}


resource "aws_lambda_permission" "apigw" {
  statement_id  = "AllowAPIGatewayInvoke"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.serverless-contact-form-lambda.function_name}"
  principal     = "apigateway.amazonaws.com"

  source_arn = "${aws_api_gateway_rest_api.serverless-contact-form-api.execution_arn}/*/*" 
}

This code creates for us three resources:

• the Lambda function itself. As you can see, the code that will be populated within the function is taken from an S3 bucket. Change here the name of the S3 bucket you have created previously

• Lambda execution role to permit Lambda to send emails via SES

• Permission for API Gateway to invoke the Lambda function

Next you can create another file in the same folder: resource-api-gateway.tf:

resource "aws_api_gateway_rest_api" "serverless-contact-form-api" {
  name = "serverless-contact-form-api"
  description = "API Gateway for the serverless contact form"
}


resource "aws_api_gateway_integration" "lambda" {
  rest_api_id = "${aws_api_gateway_rest_api.serverless-contact-form-api.id}"
  resource_id = "${aws_api_gateway_method.proxy_root.resource_id}" 
  http_method = "${aws_api_gateway_method.proxy_root.http_method}" 

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = "${aws_lambda_function.serverless-contact-form-lambda.invoke_arn}"
}

resource "aws_api_gateway_method" "proxy_root" {
  rest_api_id   = "${aws_api_gateway_rest_api.serverless-contact-form-api.id}"
  resource_id   = "${aws_api_gateway_rest_api.serverless-contact-form-api.root_resource_id}"
  http_method   = "POST"
  authorization = "NONE"
}


resource "aws_api_gateway_deployment" "test" {
  depends_on = [
    "aws_api_gateway_integration.lambda",
  ]

  rest_api_id = "${aws_api_gateway_rest_api.serverless-contact-form-api.id}"
  stage_name  = "test"
}

This creates for us four resources:

• the API gateway called 'serverless-contact-form-api'

• an integration with the Lambda function for the POST method

• a POST method

• API Gateway deployment stage called 'test'

Another file that we need in this folder is called outputs.tf. In this file, we are going to define outputs that we want to expose outside of this module after the infrastructure has been provisioned. In this case, there will be an output for the API gateway and API Gateway URL. We need to refer to the APi Gateway resource in main.tf and that is why we need to expose it as an output. However, the API Gateway URL is not referred to in the main.tf file, instead, we imply want to print it in the console as we are going to need it for the API call in the frontend application.

output "api_gateway_contact_form" {
  value = aws_api_gateway_rest_api.serverless-contact-form-api
}
output "api_gateway_url" {
  value = aws_api_gateway_deployment.test.invoke_url
}

Now we have created the files that define what resources we want to create at AWS. We still need to add some general configuration to explain to Terraform what exactly we want it to do. This is done in main.tf file:

terraform {
  cloud {
    organization = "MyOrganization" #Add here your organization
    workspaces {
      name = "contact-form" #Add here your workspace name
    }
  }
}

provider "aws" {
  region = "eu-west-2" #Add here your region
}

module "serverless-backend-aws" {
  source = "./modules/serverless-backend-aws"
}

module "cors" {
  source = "squidfunk/api-gateway-enable-cors/aws"
  version = "0.3.3"

  api_id          = module.serverless-backend-aws.api_gateway_contact_form.id
  api_resource_id = module.serverless-backend-aws.api_gateway_contact_form.root_resource_id
  allow_headers = ["Content-Type"]
  allow_methods = ["OPTIONS", "POST"]
    #Add here the URL where your frontend application is running:
  allow_origin = "http://localhost:5173" 
}

The first thing this file tells Terraform is the details of the Terraform Cloud workspace we want to use for this project. For it to access your workspace in the cloud, you need to log in from the command line by using Terraform login.

The provider is the plugin that allows Terraform to interact with the API of the specific service provider. As we are creating services at AWS, we will be using the AWS provider for this.

Lastly, we need to list all of the modules we want Terraform to create for us. In this tutorial, we are creating a module for the serverless backend, additionally, we could add here a module for the frontend as well if we were deploying it with this same infrastructure.

We also create a separate module for CORS settings, which leverages a community-provided module that simplifies the CORS configuration for API Gateway. We need to refer here to the API Gateway resource we created and this is why we in the previous steps added that to to the outputs.tf in the serverless-backend-aws module.

The below diagram summarizes the different parts of the Terraform code:

The last file needed at the project root level is outputs.tf, just like the one we had in the module folder. We want to refer here to the one value we want to print in the console, namely the URL of the API Gateway. Once the backend has been deployed, we want to add this URL to the front-end application

output "api_gateway_url" {
    description = "Bucket name for our static website hosting"
    value = module.serverless-backend-aws.api_gateway_url
}

Finalizing the Project

It is now time to run Terraform plan and Terraform deploy to create the serverless backend. You can monitor in the terminal whether the creation of all resources is successful and you also see your resources listed in the Terraform cloud console. If all goes well, you will see your API Gateway URL printed in the terminal and can add that to your front-end application. Now submitting the form should be successful and the form data should arrive to your email:

If you end up getting an error, it is probably a small mistake you have made somewhere along the way and a great opportunity to dig in and learn more about the services. It could be something as simple as not adding the correct origin URL to CORS settings - going to the AWS console and testing the Lambda function and API Gateway in isolation might help you find out where the issue is between the frontend, API Gateway, Lambda and SES.

Next Steps

Now that you have the simple contact form working, you could do several things to extend and improve this project. As already touched upon previously, if frontend development is your thing, you could build the React application into a real application for example by leveraging Mantine UI's AppShell and other components. The form itself would also need some improvements - you could add error handling and improve the user experience for example with a notification to show when the form has been submitted.

You could also figure out how to deploy the front end on S3 and CloudFront and connect it to your custom domain. This could be done by adding an additional front-end module alongside the serverless-backend module. See my repository for some tips and example code for this.

The Lambda code could also be improved, it doesn't for example have any error handling at the moment. There could also be a more automated way of uploading the code rather than zipping the code manually and uploading it to the S3 bucket.

After making some improvements to the project, you could then move on to automating the deployment for example by using GitHub Actions. You would define the deployment steps in a workflow file and configure triggers for actions (for example push events could trigger a new deployment).

I hope this tutorial has helped you build your first serverless contact form. My code is available in this repository.

As Terraform is a cloud-agnostic infrastructure as code (IaC) tool, it can interact and manage resources in practically any cloud provider. The project created during this boot camp included AWS resources, so we of course used the AWS provider. We also experimented with the ‘Random’ provider, which can be used to create for example random strings. It does this by using entirely Terraform logic and without interacting with any other services.

However, to further experience how Terraform can indeed be used for almost anything, we went ahead to create a provider of our own. Our custom provider is used to create resources on the ‘TerraTowns Cloud’ platform, which has been created for the purposes of this boot camp. Participants can create ‘homes’ in the Terratown by using Terraform to create these resources.

To summarize, we are using in total three providers:

Creating the Custom Provider

The development process of the custom provider can be summarized in five steps. We needed to have a mock server that could be used locally for testing, a bunch of bash scripts and Go code for creating the provider. These steps are visualized below and explained in further detail in the following paragraphs:

Step 1 - Mock Server

The mock server was created using Sinatra, which is a lightweight Ruby framework that can be used to build simple web servers. When running the mock server on localhost, we were able to make sure that our bash scripts were working as intended to perform CRUD (create, read, update, delete) operations on the mock server.

Step 2 - Skeleton for the Custom Terraform provider

Terraform providers are typically created using the Go (Golang) programming language so this was the chosen language for our provider as well.

The main.go was first created with a simple 'hello world' skeleton, just to test if our set-up is working and if we are able to create a provider. To test the functionality, we obviously also needed a bash script that creates the custom provider. As Go files are compiled binaries, they are not dynamically run. You compile them into binaries and then you run the binaries. So running the bash scripts basically creates for us a binary file that is then used as a provider.

Step 3 - Connecting the Custom Provider and Mock Server

At this point, we knew that we were able to use a bash script to take the Go code, compile it and generate the binary executable that can be used as a Terraform custom provider. We also knew that the mock server was functional and CRUD operations should work. The next step was to connect these two things and make sure that the custom provider was able to call the endpoints on the mock server. To test this, we added Terraform configuration that utilizes a custom Terraform provider named "terratowns.:

terraform {
  required_providers {
    terratowns = {
      source = "local.providers/local/terratowns"
      version = "1.0.0"
    }
  }
}

provider "terratowns" {
  endpoint = "http://localhost:4567"
  user_uuid="e328f4ab-b99f-421c-84c9-4ccea042c7d1" 
  token="9b49b3fb-b8e9-483c-b703-97ba88eef8e0"
}

Running terraform apply now instructs Terraform to utilize the 'Terratowns' custom provider. The configuration specifies that the custom provider can be found in a local file, which is the binary file created from the Go code. This custom provider then interacts with the CRUD endpoints on the mock server.

Step 4 - Testing the Production Server

Now that everything was working as intended locally, it was time to make sure that we could interact with the production server. In order for this to work, the endpoint at main.tf had to be changed and we also had to make sure that we had access to the Terratown Cloud by having a valid user_uuid and access token. This test was successful and our Terratown resource was created on the production server.

Step 5 - Creating the whole infrastructure

As a final step after the custom provider was working, we included the Terraform configuration for the AWS infrastructure as well. We could now run Terraform apply, which would use two different providers to create resources on two different cloud platforms. By running just this one command, we created a 'Terratowns home' recourse on Terratowns cloud, as well as an S3 bucket and CloudFront distribution on AWS cloud. Furthermore, the domain URL of the CloudFront distribution was referenced by the 'Terratowns home' resource, which meant that we could provide a direct link from the deployed resource on Terratowns cloud to the static website that was deployed on AWS.

The code for this project can be found in my repository, which you can access here.

Before deciding the design of your stacks it's important to consider how they are all connected. Which stack needs to be created first and which stacks are going to reference each other? Our stacks and the way they cross-reference each other are shown in the diagram and described in more detail below.

Networking, Cluster, Service and Database Layers

All of the CloudFormation artifacts are going to be stored in an S3 bucket that was created manually through the AWS console before we started creating the individual CloudFormation templates.

The first stack was created for the 'networking layer'. We had previously used the default VPC during the bootcamp, but at this point, it was decided that we want to create a custom VPC. Apart from VPC, the networking stack creates also an internet gateway, VPC gateway attachment, route tables, subnets (3 public subnets and 3 private subnets that are not yet used) and subnet route table associations. Any values that other stacks might need are exported so that those can be referenced by other stacks.

The second stack to be created was for the 'cluster layer'. It contains the ECS cluster, application load balancer, HTTP&HTTPS listeners, listener rules, ALB security group, service security group and target group. To the HTTPS listener we imported the CertificateArn we had already created during one of the previous weeks.

The 'service layer' is one where you have to carefully consider which services you want to include within one template. You might not want to have something like ECR or task definitions tightly coupled in a CloudFormation template. We ended up including task definition in the template, however later realized this was not the best possible approach as now updating our task definition causes our Fargate service to restart as well, and that is something that we would like to handle only through the CI/CD pipeline.

The 'database layer' includes an RDS database instance, security group and database subnet group. The database layer ends up being quite tightly coupled with the service layer as the service layer keeps hanging and re-starting containers without a successful db connection. For this reason, we had to move the service security group from the service stack to the cluster stack, so that it is created first.

The above-mentioned stacks are connected as they reference each other as shown in the diagram. The stacks that are introduced next are different as they are independent and don't cross-reference other stacks.

Frontend (CloudFront)

The React.js frontend of this application was originally built as a container. We containerized the application on week 1 and deployed it as a service to Fargate on week 6. However, throughout the bootcamp there was a discussion about whether the frontend should be deployed using S3 and CloudFront instead. The benefits of using ECS would be on-demand-based scaling, easier version control and rollbacks and advanced deployment options such as canary deployments.

For purposes of this bootcamp it was decided that utilizing S3 and CloudFront is sufficient and makes more sense than deploying it as a container. We proceeded by implementing this directly via CloudFormation rather than creating it manually first like we have done with other stacks.

The frontend stack creates a CloudFront distribution, S3 bucket, bucket policy and Route 53 record set. Deploying any changes happens now by manually creating a static build and then using a library called `aws_s3_website_sync`, which syncs a folder from the local dev environment to the S3 bucket and then invalidates the CloudFront cache. A CI/CD pipeline could be created for this by using GitHub Actions.

DynamoDB Layer (SAM)

The DynamoDB layer was created using AWS SAM, which is a subset of CloudFormation. It is designed especially for serverless applications and provides a simplified and higher-level abstraction for defining and deploying serverless resources. The stack created a DynamoDB table, Lambda function, Lambda log group, Lambda log stream an execution role.

As it's not a security best practice to use our main user's access key to update our DDB table, a new CloudFormation stack for 'machine user' was created. This IAM user has access to update DynamoDB. The access keys were manually updated to the parameter store.

Serverless Image Processing (CDK)

The serverless image processing was created by using AWS CDK (Cloud Development Kit). With CDK you can use your preferred programming language (in this case TypeScript) to define your AWS resources. CDK automatically generates CloudFormation templates based on the infrastructure code you write.

The CDK stack creates two S3 buckets, bucket policy, Lambda function and SNS topic. It is worth noting that the whole serverless image processing architecture required two further Lambda functions and a CloudFront distribution. These were created manually through the AWS console and haven't yet been managed by any IaC tool.

Final Thoughts

In conclusion, our journey to automate infrastructure provisioning has achieved significant milestones. What still remains, is incorporating the serverless image processing fully into CloudFormation by leveraging the CDK stack as a nested stack and ensuring all necessary Lambda functions are included in the CloudFormation template. Additionally, the frontend could be fully automated by implementing a CI/CD pipeline, such as GitHub Actions.

Link to my previous article about AWS Cloud Project Bootcamp's DynamoDB week.

Every single week has been challenging, but the DynamoDB week has turned out to be the most stretching until now. Below I cover the main points regarding DynamoDB data modelling for our application.

Data modelling - access patterns

We started with a very insightful 2-hour live stream about data modelling. The design that was chosen for our database is a simple table design. It is a popular choice these days and works well in this kind of scenario where all data is closely linked together. Based on its name it sounds simple but turns out to be quite complicated in terms of data modelling. To get everything working and to keep the cost down, it is crucial to have your data mapped with all different access patterns.

When designing a relational database you simply map the data in logical entities, see what data belongs together in each table and then figure out how to access the data from these tables by using joins. However, with DynamoDB you have to approach this from a completely different perspective. You have to think of your application and what data it is going to need and how. When you know your access patterns, you can start to think about how to organize your data. You can break the rules you would have with relational databases - data can even be duplicated if that works with your access patterns! Storage is cheap and you want your base table to support as many of your access patterns as possible so duplicating data could make sense depending on the situation. You could also choose to save some of the data as JSON instead of separate items if it's not going to be used in any of your queries.

There are so many options for designing the data model for your database. To get the best results from DynamoDB in terms of cost-effectiveness and performance, you really need to do these initial steps correctly.

Access patterns in our application

Our application is a messaging app where the user is able to see a list of their conversations (message groups) and then click an individual message group and see all messages that belong to that message group. Additionally, the user is obviously able to send messages - these could be either completely new messages that start new message groups or further messages to existing message groups. Based on this it was possible to list our initial access patterns:

• pattern A: showing a single conversation (message group).

• pattern B: a list of conversations (message groups).

• pattern C: create a new message

• pattern D: add a message to an existing message group

• pattern E: update a message group using DynamoDB streams

So the database is going to have one table, which is going to contain messages and message groups. Each item is going to have a unique uuid among other fields such as date, display name and message content. Each message group is also going to be listed twice as two individual items, from the perspective of the two users who are parts of the conversation. This is because a list of conversations cannot be displayed identically to both users, the person who is looking at their message groups wants to see the name of the other user listed as a topic of that message group.

Partition keys and sort keys

Then we come to the hardest part of data modelling, choosing the partition key. Partition key means an identifier for the item and it dictates under which partition DynamoDB puts the item under the hood. The partition key doesn't have to be unique and several items can have the same partition key. Sort key instead allows you to uniquely identify that item and allows it to be sorted. The primary key in DynamoDB can be either a simple primary key or a composite primary key (a combination of partition key and sort key). A partition key is always obligatory for any query and only an equality operator can be used. The sort key is not obligatory and not using it would simply return everything.

Our application has two access patterns that relate to messages and three that relate to message groups. For messages, we have to be able to write new messages and display the messages that belong to a certain message group. The best option is to use message_group_uuid as the partition key and created_at as the sort key for it. This is quite logical as we want to display a single conversation, so its identifier uuid is the easiest way to access it. Using created_at as a sort key will give us the option to display the messages within certain timeframes:

 def create_message(client,message_group_uuid, message, my_user_uuid,                  my_user_display_name, my_user_handle):
    now = datetime.now(timezone.utc).astimezone().isoformat()
    created_at = now
    message_uuid = str(uuid.uuid4())

    record = {
      'pk':   {'S': f"MSG#{message_group_uuid}"},
      'sk':   {'S': created_at },
      'message': {'S': message},
      'message_uuid': {'S': message_uuid},
      'user_uuid': {'S': my_user_uuid},
      'user_display_name': {'S': my_user_display_name},
      'user_handle': {'S': my_user_handle}
    }

For message groups, it gets a little bit more complicated. We have to be able to list message groups, add messages to message groups and update message group details. As each user naturally needs to see the message groups that belong exactly to them, the logical option is to use my_user_uuid as the partition key. This will work well as there are two message groups for each conversation, so each participant is going to have a version of the message group with their user uuid. As we want to be able to sort the message groups based on date, the sort key is going to be last_message_at:

 def create_message_group(client, message,my_user_uuid, my_user_display_name, my_user_handle, other_user_uuid, other_user_display_name, other_user_handle):
    table_name = 'cruddur-messages'

    message_group_uuid = str(uuid.uuid4())
    message_uuid = str(uuid.uuid4())
    now = datetime.now(timezone.utc).astimezone().isoformat()
    last_message_at = now

    my_message_group = {
      'pk': {'S': f"GRP#{my_user_uuid}"},
      'sk': {'S': last_message_at},
      'message_group_uuid': {'S': message_group_uuid},
      'message': {'S': message},
      'user_uuid': {'S': other_user_uuid},
      'user_display_name': {'S': other_user_display_name},
      'user_handle':  {'S': other_user_handle}
    }

The catch is that the value of the sort key will of course have to be updated every time a new message is created and added to the message group so that it reflects the date of the actual latest message (access pattern E). This is where a global secondary index is needed.

Global secondary index

GSI is a concept that takes some time to get familiar with. It is basically an index with a partition key and a sort key that can be different from those in the base table. You can imagine creating a new index almost as creating a new table in SQL. It can contain the same items as the base table but in a different order. That means the data is the same, but we twist it and look at it differently. GSIs always add extra costs and you want to avoid them if you can - as already previously mentioned, your base table should support as many of your access patterns as possible.

For our final access pattern E, we want to update the sort key (last_message_at) to reflect the sort key of the latest message (created_at). This will be implemented by using a DynamoDB stream. Every time a new message is created and pushed to a message group, the DynamoDB stream catches the event and triggers a Lambda function. So, how do we get this Lambda function to update the sort key?

As previously mentioned, the message groups have user_uuid as the partition key. So for each update, we have two different message groups with two different user_uuids (as there are always two versions of each conversation, one from the perspective of each participant). Hence we won't be able to find the correct message groups that we need to update based on the partition key. We could of course do a scan with a filter, but that is not a cost-effective solution.

The best option in this situation is to use a GSI. This basically creates a clone of our primary table using the message_group_uuid as the partition key, but the two tables are kept in sync. This GSI allows for querying the table based on the message_group_uuid attribute, in addition to the primary key attributes 'pk' and 'sk':

The GSI was added to the schema:

GlobalSecondaryIndexes= [{
    'IndexName':'message-group-sk-index',
    'KeySchema':[{
      'AttributeName': 'message_group_uuid',
      'KeyType': 'HASH'
    },{
      'AttributeName': 'sk',
      'KeyType': 'RANGE'
    }],
    'Projection': {
      'ProjectionType': 'ALL'
    },
  }],

Now the creation of a new message will be captured by the DynamoDB stream, which triggers a Lambda function that will use the GSI to query all message groups where the message group uuid matches the partition key of the message. It will then replace the sort key (last_message_at) with the sort key value (created_at) of the message. The sort keys for the message and two message groups are now matching:

There is of course a lot more that could be said about the implementation, which was challenging and included a lot of troubleshooting and debugging. However, the whole week has been an outstanding learning experience. Now it's time to get ready for a new week of the BootCamp - ECS Fargate.