So that’s why we create Scripts by Implementing Cloud 🎉. It’s like a pastebin or gist in steroid. Scripts not only save your script, but you can directly download or run using simple curl command. Not only that, Scripts will scan all the code uploaded to Scripts and give you a notes for best practice and security level, thanks to AI checker. We even give you warning or prevent you to run a public script that may dangerously make your computer havoc, steal your data or connect to suspicious server. No more worry about using code from internet. See the demo below and start storing your script in Scripts.

Scripts is a one of many dev tools that we build under Implementing.Cloud umbrella. While the main project is still under development, we proud that we finish the development of Scripts that will heavily used in the main project. Scripts also participate with Hashnode >< Hypermode in ModusHack Hackathon.

[Placeholder for demo video]

When we say Scripts will securing your script, don't take my word for it; look the code in Github. You can easily reproduce it by using my nix configuration. Simply run nix develop and you got all the requirements you need.

And while we was working on it, we create a PR for the use of JWKS documentation.

Initial System Design, Iteration and Exploration

We want our user interact with Scripts in two way:

1. Submit and browse their code via web app

2. Get or execute their code via terminal

To make sure there is no abuse to platform, only authenticated user can submit their code but everyone can download it. The submission is not a simple insert to database, but there will be some AI analysis to determine whether it follow best practice and most importantly is it secure or not to execute. We also wrap the original code with our predefined script to restrict the execution and give user some warning based on the analysis and community notes before the execution.

Actually we are really interested to utilize an automatic sandboxing in Modus by executing the script in the platform, but we don't have much time to explore and debug it.

After some time exploring modus, we learn several things (and detail about it in the next few sections):

1. We are using Go and it’s incredibly easy to make a function available for request

2. Currently modus only support GraphQL. I hope in the future I can see support for RESTful API (Achievable with API Gateway) and async request for background process, especially AI process without blocking user request (Achievable with embedding NATS outside modus).

3. It support Auth via JWT out of the box, but we have difficulties to mix private and public space

4. Because the sandboxing nature, we need to “whitelist” to external server. I use this to connect my database via PostgREST, connecting to PostgreSQL via RESTful API.

5. [Placeholder to talk about AI]

Comparing Syntax for Modus with FaaS

Let’s assume that we will have a function that receive a owner of the script and the name of the script and will return the Code struct like this. It will call a database but let’s leave it for later discussion.

type Code struct {
    Owner         string `json:"owner"`
    Name          string `json:"name"`
    CodeType      string `json:"code_type"`
    ScriptContent string `json:"script_content"`
}

First of all, let's see the contender

AWS Lambda

package main

import (
    "context"
    "encoding/json"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    owner := request.QueryStringParameters["owner"]
    name := request.QueryStringParameters["name"]

    code := GetCode(owner, name)

    responseJSON, _ := json.Marshal(code)
    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       string(responseJSON),
    }, nil
}

func main() {
    lambda.Start(handler)
}

GCP Functions

package function

import (
    "encoding/json"
    "net/http"
)

type Code struct {
    Owner         string `json:"owner"`
    Name          string `json:"name"`
    CodeType      string `json:"code_type"`
    ScriptContent string `json:"script_content"`
}

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func CodeHandler(w http.ResponseWriter, r *http.Request) {
    owner := r.URL.Query().Get("owner")
    name := r.URL.Query().Get("name")

    code := GetCode(owner, name)

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(code)
}

Azure Functions

package main

import (
    "encoding/json"
    "net/http"
)

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func CodeHandler(w http.ResponseWriter, r *http.Request) {
    owner := r.URL.Query().Get("owner")
    name := r.URL.Query().Get("name")

    code := GetCode(owner, name)

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(code)
}

func main() {
    http.HandleFunc("/api/GetCode", CodeHandler)
    http.ListenAndServe(":8080", nil)
}

Cloudflare Worker

Worker known for Javascript runtime, but like Modus, Cloudflare Worker also available for Go by compile it to WASM.

package main

import (
    "encoding/json"
    "syscall/js"
)

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func handleRequest(this js.Value, args []js.Value) interface{} {
    query := args[0]
    owner := query.Get("owner").String()
    name := query.Get("name").String()

    code := GetCode(owner, name)
    codeJSON, _ := json.Marshal(code)

    return js.ValueOf(string(codeJSON))
}

func main() {
    js.Global().Set("handleRequest", js.FuncOf(handleRequest))
    select {}
}

OpenFaas

package function

import (
    "encoding/json"
    "fmt"
    "net/http"
)

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func Handle(w http.ResponseWriter, r *http.Request) {
    // Parse query parameters
    owner := r.URL.Query().Get("owner")
    name := r.URL.Query().Get("name")

    if owner == "" || name == "" {
        http.Error(w, "Missing owner or name query parameter", http.StatusBadRequest)
        return
    }

    // Get the code details
    code := GetCode(owner, name)

    // Respond with JSON
    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(code); err != nil {
        http.Error(w, "Error encoding response", http.StatusInternalServerError)
        return
    }
}

Knative Functions

Open source solution that utilize Knative that work on top of Kubernetes.

package main

import (
    "encoding/json"
    "net/http"
)

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:         owner,
        Name:          name,
        CodeType:      code.Type,
        ScriptContent: code.Content,
    }
}

func CodeHandler(w http.ResponseWriter, r *http.Request) {
    owner := r.URL.Query().Get("owner")
    name := r.URL.Query().Get("name")

    code := GetCode(owner, name)

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(code)
}

func main() {
    http.HandleFunc("/", CodeHandler)
    http.ListenAndServe(":8080", nil)
}

Hypermode Modus

With Modus, we remove all the boilerplate of parsing request and response. The true function as a service.

func GetCode(owner, name string) *Code {
    code := db.GetCode(owner, name)
    return &Code{
        Owner:    owner,
        Name:     name,
        CodeType: code.Type,
        ScriptContent:   code.Content,
    }
}

As you can see, it’s incredibly easy to write Modus compared to other solutions. Especially, it already parse the request into our defined struct without manual binding.

Exploring Modus SDK and Runtime

I have quite a long discussion and reporting a bug in Hypermode discord. It’s fun to learn how the modus work, thanks to the beauty of open source. First we can see, beside modus dev, there is another things like “runtime” and “sdk”. We can use different version of runtime and SDK by installing it by using modus runtime install v0.14.3 or modus sdk install go v0.14.3, see the source of release in github release.

After it install, we can use the specific runtime by running modus dev -r v0.13.1. For SDK, add the version we want to use in the go.mod

module infinite-script

go 1.23.3

require github.com/hypermodeinc/modus/sdk/go v0.14.3 // indirect

There are a “hidden” tool that generate additional go code modus_pre_generated and modus_post_generated.go, it’s called modus-go-build that inside the .modus/sdk/go/<version> folder. Sometimes, it doesn’t install along with the modus sdk install command, so as a workaround I run GOBIN=~/.modus/sdk/go/v0.14.3/ go install github.com/hypermodeinc/modus/sdk/go/tools/modus-go-build@v0.14.3 manually. Well, I found the command after looking at the source code.

All SDK and runtime is saved under $HOME/.modus

Exploring GraphQL and Auth

This simple modus.json that came from template will generate this nice GraphQL Explorer when running locally.

{
  "$schema": "https://schema.hypermode.com/modus.json",
  "endpoints": {
    "default": {
      "type": "graphql",
      "path": "/graphql",
      "auth": "bearer-token"
    }
  }
}

As you can see, it has "auth": "bearer-token" but actually it will be still be public as long as we are not set MODUS_PEMS like defined in the doc here. But honestly I don’t like how we set it up since we need to put the public key (that has very sensitive even we miss a single newline character) in the JSON and serialize it in a single line of string. On the other hand, I use Authentik for my Auth Provider (Like auth0 or clerk, but self-hosted) and it support providing the public key by serve a JWKS endpoint. Does modus support it? Yes! Unfortunately undocumented.

I found it in the Changelog (what a great changelog!), it introduce at version v0.13.0 with this PR and a fix in version v0.14.0. TLDR, all you need is to add MODUS_JWKS_ENDPOINTS in .env.dev.local. If you deploy your app, don’t forget to add it in the dashboard.

MODUS_JWKS_ENDPOINTS='{"implementing.cloud":"https://auth.tegar.my.id/application/o/implementing-cloud/jwks/"}'

But you don’t need authentik to use JWKS, you can use your favorite auth provider and the popular one most likely have this feature shipped. The interesting part is we can have multiple auth provider at once!

Add Postman for Testing Auth Protected API

After we put MODUS_JWKS_ENDPOINTS or MODUS_PEMS, the built in GraphQL explorer will not work. It also quite a troublesome to copy-paste JWT token from network to use in our testing, especially if we set the life time of access token is only a few minutes.

Since I use OAuth 2.0 based with browser login, you can use Postman to automatically retrieve the token.

If success, you can get the schema and make a request with postman to Modus. Don’t forget to enable Using GraphQL introspection instead of import a GraphQL Schema.

Mixing Public and Private Route

Attempt #1: Make every functions behind authentication

This is my initial setup that I research above. But when integrating with frontend, I just realized that not all API should behind the auth, especially when get the script or list the public scripts. I ask in discord but unfortunately it’s not supported yet. They suggest I need to deploy 2 different app, but let’s try some workaround in the next attempt.

Attempt #2: Create 2 endpoints in modus.json

I try to modify the file with this configuration

{
  "$schema": "https://schema.hypermode.com/modus.json",
  "endpoints": {
    "private": {
      "type": "graphql",
      "path": "/private",
      "auth": "bearer-token"
    },
    "public": {
      "type": "graphql",
      "path": "/public",
      "auth": "none"
    }
  }
}

I try to run it locally and it work. Both endpoints will have the same schema and same target functions. I can add Public and Private in the function name and just reject unauthorized request that going to private.

Unfortunately, it’s currently not working when I deploy it. It’s looks like they hardcode /graphql path in the dashboard and only open that path, my non /graphql path like /private and /public return 404. Please check the discussion here.

Attempt #3: Create 2 deployment

Okay actually I want to make this my last attempt. So I try to make a monorepo, so multiple modus app in one app. It will help me shared the same code with ease. Modus github action template is awesome, it also have MODUS_DIR env. But unfortunately it’s not yet supported. Even though I have 2 CI and separate trigger, there is no way to specify this target should listen to which CI or path. So the current solution is implement with 2 projects with 2 different repositories.

Bonus Attempts: Add KrakenD API Gateway to make a RESTful API Request

GraphQL is offering a great flexibility to query data, so there will be no over-fetching in the client side. But if we publicly make the GraphQL available for everyone without limit, then it will be easily abused by bad actor. It’s usually call Nested Query Attack, like discussed in the stackoverflow. While I’m not sure Modus will suffer this attack (in my case, I can’t see how it possible) but the solution usually is to implement a persisted queris that looks like this, I’m not the fan of it though.

{
  "variables": {
    "path": "/article-path"
  },
  "extensions": {
    "persistedQuery": {
      "sha256Hash": "c205cf782b5c43c3fc67b5233445b78fbea47b99a0302cf31bda2a8e2162e1e6"
    }
  }
}

I will add API Gateway in front of them to make it easier to read, and additionally give more features like rate-limit, parallel query, logging, observability and hide our both public and private Modus deployment. I choose KrakenD because I have some experience with it, but every other API Gateway will also works. Honestly I wish Hypermode will provide it in the future out-of-the-box.

KrakenD is open source API Gateway that can connect to GraphQL. The example of the configuration is something like this.

{
    "endpoint": "/code/{username}/{scriptname}",
    "method": "POST",
    "backend": [
        {
            "timeout": "4000ms",
            "url_pattern": "/graphql?timeout=4s",
            "extra_config": {
                "backend/graphql": {
                    "type":  "mutation",
                    "query_path": "./graphql/mutations/code.graphql",
                    "variables": {
                        "user": "{username}",
                        "scriptname": "{scriptname}",
                        "other_static_variables": {
                            "foo": false,
                            "bar": true
                        }
                    },
                    "operationName": "addMktPreferencesForUser"
                }
            }
        }
    ]
}

Attempt #4: Back to #1 => make every functions behind authentication from API Gateway

Well, everything is a cycle. I just realize this approach after implementing API Gateway. Remember that we can have multiple JWKS and PEM in one app? Yes! Why not make the public route is accessible from public, but API Gateway will supply the authentication. With this implementation, I can remove my second application and all duplication I made.

Above are two requests against same deployed modus app. One is using OAuth 2.0 for User and the second one is using normal JWT created by jwt.io using my own private+public key pair. The JWT I create is using "sub": "service-account" and I arbitrary give the expired time 2099. Just make sure the JWT is not leaked :D.

Add Async Request for Background Process

Unlike other FaaS, modus doesn't have this feature built-in yet. Async request is great for detach current user request to run long running process in the background. But it doesn't mean we can't implement it with modus. Here is a workaround.

To make a async request, usually we need intermediary service as a broker. In this scenario I'm using nats and embed it in our existing go app to keep the architecture simple.

When functionA in modus finish to validate and save to database, it will send an async request to NATS with the inserted id and response to user. NATS that received the request will make another request to functionB in modus to run long running process: AI analysis. The process will have a 2 steps ideally, first using faster AI model we will evaluate the score of best practice and security risk. If there is a single potential risk, we will re-evaluate it with more advance AI model and ask for more comprehensive reasoning.

With this mechanism, we will get both benefit of run an advance model if needed or use smaller and cheaper model in most case without compromised user experience.

Final Design System

Code Implementation

Before we talk about code, we think it's wise to learn about what features will be provided to our users:

1. User can submit the code with type of runnable script or config file, more types coming soon. Additionally, user can add metadata like description and user steps (like custom input or change path).

2. After submission, modus call an async request. The request will detach user request, so no additional time to wait the request and we can run AI analysis in the background.

3. User can import script from URL, modus will get the scripts behind the scene and do the submission like usual.

4. User can browse uploaded scripts.

5. User can check scripts detail, including how to execute or get the config file, AI analysis result, AI and Community notes and of course the full scripts and wrapped scripts.

6. Additionally, you can ask AI to generate scripts for you.

[Placeholder for code]

Conclusion

Modus is really new but promising. In current state, we just wonder why not every Function as a Service done something like this. I’m amaze how fast I create the backend logic. Not mentioning this is my first time creating an API with LLM capabilities and compiling to WASM, even though all of it is abstracted by modus. Most of the time working in this project is mostly in the frontend side and read the internal of modus.

Here is summary of some feedback I mention above that I hope Hypermode can add in the future:

• RESTful API / API Gateway capabilities or built-in integration

• Async / Event Based Request

• Mixing public and private endpoint or schema

• Monorepo deployment

Nevertheless, it was really enjoyable journey. Thank You.

Nb: Even though in this article we use “we” as a author, in reality only me without a team doing all the things :D

Bayangkan momen penting bersama keluarga...

Sebentar lagi mau Idul Fitri dan Anda bertemu dengan keluarga tercinta setelah sekian lama tidak bertemu. Anda tidak lupa untuk mengabadikan momen spesial tersebut dengan berfoto bersama dan dikirim lah oleh sepupu via Whatsapp Group. Sebulan kemudian Anda ingin melihat foto tersebut tapi ternyata sudah tidak ada dan lupa di download. Mau minta kirim lagi tapi sungkan atau malah fotonya juga sudah terhapus.

Bayangkan invoice belum terbayar tapi sudah hilang...

Anda ingin mengirimkan invoice atau dokumen penting lainnya ke client atau bos. Setelah berkali kali revisi, akhirnya dokumen final terkirim juga. Sekian purnama berikutnya, ternyata beliau minta dikirim ulang secepatnya karena akan ada audit dan kalian sedang liburan. Dokumen itu juga sudah tidak ada di hp maupun laptop, download juga sudah tidak bisa. Satu satunya cara adalah buat lagi, tapi lupa versi mana yang benar dan kebetulan sedang mudik ke kampung.

BackMeUp membantu anda menyimpan Gambar, Video, Dokumen dan Chat dengan mudah

Forward Gambar, Video, Dokumen atau Chat ke Official Whatsapp Kami dan file akan tersimpan pada Google Drive Anda. Berbagai fitur tambahan lain akan diimplementasikan sesuai dengan kebutuhan Anda.

Privacy Dijamin. Dengan sistem canggih kami, file yang Anda kirim tidak dapat disimpan atau dilihat selain oleh Anda.

Ayo bergabung menjadi Early Adopter dan dapatkan diskon hingga 67% dengan mendftar di link ini

Note 2: This article is still a work in progress that expected to finish on September 26th. You may see something like <Placeholder for xxx> or TBA(To Be Added). Follow me or check periodically to know more updates.

Outerbase is like a dbeaver or phpmyadmin, where you can connect your database, then you can browse tables and query, insert, update and delete your data easily. But not like those two, Outerbase doesn't require you to install anything, you can customize the look and feel (not only table like spreadsheet) and can manipulate the data via Javascript, SQL and Python through RESTful API. The ability to create an API without worrying about backend deployment is called command in Outerbase. In this article, we will try to use the command, deep dive into some usage and current limitations and see what am I prepared for you.

TLDR: I prepared how to build a command with splitting files and 3rd party library (+ GitHub repo template -- check below), avoid writing a hard-coded secret to call 3rd party services, and protect an endpoint so not everyone can access the data. Everything while we create some sample integration. See the demo video below.

<placeholder for demo video>

Github for repo template: outerbase-commands-workspace

Github for complete command with integrations: <placeholder for github url>

In this article, we will mostly talk about the backend side, thus we will talk about Outerbase command, database configuration and integration with several 3rd parties that help us finish our job. List of integrations:

• PNPM and Vite for building complex node

• Supabase Vault for storing secrets (+ AWS service example)

• Token-based and JWT for the protected endpoint (+ Clerk integration)

• Qstash from Upstash for the delayed message

• Ntfy for Push notification

• Mailgun for Monthly report email

• Axiom for app log and audit log (help me as developer)

Preparation

Idea and Prepare Some Data

Some weeks ago, I had a chat with my father. He wants to track his salespeople visits to several selected stores in the same region. For that, he needs to have salesperson data, store data and a list of stores that the salesperson needs to visit every day. When salesperson arrive in the store and do their things, they need to check in by filling form and capturing a photo. My father as a manager will see a dashboard that shows an overview of all salespeople's progress in real-time.

For that purpose, I will use a PostgreSQL database provided by Supabase. Connecting Outerbase and Supabase is really easy, but we will look further to make use of both features to the fullest. For this demo, I will use some dummy data that represents actual data. You can see the data in Google Sheet: https://docs.google.com/spreadsheets/d/1gnyEaF6oJ8XNrOWkWzd9ZdwXkA6ivgJMfAF-QTSbijM/edit?usp=sharing

-- Image: Example Data for Our Internal Tool

Prepare Database and Import Data

You can get a PostgreSQL database for free in Supabase. For connecting supabase with Outerbase, you can follow official docs of Outerbase or Follow the steps for more visuals provided by my fellow tech writer.

-- Image: Outerbase Screen After You Connect to a Database

After you get a database instance, now is your time to create some tables and provide some data. Download the Google Sheet above as two CSV files, one for salesperson and one for store. With Supabase, you can import data from CSV. Open your Project -> Database -> + New Table -> (in Columns section) import data via spreadsheet. Just drop the file here, give it a table name, set the type of data and you will see the column and data will be automatically created.

-- Image: Create a Table By Importing a CSV in Supabase

After the import process is finished, we can go back to Outerbase and create some queries. Go to your database and click New -> Query.

-- Image: Outerbase Show Our Newly Created Table. Now We Will Create a Query Manually or Via EZQL

If you are familiar with Metabase or Redash, you normally write your own SQL to get the data. But Outerbase gives us a fresh way to get the data, it's called EZQL, you just write what question you want to ask about the data and it will generate the SQL and give you the data. Just like ChatGPT, but it knows the context of data. Let's give it a try by asking about how many salespersons and stores are in my database and the ratio of store to salesperson.

-- Image: It took me 3 prompts (1 initial, 1 revise and 1 to optimize) until I got the expected result.

-- prompt: I want to know how many I have salesperson, store and ratio of store to salesperson
SELECT COUNT(*) as salesperson_count FROM public.salesperson

-- prompt: well, that only count the salesperson, I also want to see store and the ratio
SELECT
  (SELECT COUNT(*) FROM public.salesperson) AS salesperson_count,
  (SELECT COUNT(*) FROM public.store) AS store_count,
  (
    SELECT COUNT(*)::float / (SELECT COUNT(*) FROM public.salesperson) FROM public.store
  ) AS store_salesperson_ratio

-- prompt: That cool! But I see we query each table twice. Can't we do that only once?
SELECT salesperson_count, store_count, store_count::float / salesperson_count AS store_salesperson_ratio
FROM
  (
    SELECT COUNT(*) AS salesperson_count
    FROM public.salesperson
  ) sp,
  (
    SELECT COUNT(*) AS store_count
    FROM public.store
  ) st

Great! So I know there is 30 salesperson and each one of them needs to handle 16-17 stores, not ideal BTW. Side note: I also ask ChatGPT to revise my first SQL, but there is no luck since it doesn't understand the table schema.

Okay, let's go with our next step. We need to create an agenda table that stores a list of salesperson visits. Let's give EZQL a try.

-- prompt: create a new table called "agenda" that have relation to salesperson and store, 
--         and also add a columns: visit_date, status. And don't forget uuid as primary key id.
CREATE TABLE IF NOT EXISTS agenda (
  id uuid PRIMARY KEY,
  salesperson_id uuid REFERENCES public.salesperson(Id),
  store_id uuid REFERENCES public.store(Id),
  visit_date date,
  status varchar(255)
);

Looks good, but unfortunately, that returns an error. By the time this article was written, Outerbase hang and didn't throw an error. But, I believe they will fix it soon before graduate from beta version. Fortunately, I can get the error log from Supabase.

It took me some time to resolve the error, by using EZQL, ChatGPT, StackOverflow and in the end, I resolved it by making my hand dirty with direct trial error with SQL Query. The solution is to use quote to Id in reference definition. So, instead of public.salesperson(Id) I need to write public.salesperson("Id"). Maybe that's because I use capital in the column name (since the CSV is like that and I was too lazy to change it), please leave a comment if you know about this.

Removing Unused Table from Outerbase

Supabase already created some tables for it's internal and we don't need it for our app. And because we use a root account (postgres) for Outerbase, it will get all schema and tables. For my non-technical father, it will be too much!

You may need to read about Access Roles to understand how Outerbase will behave with your data. TLDR, we will create new Postgres roles (accounts) that have access only to our table.

create role "outerbase" login password 'your-secret-password';
grant insert, select, update on all tables in schema public to outerbase;

After that, we can change the connection setting in Outerbase (In your project, go to Settings -> Database) and see if we only see tables for our app.

-- Image: Before you will see all schema, but after login with new roles and granting only specific permission, the table will be neater.

Notes: You may want to disable RLS (Row Level Security) or create a new policy for Outerbase to access the data. If not, you will not be able to get the data.

Creating an Outerbase Command to Manipulate Data

Okay, we have data and access to it via query. Now is the most interesting part: creating a logic and manipulating our data with Outerbase Command.

Hello World for Command

First of all, go create a new command.

A command is a single endpoint path (e.g. /hello) that can be called using GET, POST, PUT and DELETE verbs. After we hit the endpoint, a node will be processing the request. If we have more than one node, it will execute the next node -- let's call it node-1, node-2, node-3, and so on -- sequentially until the last node returns a response to the user. node-2 can get a return value from node-1 and node-3 can get return values from node-1 and node-2. All nodes can get data from request.body and request.query.

To make you understand the process, I created a simple hello world for testing the result.

// Node name: Node 1
function userCode() {
    return {
        from: "node-1",
        body: {{request.body}},
        query: {{request.query.token}}
    }
}

// Node name: My Custom Name
function userCode() {
    return {
        from: "node-2",
        body: {{request.body}},
        query: {{request.query.token}},
        "node-1": {
            full: {{node-1}},
            from: {{node-1.from}},
            body: {{node-1.body}},
            query: {{node-1.query}}
        }
    }
}

// Node name: Node 3
function userCode() {
    return {
        from: "node-3",
        body: {{request.body}},
        query: {{request.query.token}},
        "node-1": {
            full: {{node-1}},
            from: {{node-1.from}},
            body: {{node-1.body}},
            query: {{node-1.query}}
        },
        "node-2": {
            full: {{my-custom-name}},
            from: {{my-custom-name.from}},
            body: {{my-custom-name.body}},
            query: {{my-custom-name.query}}
        }
    }
}

Keep in mind, that's not a valid javascript so you will find your linter will complain. Don't worry, we let Outerbase replace the variable before executing it as a valid javascript. See my workaround using PNPM workspace and vite in the below section for the solution.

And here is the result

{
  "from": "node-3",
  "body": "",
  "query": "test",
  "node-1": {
    "full": "{\"from\":\"node-1\",\"body\":\"\",\"query\":\"test\"}",
    "from": "node-1",
    "body": "",
    "query": "test"
  },
  "node-2": {
    "full": "{\"from\":\"node-2\",\"body\":\"\",\"query\":\"test\",\"node-1\":{\"full\":\"{\\\"from\\\":\\\"node-1\\\",\\\"body\\\":\\\"\\\",\\\"query\\\":\\\"test\\\"}\",\"from\":\"node-1\",\"body\":\"\",\"query\":\"test\"}}",
    "from": "node-2",
    "body": "",
    "query": "test"
  }
}

As you can see if you directly refer to {{node-1}}, the return will be a string of JSON. So you need to parse it first using JSON.parse({{node-1}}). And make sure there is no quote before and after {{node-1}} since it will break the template mechanism by Outerbase.

And if you see directly, if I rename a node, then I will need to rename how I call it. For example, I call the second node {{my-custom-name}}, not {{node-2}}, since the name of the second node is My Custom Name.

Accessing Data with SQL

We can use any SQL command and add input from request data or previous nodes. The use case is not only for reading data from db, but we can also insert, update or delete data.

Hit the endpoint /hello/world?token=test&limit=2 will bring the result of the SQL node like this

{
  "success": true,
  "response": {
    "items": [
      {
        "Id": "7036b866-e2f8-4b34-a7d8-88e0b6e523db",
        "Name": "Erin Vargas",
        "Email": "oscott@example.org",
        "Phone": "+6281267188370",
        "Avatar": "https://i.pravatar.cc/100?u=oscott@example.org"
      },
      {
        "Id": "42a4331c-8e85-43a6-afbf-1791b963e909",
        "Name": "Dr. Susan Combs DDS",
        "Email": "victoria95@example.com",
        "Phone": "+6281280966669",
        "Avatar": "https://i.pravatar.cc/100?u=victoria95@example.com"
      }
    ],
    "schema": false
  }
}

We can use this result for the next node, in this case, I use Javasript node. Please keep in mind if you pass directly from {{node-name}} then you need to parse it first. For that, usually, I create a helper function like this:

function parseResult(resultStr) {
    try {
        const result = JSON.parse(resultStr)
        if (!result.success) return null
        return result.response.items
    } catch {
        // For now, if the SQL throw error, the return is not a valid JSON
        // so we will catch it here
        // bug fix in progress by Outerbase team
        return null
    }
}

function parseResultOne(resultStr) {
    const result = parseResult(resultStr)
    if (result === null) return null
    return result.response.items[0]
}

function userCode() {
    const result = parseResultOne({{node-1}})
    return 
}

--

A little bit about SQL Injection. I have very little knowledge about that so please take it with a grain of salt. Assume we have this SQL: SELECT {{request.query.limit}} as limit, {{request.query.token}} as token;

Hitting this endpoint /hello/world?token=test&limit=(select "Id" from salesperson limit 1) will not return the data from salesperson, instead, it returns as a literal string.

{
  "success": true,
  "response": {
    "items": [
      {
        "limit": "(select \"Id\" from salesperson limit 1)",
        "token": "test"
      }
    ],
    "schema": false
  }
}

Limitation Using This Approach

• For a normal backend, we most likely use more than one endpoint path (command). For a single command, we will have more than one node. Managing all of that by writing directly in the Outerbase will be risky, e.g. incidentally deleting a node or command.

• We can't use 3rd party library or split our code into several files

• We can't reuse the same code

• The template format {{request.body}} will break the linter/syntax highlighter/IntelliSense in VSCode or your favorite IDE

These problems made me create a solution in the next section.

Using PNPM Workspace and Vite

First of all, I already create a template you can work for so you don't have to prepare it yourself. Go to this Github repo outerbase-commands-workspace and click use this template(don't forget to star 😆). The next 2 subsections will talk about how the template works internally after that we will learn how to use it in real code.

PNPM Workspace for Multiple Command and Nodes

I use pnpm instead of npm and yarn, and pnpm workspace is how we built a monorepo (don't confuse it with monolithic). With monorepo, we can put all of our code for command and node in a single place. Interestingly, we can also share functions to be used by some nodes without maintaining another repo (for that lib). Using pnpm, we also save some storage since the library will be symlinked instead of installing node_modules for each of our nodes.

After using the template and cloning your repo, you can run pnpm install to initialize. Here is the folder structure

$ tree -L 1
.
├── commands            # the source code
├── dist                # the build code
├── node_modules
├── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml  # define where is our source code
├── README.md
└── scripts              # custom bash script and templates

And here are the available commands

$ # create a new code
$ pnpm run new COMMAND_NAME NODE_NAME 

$ # install dependency from package.json
$ pnpm --filter "COMMAND_NAME-NODE_NAME" install

$ # build into a single file 
$ # output: COMMAND_NAME-NODE_NAME.js
$ pnpm --filter "COMMAND_NAME-*" run build 

$ # build into a single file and minify (but preserve "userCode" function name)
$ # output: COMMAND_NAME-NODE_NAME.min.js
$ pnpm --filter "COMMAND_NAME-*" run build-minify 

$ # build into a single file and minify and remove quote for {{request.*}} and {{node-*}}
$ # output: COMMAND_NAME-NODE_NAME.min.js.outerbase
$ pnpm --filter "COMMAND_NAME-*" run build-minify-outerbase 

$ # minify an existing build that not minified yet 
$ pnpm run minify COMMAND_NAME-NODE_NAME # without extension

$ # convert to outerbase compatible template (remove quote for {{request.*}} and {{node-*}}) for existing build
$ pnpm run convert-to-outerbase COMMAND_NAME-NODE_NAME.js # with extension

Running pnpm run xxx will see in package.json the root and run xxx script. But running pnpm --filter "COMMAND_NAME-*" run xxx will see package.json in all matched project names and run xxx script defined there.

Using pnpm run new actually runs a bash script located in scripts/new_command.sh which copies scripts/templates folder to the commands/ folder and make some changes for command and node names in the template. You can modify the content of the template and it will be included in your next generated code.

Vite for Spliting Files and Use 3rd Party Lib

I use Vite for bundling the code (actually, rollup does the work). Let's try to create a hello world for that: pnpm run new hello node-1. After that, we will have a new folder in command/hello-node-1. Let's see what we have there.

$ tree commands/hello-node-1/

commands/hello-node-1/
├── package.json
├── src
│   ├── index.js
│   └── world.js
└── vite.config.ts

1 directory, 4 files

$ tail commands/hello-node-1/src/*

==> commands/hello-node-1/src/index.js <==
import world from "./world"

export function userCode() {
  console.log = "test"
    return {
      status: "success",
      message: `hello ${world()}`,
      msg: "{{request.query.INPUT_NAME}}"
    };
  }

==> commands/hello-node-1/src/world.js <==
export default function user() { return "Tegar"}

In the sample code, I prepared index.js for our entry point that will import a local file world.js. You may notice that I add a quote in {{request.query.INPUT_NAME}}, that will be broken if we use it directly as a Outerbase node. But if we do not add the quote, it will be broken in our IDE since it's not a valid javascript syntax. So my solution is we will add the quote in our development environment and then run script/outerbase_compatible.sh to remove the double quote in the build file. Don't worry, I already simplified it as pnpm run script like previously I mentioned.

Let's add a 3rd party dependency, build it and deploy it to Outerbase. I will pnpm --filter "hello*" install is-odd and make some changes

import world from "./world"
+ import isOdd from "is-odd"

export function userCode() {
  console.log = "test"
    return {
      status: "success",
      message: `hello ${world()}`,
-     msg: "{{request.query.INPUT_NAME}}"
+     input: isOdd(parseInt("{{request.query.number}}"))
    };
  }

After that, run pnpm --filter "hello*" run build-minify-outerbase and copy the content of dist/hello-node-1.min.js.outerbase. Yes, it's only a few lines and all functions will be renamed except for the userCode. We also removed the quote for the Outerbase template variable.

Paste the code to the Outerbase command and deploy it by clicking Run All. You can run the code by create a GET request to /hello/world?number=3. I usually wait until the log shows me something like this to make sure it's deployed successfully.

11:28:52:325 Starting to build Spin compatible module
11:28:52:325 Preinitiating using Wizer
11:28:54:070 Optimizing wasm binary using wasm-opt
11:29:10:357 Spin compatible module built successfully

Create More Useful Commands and Nodes using This Repo

Now let's try to create a node that will create an agenda for salespeople to visit several stores on specific dates. We will create a POST endpoint, validate the input and store it in the database.

TBA

Integration with 3rd Party Services and Add Security

Encrypt Secret in Database with Supabase Vault (+ Use Case to Integrate with AWS)

More about Supabase Vault can be read here. In short, we encrypt our secret in the database and just in case someone gets access to our database backup, they only get the encrypted data. To receive the data, we can access a VIEW TABLE and get the decrypted data. Actually, we can achieve this using built-in encrypt and decrypt functions like this example, but I found that using Supabase Vault is easier (to create and delete the secret) and simpler (in SQL form).

First of all, go to your Supabase project and go to Settings -> Vault -> Add new secret. Here you will get a form to create a secret. For example to integrate with AWS Services:

After that, we can get the secret by using this SQL

-- Get from one key
select name as key, decrypted_secret as value 
from vault.decrypted_secrets where name = 'AWS_ACCESS_KEY_ID';

-- Get from multiple keys
select name as key, decrypted_secret as value 
from vault.decrypted_secrets where name in 
('AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY');

The output will be something like this

{
  "success": true,
  "response": {
    "items": [
      {
        "key": "AWS_ACCESS_KEY_ID",
        "value": "AKIAX7Y4GCQPDK397AVM"
      }
    ],
    "schema": false
  }
}

Don't use this as the last node in the command except for debugging, or else the secret will be leaked to the user

Usually, I make a wrapper so I can access it as a key-value pair. parseResult is from the previous section accessing data with SQL.

// Node name: Get Secret
function userCode() {
    const secretStr = {{get-secret-node}}
    const secret = parseResult(secretStr); // Refer to previous section
    const secretKV = secret.reduce((obj, item) => {
      obj[item.key] = item.value;
      return obj;
    }, {});
    return secretKV
}

// In other node
const config = new AWS.Config({
  accessKeyId: {{get-secret.AWS_ACCESS_KEY_ID}}, 
  secretAccessKey: {{get-secret.AWS_SECRET_ACCESS_KEY}}, 
  region: 'us-east-1'
});

The example of the AWS config is from the official documentation. You can use your imagination to use AWS Services.

While this is possible, I still prefer to add secrets in environment variables like how we usually do and comply with the 12-factor app. I hope Outerbase will support this soon.

Protect Endpoint with Token-Based Authentication and JWT (+ Use Case with Clerk)

Since Outerbase command still does not support built-in Authentication nor header and cache, then we don't have many options to make sure our endpoint can respond only to authorized requests. For this, I will use token key in the query string. So for each command, I will expect something like this in the first node:

// Node name: "Check Auth"
function userCode() {
    // let's hard code the token for now
    // and we will modify later
    if ({{request.query.token}} === "letmein") {
        return { authorized: true }
    } else {
        return { authorized: false }
    }
}

Unfortunately, by the time this article was written, Outerbase didn't support exiting early without continuing to chain the node. So I created a small lib to check authorization and wrap the main function with it. I do this for all nodes after the Check Auth node. Assume you are using the template repo I provided:

// lib/checkAuth.js
export default function checkAuth(fn) {
    // The build step will remove the quote
    // Make sure your auth node name is "Check Auth" or else it doesn't work
    if ("{{check-auth.authorized}}") {
        return { status: "error", code: 403, message: "Unauthorized" }
    else {
       return fn()
    }
} 

// Import to main.js
import checkAuth from "@/lib/checkAuth"

function main() {
    // Do something heavy
}

function userCode() {
    return checkAuth(main)
}

So later if Outerbase finally supports it, I just simply remove the import and wrapper function.

Back to the token, the token checker is hard-coded and that's not ideal. We can use our previous about encrypt secret in database and then use it. It's very simple to implement but now since the token is only one, we can't differentiate who makes a call. It's not suitable for apps that are used by multiple users.

And now for that scenario, we can use a different approach: JWT. We can create our own JWT using JWT libraries from your favorite programming language. We can use Auth services like Clerk or Auth0, but in this article, we will focus on using Clerk. The idea is the user will sign in to clerk -> clerk generates JWT -> client side (browser) will create a request with JWT -> Outerbase command in Check Auth node will verify whether it's a valid JWT or not.

Normally, we will retrieve the JWT from the request header or session. But for our approach now, we will expect the client to send the JWT via query string.

For getting JWT from Clerk, you can see the code I use from stackblitz here. (Refresh the stackblitz if shows error)

For verifying JWT, you can refer to the Clerk documentation about manual verification. This is a modified version that is simplified for our use case. Since this needs 3rd party library, you may need the build tool.

// Node name: "Check Auth"
import jwt from "jsonwebtoken";

function userCode() {
    const token = "{{request.query.token}}"

    // Get from API Key -> Advanced -> JWT Public Key -> PEM
    const publicKey = `-----BEGIN PUBLIC KEY-----
    ....
    -----END PUBLIC KEY-----`;
    try {
        const decoded = jwt.verify(token, publicKey);
        return { authorized: true, sessToken: decoded }
    } catch (error) {
        return { authorized: false, error: "Invalid Token" }
    }
}

Too complicated? Actually, you can use this API from Clerk to verify the token from the server side. But I'm not a fan of this method since it needs time to call API (and another time if you also retrieve secret token for Clerk from the database secret)

Scheduling Job to Remind Sales Person with Upstash

TBA

Add Push Notification If a Deal is Closed with Ntfy

TBA

Automatic Monthly Report Send to Email with Mailgun

TBA

App and Audit Log for Debugging with Axiom

TBA

Conclusion

That's all my exploration with Outerbase command. It's not a complete app yet, in the next post we will see how I created a dashboard and plugin to make my father oversee his salesperson.

Thank you Hashnode and Outerbase for hosting a Hackathon. That makes me accelerate my development process to finish this dormant idea.

I usually talk about rapid development and love to review Platform as a Service that helps developers' lives easier. Connect with me on Twitter and Linkedin to talk more about that.