Using WebSockets for GPT response

Grand offers GPT models through WebSockets for realtime response.

#   WebSocket API Concepts

Here are a few concepts you should know for using Grand via WebSockets:

1. You need a unique URL to connect. You can get that URL by calling our /socket/create endpoint.
2. That URL is pre-signed, which means you do not need to pass any keys or secrets along for connection. That also means that you should not share it publicly or anyone would be able connect and use it on your expense.
3. Once started, a connection is only valid for 1 hour (3600 seconds) at the most. When it expires, get the connection URL again.
4. All the messages sent to, and received from Grand will be in the JSON format.

Here's how you can connect to Grand via WebSocket. Use our /socket/create endpoint to get a WebSocket connection url. Afterwards, you can simply connect like this:

const connection = new WebSocket(url)

  // listen to the response from Grand
  connection.onmessage = (msg) => {
    try {
      let message = JSON.parse(msg.data)
      console.log('from Grand:', message)
    } catch (error) { // probably due to some network issues
      console.log('JSON parsing error:', error)
    }
  }

#   WebSocket Requests

Send stringified JSON data as a request. Following is the expected schema:

{
    "action": "<from actions list>",
    "request": "<request body>"
  }

At the moment, Grand supports following actions through WebSockets:

1. Generate: action: 'generate'
2. Use Playbook: action: 'playbook-use'

Making a request through a socket connection is as simple as sending JSON data through connection.send method. Here's an example of generating text:

connection.send(JSON.stringify({
    action: "generate",
    request: {
      stop: ".", // required
      creativity: 40, // required
      text: "Grand API is fast, easy and" // required
    }
  }))

👋🏼 request expects the same parameters as listed on the Server APIs page. For example, when using generate action, you can pass the parameters as listed here.

#   WebSocket Response

Grand will keep streaming the next words/tokens until the stop is found or max limit is reached. As mentioned above, the response will also be in the JSON format. From our onmessage listener above, we'll get something like this in the parsed variable:

{
    ok: true, // if the response is valid and not an error
    done: false, // if the response is finished
    id: "19eeff4e", // request id
    data: {
      text: " secure" // next word/token
    }
  }

A new message is sent for each word/token. A standard practice is to keep using the data.text property unless you find done to be true, or ok to be false.

#   Error codes

Folowing errors are expected from a WebSocket connection. When ok is false, following error strings are expected in reason property. Errors not listed below will be related to the related action. The reason property should be self-explanatory in that case.

#   Guide

GPT-J is one of the best models for generating text, among other things. We already have a GPT-J API for conventional request/response usage, this guide will cover using GPT-J model for text generation over WebSockets. If you don't already know, here's a quick refresher on using WebSocket.

WebSockets provide asynchronous form of communication, which means we send and receive data differently than the regular, synchronous, HTTP requests.

Common HTTP request example:

fetch(url, request).then(response => {
    // do something with the response
  })

WebSocket request example:

const connection = new WebSocket(url)

  // first add a listener
  connection.onmessage = (response) => {
    // do something with the response
  }

  // then
  connection.send(request)

To get started, first thing we're going to need is a URL for WebSocket connection. We'll send a POST request to the /socket/create endpoint to get that.

You can use a tool like Insomnia or Postman but we'll just cURL it for the guide.

curl -XPOST -H 'x-auth-key: YOUR KEY' -H 'x-auth-secret: YOUR SECRET' -H "Content-type: application/json" -d '{}' 'http://api.grand.app/socket/create'

{
    "ok": true,
    "ttl": 3600,
    "id": "vf93jf001",
    "url": "wss://api.grand.app/connect/6W3TjJFKID8290mnTLo/vf93jf001"
  }

What we care about is the url property in that response. Once we have it, we can use it to connect to Grand WebSocket server:

// use the URL from /socket/create response.
    const connection = new WebSocket(response.url)

    // Important:
    //    We need to start listening as soon as possible
    //    connection errors will also be received here
    connection.onmessage = (msg) => {
      try {
        let message = JSON.parse(msg.data)
        console.log('from Grand:', message)
      } catch (error) { // probably due to some network issues
        console.log('JSON parsing error:', error)
      }
    }

In the above code can be used on the client-side to connect.

As you can see, we've added a onmessage listener to receive messages from Grand. We're using try/catch because of the JSON parsing; Grand will always return JSON data but sometimes, due to a network issue or something, it may not be and JSON.parse will throw an error.

We'll also add a couple of other listeners to keep track of the connection state:

// keep track of the connection
    let isConnected = false

    connection.onopen = () => {
      isConnected = true
      console.log('connection opened')
    }

    connection.onerror = (error) => {
      isConnected = false
      console.log('connection closed due to an error', error)
    }

    connection.onclose = () => {
      isConnected = false
      console.log('connection closed')
    }

With the listeners in place, we're ready to start sending some requests. We'll send a generate request for open-ended text generation.

const requestBody = {
      action: "generate", // it could also be 'playbook-use'
      request: {
        stop: ".", // when to stop the generation
        creativity: 40, // how creative should the response be
        text: "Grand API is fast, easy and" // initial text
      }
    }

    if (isConnected) {
      connection.send(JSON.stringify(requestBody)) // stringify is important here
    } else {
      console.log('No connection found')
    }

When ready, Grand will start sending data back and we'll get that in onmessage listener. We can change the listener a little bit to read the text that Grand is generating for us:

function handleMessage (message) {
      // check if the message is an error or a valid response
      if (!message.ok) {
        console.log('Error from Grand:', message.reason)

        if (['connection_expired', 'connection_closing'].includes(message.reason)) {
          // get a new connection URL
          console.log('connection expired')
        }

        return
      }

      // the text being sent by Grand in realtime
      console.log(message.data.text)

      // if done = true, there'll be no more messages for this request
      if (message.done) {
        console.log('Response finished!')
      }
    }

    connection.onmessage = (msg) => {
      try {
        let message = JSON.parse(msg.data)
        handleMessage(message)
      } catch (error) { // probably due to some network issues
        console.log('JSON parsing error:', error)
      }
    }

The comments in the code above are self-explanatory but just to summarize:

• Check the .ok property; if false, check .reason property too. If it's connection_expired or connection_closing, get the connection URL again.
• Use .data.text if everything is fine
• Check if .done property is true. If so, response is finished for that perticular request.
• You rock!

We hope this guide will help you implement Grand into your application. If you face any issues or have some question, please feel free to reach out to us through chat.