Lumos provides a way to manage hundreds of apps for a company. To build a platform across hundreds of different applications with different use cases, data models, and API protocols, we have a standard connector interface into which we translate all the app APIs.

We have a connector SDK to make it easier to build these connectors.

https://pypi.org/project/connector-py/

Terminology

connector: A package or program Lumos can run, to read data from and write to a third party app.
custom connector: A connector that is built and operated by one of Lumos’ customers; usually to talk to their internally built apps, or apps that aren’t accessible via the public internet.
capability: a standardized Lumos API on a connector. Example: list_accounts returns a list of app user accounts, in standardized Lumos Account format, and enough data to request the next page if applicable. See Connector capabilities for details.
On-premise (aka on prem): a way for a customer to run a Lumos connector on their own hardware, in their own network.
SDK: Our set of libraries for making it easier to build connectors. Currently only in Python.
account: The Lumos term for a user or service account in an underlying app
entitlement: Membership in a group, assignment to a role, or some other access-granting permission in the underlying app. Usually something assignable directly to an account.
resource: Something in an underlying app that an account can have an entitlement to. In the case of a global entitlement, like a tenant-wide role of Admin, there may be no resource.

How Lumos connectors are used in production

Connectors describe themselves

When Lumos discovers a new connector, our first job is to learn what the connector supports, and how to call it. We do this via the info capability.

Info tells us…

A name, logo, and description of the app in Lumos
How to authenticate against the underlying app
What settings are available or required for the connector
What capabilities the connector supports
What resource and entitlement types the connector provides access to
An app ID

Part of theinfo response

{
  "response": {
	  "user_friendly_name": "Pretend App",
	  "description": "A pretend app for these examples",
	  "logo_url": "https://some-logo-hosting-site.com/pretend_app",
	
	  "authentication_schema": { ... }
	  "request_settings_schema": { ... },
	
	  "capabilities": [ ... ],
	
	  "resource_types": [ ... ],
	  "entitlement_types": [ ... ],
	
	  "app_id": "pretend_app",
	}
}

For more details on how connectors self-describe, see Connector capabilities

The end-user configures a connector

Connectors have two avenues for configuration: auth and settings. These are separate keys on the request envelope, with slightly different semantics.

The flow here is

Connector Discovery: The connector describes what kind of auth and non-auth settings are required or available in the info response.
Customer configuration: Lumos uses the settings schema and auth type to render a webform for the customer IT admin, to allow them to establish the connection with a specific integration tenant.
Connector execution: Each call to a connector capability includes the auth + settings collected in the Customer Configuration phase.

Auth

Lumos assumes that all connectors require some credential materials to successfully talk to the underlying app.

Token and Basic credentials require the user to pass one or two credentials, respectively, in the auth section of the request envelope.

OAuth requires an access token for most capabilities, and requires a variety of different request credentials for OAuth-specific capabilities. Example: to refresh the access token, you must call the connector with the OAuth refresh token.

The auth model for a connector is declared in the "authentication_schema" field in the Info response.

To learn more about using OAuth in the connector, see the section on OAuth capabilities .
To see how the auth settings are passed to the caller, see the Info response .

Settings

There are non-sensitive (usually) settings fields that a connector may require to connect to the underlying app, or may optionally accept to adjust behavior.

Some examples might be

The specific DNS name to use to connect to the underlying service.
Which role to assume when connecting to the service.
Whether to include groups when reading entitlements.

The settings for a connector are defined as a JSON schema in the "request_settings_schema" field in the Info response.

Connectors are “connected” when their credentials are validated

A specific capability, validate_credentials, is the last step after a customer has configured and authenticated the connector, to ensure Lumos can call other capabilities successfully.

Lumos connector capabilities

Capabilities can be called via Python API or CLI

When running connectors inside Lumos, we import and run them directly in a Python VM.

from pretend_app import integration

await integration.dispatch(CapabilityName.LIST_ACCOUNTS, '{"request":{}}')

When running them as in an on-prem configuration, we call them via CLI.

pretend_app list_accounts --json '{"request":{}}'

Capabilities are invoked with request and response envelopes

The top-level JSON objects passed to and from capabilities are considered envelopes. Inside the envelope is the capability-specific data, at "request" and "response" fields respectively.

Other notable fields in these envelopes are

"auth" (see next section)
"settings" (see next section)
"page" (see section “Pagination”)

Types of capabilities

Capabilities can be roughly clustered into four sections.

Pagination

Read capabilities can read large sets of data; sometimes in the hundreds of thousands or millions of rows, e.g. associations between users and groups.

Read capabilities are all expected to support a "page" field on their requests and responses. A response can have a "token" string attached; if it is, that token can be used by the caller on another request to get the next page of results.

Deploying Lumos connectors

There are two models for deploying connectors that Lumos can talk to; Lumos-hosted and Customer-hosted (on-prem) connectors.

The connector SDK and connector capabilities are the same for different hosting models. The biggest difference is how the request for the capability gets to the connector, and how the response gets back to Lumos.

The biggest difference, from the connector’s perspective, is that it gets called via CLI instead of directly loaded in a running Python process.

How to run an on-prem agent locally

See On-Premise Agent for details on running an on-prem agent on WIndows or Linux.

Best Practises

Make use of the scaffolding and prepared class systems
When available, feel free to use external SDKs and libraries to connect with the end system
Connectors are idempotent by nature, and simple by design. Make sure you are within the context of a capability and only modifying necessary parameters for the capability to function.
Define your entitlements and resources clearly
Utilize pagination and consider using it as a minimal state store
Use pydantic for all models and DTOs, due to the describable nature of the connectors it is perfect for validation and documentation
Explore and make use of the SDKs utilities and helpers
Read the entire guide before beginning development

Troubleshooting Tips

Test out each capability individually
1. You can call the capability through HTTP or CLI, make sure to set a proper log level if using the latter.
Check agent logs for errors
Run the capability through Swagger UI / Postman
Consult AI on the application docs and code to check problems
1. One of the great ways to provide context to an LLM is exporting the OpenAPI Specification of your connector, either using the /docs endpoint, or by calling abc app_info --json '{"request": {}}' and copying the schema in response.
2. Provide additional context from Lumos's developer documentation for the areas that are causing problems