Skip to content

Commit e0cb2af

Browse files
author
Zhe Yu
committed
docs: Document database configuration and connector development
1 parent d134ae2 commit e0cb2af

3 files changed

Lines changed: 114 additions & 71 deletions

File tree

‎docs/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,7 @@ the configs.
264264
The JSON configuration file may hold the following values:
265265
- `db_type`: string, default: `"ChromaDB0Connector"`, the database backend to use;
266266
- `db_params`: dictionary. See
267-
[the corresponding database source code](../src/vectorcode/database/) for the
267+
[the database connector documentation](../src/vectorcode/database/README.md) for the
268268
default values;
269269
- `embedding_function`: string, one of the embedding functions supported by [Chromadb](https://www.trychroma.com/)
270270
(find more [here](https://docs.trychroma.com/docs/embeddings/embedding-functions) and

‎src/vectorcode/database/DEVELOPERS.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Database Connectors
2+
3+
A database connector is a compatibility layer that converts data structures used by a
4+
database to the ones that VectorCode works with.
5+
The connector classes provide abstractions for VectorCode operations (`vectorise`, `query`, etc.), which enables the use of different database backends.
6+
7+
<!-- mtoc-start -->
8+
9+
* [Adding a New Database Connector](#adding-a-new-database-connector)
10+
* [Key Implementation Details](#key-implementation-details)
11+
* [The `Config` Object](#the-config-object)
12+
* [Implementing Abstract Methods](#implementing-abstract-methods)
13+
* [Error Handling](#error-handling)
14+
* [Testing](#testing)
15+
16+
<!-- mtoc-end -->
17+
18+
## Adding a New Database Connector
19+
20+
To add support for a new database backend, you will need to:
21+
22+
1. **Implement a connector class**: Create a new file in this directory and implement a child class of `vectorcode.database.base.DatabaseConnectorBase`. You must implement all of its abstract methods.
23+
2. **Write tests**: Add tests for your new connector in the `tests/database/` directory. The tests should mock the database's API and verify that your connector correctly converts data between the database's native format and VectorCode's data structures.
24+
3. **Register your connector**: Add a new entry in the `get_database_connector` function in `src/vectorcode/database/__init__.py` to initialize your new connector.
25+
26+
For a concrete example, refer to the implementation of `DatabaseConnectorBase` and the `ChromaDB0Connector`.
27+
28+
## Key Implementation Details
29+
30+
### The `Config` Object
31+
32+
All settings for a connector are passed through a single `vectorcode.cli_utils.Config` object, which is available as `self._configs`. This includes:
33+
34+
- **Database Settings**: The `db_type` string and `db_params` dictionary are used to configure the connection to the database backend. As a contributor, you should document the specific `db_params` your connector requires in the class's docstring.
35+
- **Operation Parameters**: Parameters for operations like `query` or `vectorise` are also present in this object.
36+
37+
The `self._configs` attribute is mutable and can be updated for subsequent operations, but the database connection settings (`db_type`, `db_params`) should not be changed after initialization.
38+
39+
### Implementing Abstract Methods
40+
41+
When implementing the abstract methods from `DatabaseConnectorBase`, you should:
42+
43+
- Read the necessary parameters from the `self._configs` object.
44+
- Perform the corresponding operation against the database.
45+
- Return data in the format specified by the method's type hints (e.g., `QueryResult`, `CollectionInfo`).
46+
47+
**Please refer to the docstrings in `DatabaseConnectorBase` for the specific API contract of each method.**
48+
They contain detailed information about what each method is expected to do and what parameters it uses from the `Config` object.
49+
50+
There are also some helper methods (non-abstract methods) in `DatabaseConnectorBase` that may be
51+
helpful.
52+
For example, `self.get_embedding(texts)` provides a convenient way to get the embeddings of some strings that takes all config parameters into account (embedding function, embedding dimension, etc.).
53+
Using these methods helps keeping your implementation consistent with the overall design.
54+
55+
### Error Handling
56+
57+
Some exceptions raised by database backends are actually DB-agnostic (despite that each database may have different exception classes for them).
58+
For example, ChromaDB 0.6.3 have a `chromadb.errors.InvalidCollectionException` class, which is raised when accessing a collection that doesn't exist.
59+
This is not a chroma-specific error, but different database backends _probably_ have their own implementations.
60+
For better error handling, it's recommended to wrap them into [VectorCode's in-house exception classes](./errors.py).
61+
This ensures consistent error handling in the CLI and other clients.
62+
63+
For example:
64+
```python
65+
from vectorcode.database.errors import CollectionNotFoundError
66+
67+
try:
68+
some_action_here()
69+
except SomeCustomException as e:
70+
raise CollectionNotFoundError("The collection was not found.") from e
71+
```
72+
73+
## Testing
74+
75+
The unit tests for database backends should go under [`tests/database/`](../../../tests/database/).
76+
The tests should mock the request body and return values of the database. Integration
77+
tests that interact with an actual database are out of scope for now.
78+
79+
> The tests for the subcommands currently use mocked database connectors. They're not
80+
> supposed to interact with live databases.

‎src/vectorcode/database/README.md‎

Lines changed: 33 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,77 +1,40 @@
1-
# Database Connectors
1+
# Database Configuration
22

3-
A database connector is a compatibility layer that converts data structures that a
4-
database natively works with to the ones that VectorCode works with. The connector
5-
classes provide abstractions for VectorCode operations (`vectorise`, `query`, etc.),
6-
which enables the use of different database backends.
3+
This document provides the `db_params` configuration for the supported database connectors in VectorCode.
74

8-
<!-- mtoc-start -->
9-
10-
* [Adding a New Database Connector](#adding-a-new-database-connector)
11-
* [Key Implementation Details](#key-implementation-details)
12-
* [The `Config` Object](#the-config-object)
13-
* [Implementing Abstract Methods](#implementing-abstract-methods)
14-
* [Error Handling](#error-handling)
15-
* [Testing](#testing)
16-
17-
<!-- mtoc-end -->
18-
19-
# Adding a New Database Connector
20-
21-
To add support for a new database backend, you will need to:
22-
23-
1. **Implement a connector class**: Create a new file in this directory and implement a child class of `vectorcode.database.base.DatabaseConnectorBase`. You must implement all of its abstract methods.
24-
2. **Write tests**: Add tests for your new connector in the `tests/database/` directory. The tests should mock the database's API and verify that your connector correctly converts data between the database's native format and VectorCode's data structures.
25-
3. **Register your connector**: Add a new entry in the `get_database_connector` function in `src/vectorcode/database/__init__.py` to initialize your new connector.
26-
27-
For a concrete example, refer to the implementation of `DatabaseConnectorBase` and the `ChromaDB0Connector`.
28-
29-
# Key Implementation Details
30-
31-
## The `Config` Object
32-
33-
All settings for a connector are passed through a single `vectorcode.cli_utils.Config` object, which is available as `self._configs`. This includes:
5+
For instructions on how to add a new database connector, please refer to [DEVELOPERS.md](./DEVELOPERS.md).
346

35-
- **Database Settings**: The `db_type` string and `db_params` dictionary are used to configure the connection to the database backend. As a contributor, you should document the specific `db_params` your connector requires in the class's docstring.
36-
- **Operation Parameters**: Parameters for operations like `query` or `vectorise` are also present in this object.
377

38-
The `self._configs` attribute is mutable and can be updated for subsequent operations, but the database connection settings (`db_type`, `db_params`) should not be changed after initialization.
39-
40-
## Implementing Abstract Methods
41-
42-
When implementing the abstract methods from `DatabaseConnectorBase`, you should:
43-
44-
- Read the necessary parameters from the `self._configs` object.
45-
- Perform the corresponding operation against the database.
46-
- Return data in the format specified by the method's type hints (e.g., `QueryResult`, `CollectionInfo`).
47-
48-
**Please refer to the docstrings in `DatabaseConnectorBase` for the specific API contract of each method.**
49-
They contain detailed information about what each method is expected to do and what parameters it uses from the `Config` object.
50-
51-
There are also some helper methods (non-abstract methods) in `DatabaseConnectorBase` that may be
52-
helpful.
53-
For example, `self.get_embedding(texts)` provides a convenient way to get the embeddings of some strings that takes all config parameters into account (embedding function, embedding dimension, etc.).
54-
Using these methods helps keeping your implementation consistent with the overall design.
55-
56-
## Error Handling
57-
58-
If the underlying database library raises a specific exception (e.g., for a collection not being found), you should consider catching it and re-raise it as one of VectorCode's custom database exceptions from `vectorcode.database.errors`. This ensures consistent error handling in the CLI and other clients.
59-
60-
For example:
61-
```python
62-
from vectorcode.database.errors import CollectionNotFoundError
63-
64-
try:
65-
some_action_here()
66-
except SomeCustomException as e:
67-
raise CollectionNotFoundError("The collection was not found.") from e
68-
```
8+
<!-- mtoc-start -->
699

70-
# Testing
10+
* [ChromaDB (v0.6.3)](#chromadb-v063)
7111

72-
The unit tests for database backends should go under [`tests/database/`](../../../tests/database/).
73-
The tests should mock the request body and return values of the database. Integration
74-
tests that interact with an actual database are out of scope for now.
12+
<!-- mtoc-end -->
7513

76-
> The tests for the subcommands currently use mocked database connectors. They're not
77-
> supposed to interact with live databases.
14+
## ChromaDB (v0.6.3)
15+
16+
The `ChromaDB0Connector` is used for ChromaDB versions 0.6.3.
17+
18+
- **`db_type`**: `"ChromaDB0"`. The `Connector` suffix is optional and will be added automatically.
19+
20+
- **`db_params`**:
21+
An example of the `db_params` for `ChromaDB0Connector` in your `config.json5`:
22+
```json5
23+
{
24+
"db_params": {
25+
"db_url": "http://127.0.0.1:8000",
26+
"db_path": "~/.local/share/vectorcode/chromadb/",
27+
"db_log_path": "~/.local/share/vectorcode/",
28+
"db_settings": {},
29+
"hnsw": {
30+
"hnsw:M": 64,
31+
},
32+
},
33+
}
34+
```
35+
36+
- `db_url`: The URL of the ChromaDB server. Defaults to `"http://127.0.0.1:8000"`.
37+
- `db_path`: Path to the directory where ChromaDB stores its data. Defaults to `"~/.local/share/vectorcode/chromadb/"`.
38+
- `db_log_path`: Path to the directory for ChromaDB log files. Defaults to `"~/.local/share/vectorcode/"`.
39+
- `db_settings`: Additional ChromaDB settings. You usually don't need to touch this, but in case you do, you can refer to [ChromaDB source](https://github.com/chroma-core/chroma/blob/a3b86a0302a385350a8f092a5f89a2dcdebcf6be/chromadb/config.py#L101) for details. Defaults to `{}`.
40+
- `hnsw`: HNSW index parameters. Defaults to `{"hnsw:M": 64}`.

0 commit comments

Comments
 (0)