diff --git a/src/content/docs/snowflake/capabilities/accounts.md b/src/content/docs/snowflake/capabilities/accounts.md new file mode 100644 index 00000000..dfd588a7 --- /dev/null +++ b/src/content/docs/snowflake/capabilities/accounts.md @@ -0,0 +1,70 @@ +--- +title: Accounts +description: Get started with Accounts in LocalStack for Snowflake +--- + +## Introduction + +An account is a unique identifier for a Snowflake instance within an organization. It acts as a container +for resources and operations related to data storage, processing, and management. + +The Snowflake emulator lets you connect to and manage resources in different accounts. + +## Getting Started + +This guide explains how to start and connect to the Snowflake emulator using specific accounts. + +You can specify any account name when connecting to the Snowflake emulator. If you don't, all resources +will be managed by the default `test` account. + +Depending on the Snowflake Driver you choose, you can pass `account` accordingly. + +### Connect using Snowflake Connection Object + +If the Snowflake driver provides a connection object, you can pass the `account` parameter in the connection object. + +Example using the Snowflake Connector for Python: + +```python +sf_conn_obj = sf.connect( + account="your_account", + # other parameters +) +``` + +Example using the NodeJS Driver for Snowflake: + +```javascript +var connection = snowflake.createConnection({ + account: "your_account", + // other parameters +}); +``` + +### Connect using Connection String + +You can also specify the account for Snowflake drivers that let you connect with a connection string. + +Example establishing a JDBC connection: + +``` +jdbc:snowflake://snowflake.localhost.localstack.cloud:4566/?account=your_account +``` + +### Check Current Account + +Once successfully connected, you can verify which account you are connected to by executing the following SQL command: + +```sql +SELECT CURRENT_ACCOUNT_NAME(); +``` + +The query statement will return the name of the account you are currently connected to. Output should look like: + +```sql ++------------------------------------------+ +| CURRENT_ACCOUNT_NAME() | +|------------------------------------------| +| YOUR_ACCOUNT | ++------------------------------------------+ +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/clones.md b/src/content/docs/snowflake/capabilities/clones.md new file mode 100644 index 00000000..6546423f --- /dev/null +++ b/src/content/docs/snowflake/capabilities/clones.md @@ -0,0 +1,113 @@ +--- +title: Clones +description: Get started with Clones in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Cloning in Snowflake allows you to create a quick, zero-copy duplicate of an existing database, schema, or table. This feature enables users to replicate data structures and content for testing or development without duplicating the underlying storage. + +The Snowflake emulator supports database cloning, enabling you to create quick duplicates of databases, schemas, or tables. Currently, [`CREATE ... CLONE`](https://docs.snowflake.com/en/sql-reference/sql/create-clone) is supported by LocalStack. + +## Getting started + +This guide assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client to execute the queries below. + +The following sections guide you through creating a database, inserting data into a table, and then cloning the database to verify data integrity. + +### Create a Database + +The following SQL snippet demonstrates how to create a database named `test_db`. + +```sql +CREATE DATABASE test_db;` +``` + +The expected output is: + +```sql ++--------------------------------------------+ +| status | +|--------------------------------------------| +| Database TEST_DB successfully created. | ++--------------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.123s +``` + +### Create a Table + +Once the database is created, you can create a table within it. The following SQL statement demonstrates how to create a table named `test_table` with columns for `id` and `name`. + +```sql +CREATE TABLE test_table (id INT, name TEXT);` +``` + +The expected output is: + +```sql ++----------------------------------------+ +| status | +|----------------------------------------| +| Table TEST_TABLE successfully created. | ++----------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.067s +``` + +### Insert Data + +To insert data into the `test_table`, use the `INSERT INTO` statement. This example inserts a single row with the values `(1, 'test')`. + +```sql +INSERT INTO test_table VALUES (1, 'test'); +``` + +The expected output is: + +```sql ++----------------------------------------+ +| status | +|----------------------------------------| +| 1 Row(s) inserted. | ++----------------------------------------+ +1 Row(s) produced. Time Elapsed: 0.024s +``` + +### Create a Clone + +With data now in `test_table`, you can create a clone of the entire `test_db` database. This will produce a new database, `test_db_clone`, containing all objects and data from the original `test_db`. + +```sql +CREATE DATABASE test_db_clone CLONE test_db; +``` + +The expected output is: + +```sql ++--------------------------------------------+ +| status | +|--------------------------------------------| +| Database TEST_DB_CLONE successfully created. | ++--------------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.101s +``` + +### Verify Data in Clone + +To confirm that the data has been cloned, query the `test_table` in the `test_db_clone` database. + +```sql +SELECT * FROM test_db_clone.test_table; +``` + +The expected output is: + +```sql ++----+------+ +| id | name | +|----|------| +| 1 | test | ++----+------+ +1 Row(s) produced. Time Elapsed: 0.012s +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/config.md b/src/content/docs/snowflake/capabilities/config.md new file mode 100644 index 00000000..98806084 --- /dev/null +++ b/src/content/docs/snowflake/capabilities/config.md @@ -0,0 +1,113 @@ +--- +title: Configuration +description: Overview of configuration options in LocalStack for Snowflake. +template: doc +nav: +label: +--- + + +{{< preview-notice >}} + +LocalStack exposes various configuration options to control its behaviour. + +These options can be passed to LocalStack as environment variables like so: + +{{< command >}} +$ DEBUG=1 localstack start +{{< / command >}} + +## Core + +Options that affect the core Snowflake emulator functionality. + +| Variable | Example Values | Description | +|----------|----------------------|-------------------------------------------------------------------------------------------------------------| +| `DEBUG` | `0` (default) \| `1` | Flag to increase log level and print more verbose logs (useful for troubleshooting issues) | +| `SF_LOG` | `trace` | Specify the log level. Currently overrides the `DEBUG` configuration. `trace` for detailed request/response | +| `SF_S3_ENDPOINT` | `s3.localhost.localstack.cloud:4566` (default) | Specify the S3 endpoint to use for the Snowflake emulator. | +| `DNS_NAME_PATTERNS_TO_RESOLVE_UPSTREAM` | `*.s3.amazonaws.com` (example) | List of domain names that should NOT be resolved to the LocalStack container, but instead always forwarded to the upstream resolver (S3 for example). this would be required when importing data into a stage from an external S3 bucket on the real AWS cloud. Comma-separated list of Python-flavored regex patterns. | +| `SF_HOSTNAME_REGEX` | `snowflake\..+` (default) | Allows you to customize the hostname used for matching the Snowflake API routes in the HTTP router. If not set, then it matches on any hostnames that contain a `snowflake.*` subdomain (e.g., `snowflake.localhost.localstack.cloud`). | +| `SF_CSV_IMPORT_MAX_ROWS` | `50000` (default) | Maximum number of rows to import from CSV files into tables | +| `SF_DEFAULT_USER` | `test` (default) | Specify the default user to be used by the Snowflake emulator. | +| `SF_DEFAULT_PASSWORD` | `test` (default) | Specify the default password to be used by the Snowflake emulator. | + +## CLI + +These options are applicable when using the CLI to start LocalStack. + +| Variable | Example Values | Description | +| - | - | - | +| `LOCALSTACK_VOLUME_DIR` | `~/.cache/localstack/volume` (on Linux) | The location on the host of the LocalStack volume directory mount. | +| `CONFIG_PROFILE` | | The configuration profile to load. See [Profiles]({{< ref "#profiles" >}}) | +| `CONFIG_DIR` | `~/.localstack` | The path where LocalStack can find configuration profiles and other CLI-specific configuration | + +## Docker + +Options to configure how LocalStack interacts with Docker. + +| Variable | Example Values | Description | +| - | - | - | +| `DOCKER_FLAGS` | | Allows to pass custom flags (e.g., volume mounts) to "docker run" when running LocalStack in Docker. | +| `DOCKER_SOCK` | `/var/run/docker.sock` | Path to local Docker UNIX domain socket | +| `DOCKER_BRIDGE_IP` | `172.17.0.1` | IP of the Docker bridge used to enable access between containers | +| `LEGACY_DOCKER_CLIENT` | `0`\|`1` | Whether LocalStack should use the command-line Docker client and subprocess execution to run Docker commands, rather than the Docker SDK. | +| `DOCKER_CMD` | `docker` (default), `sudo docker`| Shell command used to run Docker containers (only used in combination with `LEGACY_DOCKER_CLIENT`) | +| `FORCE_NONINTERACTIVE` | | When running with Docker, disables the `--interactive` and `--tty` flags. Useful when running headless. | + +## Proxy Configuration + +Options to configure Snowflake proxy settings for LocalStack. + +| Variable | Example Values | Description | +| - | - | - | +| `SF_PROXY_HOST` | `proxy.example.com` | Hostname or IP address of the proxy server to be used for connecting to Snowflake. | +| `SF_PROXY_USER` | `proxy_user` | Username for authentication with the proxy server. | +| `SF_PROXY_PASSWORD` | `password123` | Password associated with the proxy user account. | + +## Profiles + +LocalStack supports configuration profiles which are stored in the `~/.localstack` config directory. +A configuration profile is a set of environment variables stored in a `*.env` file in the LocalStack config directory. + +Here is an example of what configuration profiles might look like: + +{{< command >}} +$ tree ~/.localstack +/home/username/.localstack +├── default.env +├── dev.env +└── pro.env +{{< / command >}} + +Here is an example of what a specific environment profile looks like + +{{< command >}} +$ cat ~/.localstack/pro-debug.env +LOCALSTACK_AUTH_TOKEN=XXXXX +SF_LOG=trace +SF_S3_ENDPOINT=s3.localhost.localstack.cloud:4566 +{{< / command >}} + +You can load a profile by either setting the environment variable `CONFIG_PROFILE=` or the `--profile=` CLI flag when using the CLI. +Let's take an example to load the `dev.env` profile file if it exists: + +{{< command >}} +$ IMAGE_NAME=localstack/snowflake localstack --profile=dev start +{{< / command >}} + +If no profile is specified, the `default.env` profile will be loaded. +If explicitly specified, any environment variables will overwrite the configurations defined in the profile. + +To display the config environment variables, you can use the following command: + +{{< command >}} +$ localstack --profile=dev config show +{{< / command >}} + +{{< alert title="Note" >}} +The `CONFIG_PROFILE` is a CLI feature and cannot be used directly with a docker-compose setup. +You can look at [alternative means of setting environment variables](https://docs.docker.com/compose/environment-variables/set-environment-variables/) for your Docker Compose setups. +For Docker setups, we recommend passing the environment variables directly to the `docker run` command. +{{< /alert >}} + diff --git a/src/content/docs/snowflake/capabilities/ephemeral-instances.md b/src/content/docs/snowflake/capabilities/ephemeral-instances.md new file mode 100644 index 00000000..849c8d3a --- /dev/null +++ b/src/content/docs/snowflake/capabilities/ephemeral-instances.md @@ -0,0 +1,29 @@ +--- +title: Ephemeral Instances +description: Get started with Ephemeral Instances in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Ephemeral Instances allows you to run a LocalStack for Snowflake instance in the cloud. You can interact with these instances by configuring your Snowflake host with the endpoint URL of the ephemeral instance. + +{{< alert title="Note" >}} +Ephemeral Instances is offered as a **preview** feature and under active development. +{{< /alert >}} + +## Getting started + +Navigate to the [LocalStack Ephemeral Instance Management](https://app.localstack.cloud/instances/ephemeral) page. In the form, enter the name of the new Ephemeral Instance, choose **LocalStack Snowflake (preview)**, select the lifetime of the instance by dragging the slider, and click on **Launch**. + +Creating an Ephemeral Instance + +After the ephemeral instance is created, you can run the following command to verify the status of the instance: + +{{< command >}} +$ export SNOWFLAKE_HOST= +$ curl -d '{}' $SNOWFLAKE_HOST/session +{{< /command >}} + +You can access the Ephemeral Instance via the SnowSQL or any alternative SQL client by configuring the Snowflake host with the endpoint URL of the Ephemeral Instance. \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/index.md b/src/content/docs/snowflake/capabilities/index.md new file mode 100644 index 00000000..3ad41178 --- /dev/null +++ b/src/content/docs/snowflake/capabilities/index.md @@ -0,0 +1,9 @@ +--- +title: Capabilities +description: Capabilities +template: doc +nav: +label: +--- + +# Capabilities \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/init-hooks.md b/src/content/docs/snowflake/capabilities/init-hooks.md new file mode 100644 index 00000000..6340a42a --- /dev/null +++ b/src/content/docs/snowflake/capabilities/init-hooks.md @@ -0,0 +1,64 @@ +--- +title: Initialization Hooks +description: Writing SQL scripts to initialize your Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +LocalStack for Snowflake supports automatically executing `*.sf.sql` files via [Init Hooks](https://docs.localstack.cloud/references/init-hooks/) when mounted into the Docker container. A script can be added to one of these stages in the lifecycle: + +- `BOOT`: the container is running, but LocalStack hasn’t started +- `START`: the Python process is running, and LocalStack is starting +- `READY`: LocalStack is ready for requests +- `SHUTDOWN`: LocalStack is shutting down + +A script can be in one of four states: `UNKNOWN`, `RUNNING`, `SUCCESSFUL`, or `ERROR`. By default, scripts are in the `UNKNOWN` state when first discovered. + +## Getting started + +To begin, create a script called `test.sf.sql` with the following SQL statements: + +```sql +CREATE DATABASE foobar123; +CREATE DATABASE test123; +SHOW DATABASES; +``` + +Mount the script into `/etc/localstack/init/ready.d/` using Docker Compose or the `localstack` CLI: + +{{< tabpane >}} +{{< tab header="docker-compose.yml" lang="yml" >}} +version: "3.8" + +services: + localstack: + container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}" + image: localstack/snowflake + ports: + - "127.0.0.1:4566:4566" + - "127.0.0.1:4510-4559:4510-4559" + - "127.0.0.1:443:443" + environment: + - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?} + - DEBUG=1 + volumes: + - "/path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql" # ready hook + - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack" + - "/var/run/docker.sock:/var/run/docker.sock" +{{< /tab >}} +{{< tab header="CLI" lang="bash" >}} +# DOCKER_FLAGS are additional parameters to the `docker run` command of localstack start + +DOCKER_FLAGS='-v /path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql' DEBUG=1 localstack start +{{< /tab >}} +{{< /tabpane >}} + +Start the Snowflake emulator, and the following logs will appear: + +```bash +DEBUG --- [et.reactor-0] s.analytics.handler : REQ: POST /queries/v1/query-request {"sqlText": "CREATE DATABASE foobar123", ... +DEBUG --- [et.reactor-0] s.analytics.handler : REQ: POST /queries/v1/query-request {"sqlText": "CREATE DATABASE test123", ... +DEBUG --- [et.reactor-0] s.analytics.handler : REQ: POST /queries/v1/query-request {"sqlText": "SHOW DATABASES", ... +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/state-management.md b/src/content/docs/snowflake/capabilities/state-management.md new file mode 100644 index 00000000..065b541c --- /dev/null +++ b/src/content/docs/snowflake/capabilities/state-management.md @@ -0,0 +1,104 @@ +--- +title: State Management +description: Get started with State Management in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +State Management in LocalStack allows you to save and load the state of your LocalStack instance. LocalStack is ephemeral in nature, so when you stop and restart your LocalStack instance, all the data is lost. With State Management, you can save the state of your LocalStack instance and load it back when you restart your LocalStack instance. + +State Management in LocalStack encompasses the following features: + +- **Cloud Pods**: Cloud Pods are persistent state snapshots of your LocalStack instance that can easily be shared, stored, versioned, and restored. +- **Export & Import State**: Export and import the state of your LocalStack instance on your local machine as a local file. +- **Persistence**: Persist the state of your LocalStack instance on your local machine using a configuration variable. + +State Management is an essential feature that supports various use-cases, such as pre-seeding your fresh LocalStack instance with data, sharing your LocalStack instance’s state with your team, fostering collaboration, and more. + +## Persistence + +LocalStack’s Persistence mechanism enables the saving and restoration of the entire LocalStack state. It functions as a **pause and resume** feature, allowing you to take a snapshot of your LocalStack instance and save this data to disk. This mechanism ensures a quick and efficient way to preserve and continue your work with Snowflake resources locally. + +To start snapshot-based persistence, launch LocalStack with the configuration option `PERSISTENCE=1`. This setting instructs LocalStack to save all local Snowflake resources and their respective application states into the LocalStack Volume Directory. Upon restarting LocalStack, you'll be able to resume your activities exactly where you left off. + +{{< tabpane >}} +{{< tab header="LocalStack CLI" lang="bash" >}} +LOCALSTACK_AUTH_TOKEN=... PERSISTENCE=1 IMAGE_NAME=localstack/snowflake localstack start +{{< /tab >}} +{{< tab header="Docker Compose" lang="yaml" >}} + ... + image: localstack/snowflake + environment: + - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?} + - PERSISTENCE=1 + volumes: + - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack" +{{< /tab >}} +{{< tab header="Docker" lang="bash" >}} +docker run \ + -e LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?} \ + -e PERSISTENCE=1 \ + -v ./volume:/var/lib/localstack \ + -p 4566:4566 \ + localstack/snowflake +{{< /tab >}} +{{< /tabpane >}} + +{{< alert title="Note">}} +Snapshots may not be compatible across different versions of LocalStack. +It is possible that snapshots from older versions can be restored, but there are no guarantees as to whether LocalStack will start into a consistent state. +We are actively working on a solution for this problem. +{{< /alert >}} + +## Export/Import State + +The Export/Import State feature enables you to export the state of your LocalStack instance into a file, and then later import it into another LocalStack instance. This feature is useful when you want to save your LocalStack instance’s state for later use. + +### Export the State + +To export the state, you can run the following command: + +{{< command >}} +$ localstack state export '' +{{< /command >}} + +You can use the `` argument to specify a file path to export the state to. If you do not specify a file path, the state will be exported to the current working directory into a file named `ls-state-export`. + +### Import the State + +To import the state, you can run the following command: + +{{< command >}} +$ localstack state import '' +{{< /command >}} + +The `` argument is required and specifies the file path to import the state from. The file should be generated from a previous export. + +## Cloud Pods + +Cloud pods are persistent state snapshots of your LocalStack instance that can easily be stored, versioned, shared, and restored. Cloud Pods can be used for various purposes, such as: + +- Save and manage snapshots of active LocalStack instances. +- Share state snapshots with your team to debug collectively. +- Automate your testing pipelines by pre-seeding CI environments. +- Create reproducible development and testing environments locally. + +You can save and load the persistent state of Cloud Pods, using the Cloud Pods CLI. LocalStack provides a remote storage backend that can be used to store the state of your running application and share it with your team members. Cloud Pods CLI is included in the [LocalStack CLI installation](https://docs.localstack.cloud/getting-started/installation/#localstack-cli), so there's no need for additional installations to begin using it. + +### Create a new Cloud Pod + +To create the Cloud Pod, you can run the following command: + +{{< command >}} +$ localstack pod save '' +{{< /command >}} + +### Load an existing Cloud Pod + +To load the Cloud Pod, you can run the following command: + +{{< command >}} +$ localstack pod load '' +{{< /command >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/storage-integrations.md b/src/content/docs/snowflake/capabilities/storage-integrations.md new file mode 100644 index 00000000..943f933a --- /dev/null +++ b/src/content/docs/snowflake/capabilities/storage-integrations.md @@ -0,0 +1,117 @@ +--- +title: Storage Integrations +description: Get started with Storage Integrations in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake storage integrations enable access to external cloud storage like Amazon S3, Google Cloud Storage, and Azure Blob Storage. They manage authentication through generated IAM roles, enhancing security and simplifying data operations without exposing sensitive credentials. This approach centralizes and controls access, streamlining workflows across major cloud platforms. + +The Snowflake emulator supports storage integrations, allowing you to test interactions with external storage using the same commands and syntax as the Snowflake service. The following operations are supported: + +- [`CREATE STORAGE INTEGRATION`](https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration) +- [`DESCRIBE STORAGE INTEGRATION`](https://docs.snowflake.com/en/sql-reference/sql/describe-storage-integration) +- [`DROP STORAGE INTEGRATION`](https://docs.snowflake.com/en/sql-reference/sql/drop-storage-integration) + +## Getting started + +This guide is designed for users new to Storage Integration and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a Snowflake Storage Integration with Amazon S3 and creating an external stage to load data. + +### Create an S3 bucket + +You can create a local S3 bucket using the `mb` command with the `awslocal` CLI. + +```bash +$ awslocal s3 mb s3://testbucket +``` + +Upload some sample CSV file into the S3 bucket using the following command: + +```bash +awslocal s3 cp file.csv s3://testbucket +``` + +### Create a Storage Integration + +You can now create a Storage Integration named `s_example` which will connect Snowflake to your S3 bucket using the following statement: + +```sql +CREATE STORAGE INTEGRATION s_example + TYPE = EXTERNAL_STAGE + ENABLED = TRUE + STORAGE_PROVIDER = 'S3' + STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::000000000000:role/testrole' + STORAGE_ALLOWED_LOCATIONS = ('s3://testbucket'); +``` + +The expected output is: + +```sql ++---------------------------------------------+ +| ?COLUMN? | +|---------------------------------------------| +| Integration S_EXAMPLE successfully created. | ++---------------------------------------------+ +``` + +### Describe the Storage Integration + +After creating the storage integration, you can retrieve important configuration by running the following statement: + +```sql +DESCRIBE STORAGE INTEGRATION s_example; +``` + +The expected output is: + +```sql ++---------------------------+---------------+-----------------------------------------+------------------+ +| property | property_type | property_value | property_default | +|---------------------------+---------------+-----------------------------------------+------------------| +| ENABLED | Boolean | true | false | +| STORAGE_PROVIDER | String | S3 | | +| STORAGE_ALLOWED_LOCATIONS | List | s3://testbucket | [] | +| STORAGE_BLOCKED_LOCATIONS | List | | [] | +| STORAGE_AWS_IAM_USER_ARN | String | arn:aws:iam::000000000000:user/test | | +| STORAGE_AWS_ROLE_ARN | String | arn:aws:iam::000000000000:role/testrole | | +| STORAGE_AWS_EXTERNAL_ID | String | TEST_SFCRole=test | | +| COMMENT | String | | | ++---------------------------+---------------+-----------------------------------------+------------------+ +8 Row(s) produced. Time Elapsed: 0.050s +``` + +### Create a stage + +You can now create an external stage using the following statement: + +```sql +CREATE STAGE stage_example + STORAGE_INTEGRATION = s_example + URL = 's3://testbucket' + FILE_FORMAT = (TYPE = CSV); +``` + +The expected output is: + +```sql ++------------------------------------------------+ +| ?COLUMN? | +|------------------------------------------------| +| Stage area STAGE_EXAMPLE successfully created. | ++------------------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.083s +``` + +### List the files + +To list the files in the stage, you can run the following statement: + +```sql +LIST @stage_example; +``` + +The output will show the `files.csv` file that we uploaded earlier to the S3 bucket. \ No newline at end of file diff --git a/src/content/docs/snowflake/capabilities/tags.md b/src/content/docs/snowflake/capabilities/tags.md new file mode 100644 index 00000000..9123016a --- /dev/null +++ b/src/content/docs/snowflake/capabilities/tags.md @@ -0,0 +1,119 @@ +--- +title: Tags +description: Get started with Tags in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake tags allow you to categorize and manage Snowflake objects by associating custom metadata with them. These tags support governance, cost tracking, and data lineage by enabling organizations to label resources with business-relevant information. + +The Snowflake emulator supports tags, allowing you to apply these tags to the local Snowflake tables, views, and databases using the same commands and syntax as the Snowflake service. The following operations are supported: + +- [`CREATE TAG`](https://docs.snowflake.com/en/sql-reference/sql/create-tag) +- [`SHOW TAGS`](https://docs.snowflake.com/en/sql-reference/sql/show-tags) +- [`ALTER TAG`](https://docs.snowflake.com/en/sql-reference/sql/alter-tag) +- [`DROP TAG`](https://docs.snowflake.com/en/sql-reference/sql/drop-tag) + +## Getting started + +This guide is designed for users new to tagging in Snowflake and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client to execute the queries below. + +The following sections guide you through an example of creating a database, adding a tag, and using the tag to track metadata in Snowflake. + +### Create a Database + +The following SQL snippet demonstrates how to create a database named `tag_test_db`. + +```sql +CREATE DATABASE IF NOT EXISTS tag_test_db; +``` + +The expected output is: + +```sql ++--------------------------------------------+ +| status | +|--------------------------------------------| +| Database TAG_TEST_DB successfully created. | ++--------------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.163s +``` + +### Create a Tag + +To create a tag, use the `CREATE TAG` statement. The following example demonstrates how to create a tag named `tag1`. + +```sql +CREATE TAG tag1; +``` + +The expected output is: + +```sql ++--------------------------------+ +| ?COLUMN? | +|--------------------------------| +| Tag TAG1 successfully created. | ++--------------------------------+ +0 Row(s) produced. Time Elapsed: 0.057s +``` + +### Assign Tag to Database + +To assign a tag to a database, use the `ALTER DATABASE` statement. The following example demonstrates how to assign the tag `tag1` with the value `'test 123'` to the `tag_test_db` database. + +```sql +ALTER DATABASE tag_test_db SET TAG tag1 = 'test 123'; +``` + +The expected output is: + +```sql ++----------------------------------+ +| ?column? | +|----------------------------------| +| Statement executed successfully. | ++----------------------------------+ +0 Row(s) produced. Time Elapsed: 0.026s +``` + +### Query Tag Value + +To retrieve the value of a tag assigned to a database, use the `SELECT SYSTEM$GET_TAG` statement. The following example demonstrates how to query the value of `tag1` assigned to the `tag_test_db` database. + +```sql +SELECT SYSTEM$GET_TAG('tag1', 'tag_test_db', 'database'); +``` + +The expected output is: + +```sql ++----------------+ +| SYSTEM$GET_TAG | +|----------------| +| test 123 | ++----------------+ +1 Row(s) produced. Time Elapsed: 0.565s +``` + +### Query Tag References + +To view all references of a tag within a database, use the `INFORMATION_SCHEMA.TAG_REFERENCES` function. The following example demonstrates how to query the `tag_test_db` database for references to the `tag1` tag. + +```sql +SELECT * +FROM TABLE(tag_test_db.INFORMATION_SCHEMA.TAG_REFERENCES('tag_test_db', 'database')); +``` + +The expected output is: + +```sql ++--------------+------------+----------+-----------+----------+-----------------+---------------+-------------+----------+-------------+ +| TAG_DATABASE | TAG_SCHEMA | TAG_NAME | TAG_VALUE | LEVEL | OBJECT_DATABASE | OBJECT_SCHEMA | OBJECT_NAME | DOMAIN | COLUMN_NAME | +|--------------+------------+----------+-----------+----------+-----------------+---------------+-------------+----------+-------------| +| TAG_TEST_DB | PUBLIC | TAG1 | test 123 | DATABASE | NULL | NULL | TAG_TEST_DB | DATABASE | NULL | ++--------------+------------+----------+-----------+----------+-----------------+---------------+-------------+----------+-------------+ +1 Row(s) produced. Time Elapsed: 0.528s +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/changelog.md b/src/content/docs/snowflake/changelog.md new file mode 100644 index 00000000..8a529d90 --- /dev/null +++ b/src/content/docs/snowflake/changelog.md @@ -0,0 +1,300 @@ +--- +title: Changelog +description: Changelog for the latest releases of the LocalStack for Snowflake. +template: doc +nav: +label: +--- + +{{< preview-notice >}} + +### 0.3.0 +- Add support for multi-account setups +- Add initial support for Java UDFs +- Add initial support for native apps and `APPLICATION PACKAGEs` +- Add support for row access policies +- Add support for AVRO file format +- Add support for `ALTER STAGE` and `ALTER STREAM` statements +- Enhance `SHOW COLUMNS/SCHEMAS` feature parity +- Add support for `EQUAL_NULL` and variant type check functions +- Add support for `ARRAY_REVERSE` function +- Add support for `DESCRIBE/SHOW SEQUENCES` +- Enhance timestamp handling and timezone support +- Add support for metadata columns in JSON +- Add support for `INSERT OVERWRITE` queries +- Support ZTSD compression +- Enhance `TO_CHAR` with date and number formatting +- Support dropping multiple columns in single `ALTER` statement +- Add support for TIMEZONE session parameter +- Make default user and password configurable +- Fix various timestamp and timezone-related issues +- Improve query performance by re-using PG connections per session +- Enhance parity for comparison of VARIANTs and JSON objects +- Enhance column aliasing for joins +- Fix type mapping for rowtype metadata to enhance compatibility with .NET clients +- Cast `INSERT INTO` arguments to target type +- Add support for `REGEXP_SUBSTR` function +- Prohibit Postgres specific data types +- Add `HEADER` and `FIELD_OPTIONALLY_ENCLOSED_BY` support for COPY INTO +- Support subquery in copy into location +- Support not equal operator +- Enhance parity for $N references in WHERE clauses +- Increase identifiers length to 255 char +- Improve JDBC driver compatibility +- Support LS alias to LIST files from stages + +### 0.2.5 +- Change storage integration to not be associated with DB/schema +- Add enhanced support for SHOW ROLES and USE ROLE +- Enhance parity for COUNT(..) with NULL values +- Add multiple statements support +- Fix staging issues with parsing +- Enhance logic for timestamp assignments in MERGE INTO queries +- Add proper passing of stage parameters and validations +- Add support for `SYSTEM$CLIENT_VERSION_INFO`/`OBJECT_CONSTRUCT_KEEP_NULL`/`TIMESTAMPADD`/`DATEADD`/`TIMEADD` functions +- Fix timestamp_ltz comparison operators issue +- Fix INSERT result count type +- Show tables metadata/information for dynamic tables +- Include Snowflake emulator version in the logs +- Update web app layout +- Enhance parity for NUMBER/FIXED data types in JDBC results +- Add support for querying from `INFORMATION_SCHEMA.TABLES` +- Fix Arrow encoding for large decimal numeric values +- Fix session for /api/v2/statements endpoint +- Improve file formats handling in COPY INTO +- Enhance parity for COPY INTO with multiple stage files +- Update metadata for integer +- Enhance parity around selecting NULL values for DATE columns +- Add initial support for materialized views +- Add format parameter for `TO_TIMESTAMP` function +- Add support for async execution of multi-statement queries +- Add support for patterns in select from stages +- Allow table column projections for COPY INTO queries from stage files +- Add support for numeric operators with mixed/variant types +- Add initial support for `MAP` data type and util functions +- Create `INFORMATION_SCHEMA.FUNCTIONS` table +- Add support for querying metadata filename in select from stage statements +- Enhance parity for primary/foreign keys +- Add support for IF EXISTS clauses in ALTER COLUMN queries +- Add initial support for user-defined transaction management + +### 0.2.4 +- Support `POWER`/`POW`/`DIV0NULL`/`IFNULL` functions +- Add support for COPY INTO location +- Add initial support for table/database clones +- Establish parity with snowflake when csv imports +- Allow binding multiple values +- Fix database and schema names in copy into table +- Fix `executeUpdate` in JDBC +- Support `ORDER` and `NOORDER` from AUTOINCREMENT column def +- Add initial logic and tests for replicating resources between accounts +- Convert empty csv values to null +- Add support and tests for `LATERAL` queries +- Add initial support for EXECUTE IMMEDIATE +- Add initial support for tags +- Add identifier support to SELECT queries +- Fix inserting timestamp values correctly +- Fix timestamps in insert for current_timestamp +- Add support for init scripts +- Fix handling timestamp values on update +- Add support for `DESCRIBE STAG` +- Add initial support for storage integrations +- Enhance parity of `TIMESTAMP_LTZ` +- Create clone db using identifiers +- Add mock support for replication databases to fix TF issues +- Fix logic for setting session parameters +- Add support for some extended GRANT statements +- Support `ALTER SEQUENCE` +- Support creating stages with storage integrations +- Terraform create database fixes +- Improve general error handling +- Add initial support for `SHOW GRANTS TO/OF` + +### 0.2.3 +- Add initial support for `OBJECT_KEYS`/`PARSE_IP`/`DESCRIBE FUNCTION` functions +- Add support for various functions including `DATE_TRUNC`, `NVL2`, `LEAST`, `GREATEST`, and more +- Enhance parity for creation/deletion of schemas with fully qualified names +- Enhance parity for inserting timestamps with subsecond precision +- Enhance parity for CTAS with nested subqueries +- Enhance parity for id placeholders in JDBC prepared statements +- Enhance parity for metadata queries and schema lookup +- Adjust `MIN_BY`/`MAX_BY` aggregate functions +- Properly extract db/schema parameters for JDBC connections +- Implement trigonometric and hyperbolic functions +- Add support for GET stage files + +### 0.2.2 +- Add initial support for hybrid tables and dynamic tables +- Add support for `OBJECT_CONSTRUCT_KEEP_NULL`/`AS_DOUBLE`/`AS_INTEGER`/`AS_NUMBER`/`AS_CHAR` +- Add `/result` API endpoint to retrieve query results +- Track original types in internal VARIANTs +- Enhance parity for `SHOW WAREHOUSES` queries +- Support for `SHOW TASKS` +- Enhance parsing of stage params +- Fix selection of columns when querying stage files +- Automatically adjust PG JIT support if LLVM libs are missing +- Enhance custom JSON parsing to allow escaped characters +- Enhance parity of `TIMESTAMP_LTZ` for Flyway compatibility + +### 0.2.1 +- Add initial support for Iceberg tables +- Add initial support for external volumes and Snowflake pipes +- Support `LIST`/`REMOVE` queries for staged files +- Support `SHOW PIPES` queries +- Support `COPY GRANTS` in `CREATE TABLE` queries +- Add multiple new SQL functions +- Implement `RANK`/`DENSE_RANK` +- Implement various conversion functions +- Enhance logic for `TO_CHAR` +- Support window queries with `QUALIFY` +- Support `COUNT_IF` aggregate functions +- Make `CREATE SERVER` queries idempotent +- Fix compatibility issues +- Add MUI data-grid for results table in UI +- Add squashing of Docker image to reduce size + +### 0.2.0 +- Support various SQL functions (`BITAND`, `FLATTEN`, `RANDOM`, etc.) +- Add Snowflake proxy request handler +- Add initial version of simple UI view +- Fix execution of CTAS queries with UNION selects +- Fix logic for PUT file uploads +- Support parsing incomplete JSON +- Enhance support for TABLESAMPLE queries +- Add initial support for temporary and transient tables +- Add Snowflake v2 SQL APIs +- Fix `describeOnly` `INSERT` queries + +### 0.1.26 +- Support `CONVERT_TIMEZONE`, `IFF` SQL functions +- Implement `ALTER WAREHOUSE` as no-op +- Implement time functions +- Enhance JDBC driver compatibility +- Fix Arrow encoding +- Support SQL queries from within JS UDFs +- Add various performance improvements +- Implement additional date functions +- Add support for loading data from public S3 buckets + +### 0.1.25 +- Enhance support for various SHOW queries +- Add initial persistence support for Snowflake store +- Enhance parity for timestamp types +- Fix SHOW PARAMETERS for Terraform compatibility +- Set up CI build for Docker image + +### 0.1.24 +- Enhance parity around user-defined PROCEDUREs +- Fix upper-casing for various functions +- Enhance support for UNIQUE column constraints +- Add initial support for cross-DB resource sharing + +### 0.1.23 +- Add initial simple scheduler to periodically run tasks + +### 0.1.22 +- Fix query transforms for `ADD COLUMN` queries +- Fix wrapping of `VALUES` subquery in braces +- Add initial CRUD support for `TASK`s +- Support `DROP PRIMARY KEY` queries +- Migrate use of `localstack.http` to `rolo` + +### 0.1.21 +- Add support for consuming table stream records via DML statements + +### 0.1.20 +- Initial simple support for table streams +- Add support for `SHOW DATABASES`, `SHOW VIEWS` +- Enhance parity for Arrow results +- Fix various identifier and query issues + +### 0.1.19 +- Return `SELECT` results in arrow format for pandas compatibility +- Add `add_months` function +- Fix UDFs with raw expressions +- Upgrade to Postgres v15 +- Various parity and performance improvements + +### 0.1.18 +- Add support for various array and aggregation functions +- Enhance `FILE FORMAT` operations +- Fix `CTAS` queries +- Support `INFER_SCHEMA(..)` for parquet files +- Improve identifier handling + +### 0.1.17 +- Support creation/deletion of stages +- Add `IS_ARRAY` function +- Remove `DuckDB` based `DB` engine +- Refactor codebase to use `QueryProcessor` interface +- Enhance column name handling + +### 0.1.16 +- Add support for `SHOW PROCEDURES` and `SHOW IMPORTED KEYS` +- Add basic support for session parameters + +### 0.1.15 +- Fix result type conversation for `GET_PATH(..)` util function + +### 0.1.14 +- Enhance parity around `SHOW` queries +- Add more array util functions +- Fix `STRING_AGG` functionality + +### 0.1.13 +- Support `CURRENT_*` functions +- Enhance `LISTAGG` for distinct values +- Add test for `JS` UDFs with exports + +### 0.1.12 +- Cast params for `string_agg`/`listagg` +- Fix parity for upper/lowercase names + +### 0.1.11 +- Enhance parity for array aggregation functions +- Improve timestamp timezone handling +- Add case-sensitive identifier tracking + +### 0.1.10 +- Add query transforms for `CLUSTER BY` +- Add `SF_S3_ENDPOINT` config +- Various parity fixes + +### 0.1.9 +- Add support for `Python` UDFs +- Enhance function creation parity +- Add analytics setup + +### 0.1.8 +- Add `SF_LOG` config for request/response trace logging + +### 0.1.7 +- Add initial support for `JavaScript` UDFs +- Enhance DB/table creation responses +- Improve streaming logic + +### 0.1.6 +- Introduce session state for DB/schema retention +- Support async queries and `result_scan(..)` + +### 0.1.5 +- Enhance `DESCRIBE TABLE` results +- Support `MIN_BY`/`MAX_BY` aggregate functions + +### 0.1.4 +- Add logic to parse and replace `DB` references in queries + +### 0.1.3 +- Add `DBEngine` abstraction +- Add experimental support for `duckdb` +- Enhance `JSON` query support + +### 0.1.2 +- Add CSV file ingestion from Snowflake stage to table + +### 0.1.1 +- Initial support for `Kafka` connector +- Add `snowpipe`/streaming APIs + +### 0.1.0 +- Initial release of the extension \ No newline at end of file diff --git a/src/content/docs/snowflake/getting-started/faq.md b/src/content/docs/snowflake/getting-started/faq.md new file mode 100644 index 00000000..c788eb95 --- /dev/null +++ b/src/content/docs/snowflake/getting-started/faq.md @@ -0,0 +1,38 @@ +--- +title: FAQ +description: Frequently asked questions about LocalStack for Snowflake +template: doc +--- + + +{{< preview-notice >}} + +## FAQs + +### Are Snowflake v2 APIs supported? + +Yes, the Snowflake emulator supports the Snowflake v2 SQL API (`/api/v2/*` endpoints), as well as the legacy v1 SQL API (which is still being used by a large portion of Snowflake client libraries and SDKs) + +### Why are my Snowflake tests failing? + +The Snowflake emulator is in **preview** and may not support all Snowflake features. If your tests are failing, it could be due to the lack of support for certain Snowflake features in the emulator. We recommend checking the [function coverage](https://snowflake.localstack.cloud/references/coverage-functions/) to see the list of supported SQL functions and [feature coverage](https://snowflake.localstack.cloud/references/coverage-features/) to see the list of supported features. If you encounter any issues, you can connect with us for [support](#support-faqs). + +### Why does the Snowflake emulator run on `snowflake.localhost.localstack.cloud`? + +The Snowflake emulator operates on `snowflake.localhost.localstack.cloud`. This is a DNS name that resolves to a local IP address (`127.0.0.1`) to make sure the connector interacts with the local APIs. In addition, we also publish an SSL certificate that is automatically used inside LocalStack, in order to enable HTTPS endpoints with valid certificates. + +Note: In case you are deploying the Snowflake emulator in a Kubernetes cluster or some other non-local environment, you may need to add an entry to the `/etc/hosts` file of any client machine or Kubernetes pod that attempts to connect to the Snowflake emulator pod via the `snowflake.localhost.localstack.cloud` domain name. + +## Integration FAQs + +### Why are my CI pipelines failing with `license.not_enough_credits` error? + +If you are using the Snowflake emulator in your CI pipelines consistently, you may encounter the `license.not_enough_credits` error. This error occurs when the Snowflake emulator is unable to process the requests due to the lack of LocalStack CI credits. + +A CI key allows you to use LocalStack in your CI environment. Every activation of a CI key consumes one CI credit. This means that with every build triggered through the LocalStack container you will consume one credit. To use more credits, you can [contact us](https://localstack.cloud/contact) to discuss your requirements. + +## Support FAQs + +### How can I get help with the Snowflake emulator? + +The Snowflake emulator is currently in **preview**. To get help, you can join the [Slack community](https://localstack.cloud/slack) and share your feedback, questions, and suggestions with the LocalStack team on the `#help` channel. If your team is using LocalStack for Snowflake, you can also request support by [contacting us](https://localstack.cloud/contact). We would be happy to setup a private Slack channel for your team to provide dedicated support. \ No newline at end of file diff --git a/src/content/docs/snowflake/getting-started/index.md b/src/content/docs/snowflake/getting-started/index.md new file mode 100644 index 00000000..13d8a52c --- /dev/null +++ b/src/content/docs/snowflake/getting-started/index.md @@ -0,0 +1,116 @@ +--- +title: Installation +description: Basic installation guide to get started with LocalStack for Snowflake. +template: doc +--- + +{{< preview-notice >}} + +## Introduction + +You can use the Snowflake Docker image to run the Snowflake emulator. +The Snowflake Docker image is available on the [LocalStack Docker Hub](https://hub.docker.com/r/localstack/snowflake). +To pull the Snowflake Docker image, execute the following command: + +{{< command >}} +$ docker pull localstack/snowflake +{{< / command >}} + +You can start the Snowflake Docker container using the following methods: + +1. [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) +2. [`docker` CLI](https://docs.docker.com/get-docker/) +2. [Docker Compose](https://docs.docker.com/compose/install/) + +{{}} +Before starting, ensure you have a valid `LOCALSTACK_AUTH_TOKEN` to access the Snowflake emulator. Refer to the [Auth Token guide](https://docs.localstack.cloud/getting-started/auth-token/) to obtain your Auth Token and specify it in the `LOCALSTACK_AUTH_TOKEN` environment variable. +{{}} + +### `localstack` CLI + +To start the Snowflake Docker container using the `localstack` CLI, execute the following command: + +{{< command >}} +$ export LOCALSTACK_AUTH_TOKEN= +$ IMAGE_NAME=localstack/snowflake localstack start +{{< / command >}} + +### `docker` CLI + +To start the Snowflake Docker container using the `docker` CLI, execute the following command: + +{{< command >}} +$ docker run \ + --rm -it \ + -p 127.0.0.1:4566:4566 \ + -p 127.0.0.1:4510-4559:4510-4559 \ + -p 127.0.0.1:443:443 \ + -e LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?} \ + localstack/snowflake +{{< / command >}} + +### Docker Compose + +Create a `docker-compose.yml` file with the specified content: + +```yaml +version: "3.8" + +services: + localstack: + container_name: "localstack-snowflake" + image: localstack/snowflake + ports: + - "127.0.0.1:4566:4566" + - "127.0.0.1:4510-4559:4510-4559" + - "127.0.0.1:443:443" + environment: + - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?} + volumes: + - "./volume:/var/lib/localstack" +``` + +Start the Snowflake Docker container with the following command: + +{{< command >}} +$ docker-compose up +{{< / command >}} + +## Updating + +To update the Snowflake Docker container, pull the latest image and restart the container. The `latest` tag is the nightly build of the Snowflake Docker image. + +## Troubleshooting + +### How to check if the Snowflake emulator is running? + +You can check if the Snowflake emulator is running by executing the following command: + +{{< command >}} +$ curl -d '{}' snowflake.localhost.localstack.cloud:4566/session +{"success": true} +{{< / command >}} + +### How to enable detailed debug logs? + +You can set the `SF_LOG=trace` environment variable in the Snowflake container to enable detailed trace logs that show all the request/response message. + +When using `docker-compose` then simply add this variable to the `environment` section of the YAML configuration file. +If you're starting up via the `localstack start` CLI, then make sure to start up via the following configuration: + +{{< command >}} +$ DOCKER_FLAGS='-e SF_LOG=trace' DEBUG=1 IMAGE_NAME=localstack/snowflake localstack start +{{< / command >}} + +### The `snowflake.localhost.localstack.cloud` hostname doesn't resolve on my machine, what can I do? + +On some systems, including some newer versions of MacOS, the domain name `snowflake.localhost.localstack.cloud` may not resolve properly. +If you are encountering network issues and your Snowflake client drivers are unable to connect to the emulator, you may need to manually add the following entry to your `/etc/hosts` file: + +{{< command >}} +127.0.0.1 snowflake.localhost.localstack.cloud +{{< / command >}} + +## Next steps + +Now that the Snowflake emulator is installed, you can use it for developing and testing your Snowflake data pipelines. Refer to our [Quickstart]({{< ref "quickstart" >}}) guide to get started. \ No newline at end of file diff --git a/src/content/docs/snowflake/getting-started/quickstart.md b/src/content/docs/snowflake/getting-started/quickstart.md new file mode 100644 index 00000000..0c81ca24 --- /dev/null +++ b/src/content/docs/snowflake/getting-started/quickstart.md @@ -0,0 +1,137 @@ +--- +title: Quickstart +description: Get started with LocalStack for Snowflake in a few simple steps +template: doc +--- + +{{< preview-notice >}} + +## Introduction + +This guide explains how to set up the Snowflake emulator and develop a Python program using the Snowflake Connector for Python (`snowflake-connector-python`) to interact with emulated Snowflake running on your local machine. + +## Prerequisites + +- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) +- [LocalStack for Snowflake]({{< ref "installation" >}}) +- Python 3.10 or later +- [`snowflake-connector-python` library](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-install) + +## Instructions + +Before you begin, pull the Snowflake emulator image (`localstack/snowflake`) and start the container: + +{{< command >}} +$ export LOCALSTACK_AUTH_TOKEN= +$ IMAGE_NAME=localstack/snowflake:latest localstack start +{{< / command >}} + +Check the emulator's availability by running: + +{{< command >}} +$ curl -d '{}' snowflake.localhost.localstack.cloud:4566/session + +{"success": true} + +{{< / command >}} + +### Connect to the Snowflake emulator + +Create a new Python file named `main.py` and use the following code to connect to the Snowflake emulator: + +```python +import snowflake.connector as sf + +sf_conn_obj = sf.connect( + user="test", + password="test", + account="test", + database="test", + host="snowflake.localhost.localstack.cloud", +) +``` + +Specify the `host` parameter as `snowflake.localhost.localstack.cloud` and the other parameters as `test` to avoid connecting to the real Snowflake instance. + +### Create and execute a query + +Extend the Python program to insert rows from a list object into the emulated Snowflake table. Create a cursor object and execute the query: + +```python +print("1. Insert lot of rows from a list object to Snowflake table") +print("2. Creating a cursor object") +sf_cur_obj = sf_conn_obj.cursor() + +print("3. Executing a query on cursor object") +try: + sf_cur_obj.execute( + "create or replace table " + "ability(name string, skill string )") + + rows_to_insert = [('John', 'SQL'), ('Alex', 'Java'), ('Pete', 'Snowflake')] + + sf_cur_obj.executemany( + " insert into ability (name, skill) values (%s,%s) " ,rows_to_insert) + + sf_cur_obj.execute("select name, skill from ability") + + print("4. Fetching the results") + result = sf_cur_obj.fetchall() + print("Total # of rows :" , len(result)) + print("Row-1 =>",result[0]) + print("Row-2 =>",result[1]) +finally: + sf_cur_obj.close() +``` + +This program creates a table named `ability`, inserts rows, and fetches the results. + +### Run the Python program + +Execute the Python program with: + +{{< command >}} +$ python main.py +{{< / command >}} + +The output should be: + +```bash +Insert lot of rows from a list object to Snowflake table +1. Insert lot of rows from a list object to Snowflake table +2. Creating a cursor object +3. Executing a query on cursor object +4. Fetching the results +Total # of rows : 3 +Row-1 => ('John', 'SQL') +Row-2 => ('Alex', 'Java') +``` + +Verify the results by navigating to the LocalStack logs: + +```bash +2024-02-22T06:03:13.627 INFO --- [ asgi_gw_0] localstack.request.http : POST /session/v1/login-request => 200 +2024-02-22T06:03:16.122 WARN --- [ asgi_gw_0] l.packages.core : postgresql will be installed as an OS package, even though install target is _not_ set to be static. +2024-02-22T06:03:45.917 INFO --- [ asgi_gw_0] localstack.request.http : POST /queries/v1/query-request => 200 +2024-02-22T06:03:46.016 INFO --- [ asgi_gw_1] localstack.request.http : POST /queries/v1/query-request => 200 +2024-02-22T06:03:49.361 INFO --- [ asgi_gw_0] localstack.request.http : POST /queries/v1/query-request => 200 +2024-02-22T06:03:49.412 INFO --- [ asgi_gw_1] localstack.request.http : POST /session => 200 +``` + +### Destroy the local infrastructure + +To stop LocalStack and remove locally created resources, use: + +{{< command >}} +$ localstack stop +{{< / command >}} + +LocalStack is ephemeral and doesn't persist data across restarts. It runs inside a Docker container, and once it’s stopped, all locally created resources are automatically removed. In a future release of the Snowflake emulator, we will provide proper persistence and integration with our [Cloud Pods](https://docs.localstack.cloud/user-guide/state-management/cloud-pods/) feature as well. + +## Next steps + +You can now explore the following resources to learn more about the Snowflake emulator: + +- [User Guide]({{< ref "user-guide" >}}): Learn about the Snowflake emulator's features and how to use them. +- [Tutorials]({{< ref "tutorials" >}}): Explore tutorials to use the Snowflake emulator for local development and testing. +- [References]({{< ref "references" >}}): Find information about the Snowflake emulator's configuration, changelog, and function coverage. \ No newline at end of file diff --git a/src/content/docs/snowflake/index.md b/src/content/docs/snowflake/index.md index 10ffa5e3..ee68d87d 100644 --- a/src/content/docs/snowflake/index.md +++ b/src/content/docs/snowflake/index.md @@ -6,4 +6,23 @@ nav: 1 label: Welcome --- -Welcome \ No newline at end of file +# Welcome to LocalStack Docs! + +## What would you like to do today? + + +[LocalStack](https://localstack.cloud/) is a cloud service emulator that runs in a single container on your laptop or in your CI environment. +LocalStack for Snowflake emulates the functionality of a real Snowflake instance, allowing you to perform operations without an internet connection or a Snowflake account. +This is valuable for locally developing and testing Snowflake data pipelines without incurring costs. + +The Snowflake emulator supports the following features: + +* [**Basic operations** on warehouses, databases, schemas, and tables](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example) +* [**Storing files** in user/data/named **stages**](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage) +* [**Snowpark** libraries](https://docs.snowflake.com/en/developer-guide/snowpark/python/index) +* [**Snowpipe** streaming with **Kafka connector**](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-kafka) +* [**JavaScript and Python UDFs**](https://docs.snowflake.com/en/developer-guide/udf/javascript/udf-javascript-introduction) +* ... and more! + +Integrating the Snowflake emulator into your existing CI/CD pipeline allows you to run integration tests and identify issues early, reducing surprises during production deployment. +Check our [SQL Functions Coverage]({{< ref "coverage-functions" >}}) and [Feature Coverage]({{< ref "coverage-features" >}}) pages for a comprehensive list of supported features. diff --git a/src/content/docs/snowflake/integrations/airflow.md b/src/content/docs/snowflake/integrations/airflow.md new file mode 100644 index 00000000..8c9fa6d1 --- /dev/null +++ b/src/content/docs/snowflake/integrations/airflow.md @@ -0,0 +1,145 @@ +--- +title: Airflow +description: Use Airflow to run local ETL jobs against the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +Apache [Airflow](https://airflow.apache.org) is a platform for running data-centric workflows and scheduled compute jobs. +LocalStack supports the [AWS Managed Workflows for Apache Airflow](https://docs.localstack.cloud/user-guide/aws/mwaa/) (MWAA) service to run Airflow jobs locally. + +You can use Airflow to interact with the LocalStack Snowflake emulator and run ETL (Extract-Transform-Load) jobs, using the Airflow `SnowflakeOperator` for running queries against Snowflake. +On this page we outline how to set up the connection between local Airflow and the Snowflake emulator. + +## Create an Airflow environment via MWAA in LocalStack + +In order to create an Airflow environment in local MWAA, we can use the [`awslocal`](https://github.com/localstack/awscli-local) command: + +{{< command>}} +$ awslocal s3 mb s3://my-mwaa-bucket +$ awslocal mwaa create-environment --dag-s3-path /dags \ + --execution-role-arn arn:aws:iam::000000000000:role/airflow-role \ + --network-configuration {} \ + --source-bucket-arn arn:aws:s3:::my-mwaa-bucket \ + --airflow-version 2.6.3 \ + --name my-mwaa-env +{{< /command >}} + +## Create an Airflow DAG script that connects to LocalStack Snowflake + +We can then create a local file `my_dag.py` with the Airflow DAG definition, for example: +```python +import datetime +import json + +from airflow import settings +from airflow.models import Connection, DAG +from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator + +# prepare session and connection info + +session = settings.Session() +conn_id = "c1" +try: + # try to look up local Snowflake connection in the session + session.query(Connection).filter(Connection.conn_id == conn_id).one() +except Exception: + # create new Snowflake connection, if it doesn't exist yet + conn = Connection( + conn_id=conn_id, + conn_type="snowflake", + login="test", + password="test", + extra=json.dumps({"account": "test", "host": "snowflake.localhost.localstack.cloud", "port": 4566}) + ) + session.add(conn) + session.commit() + +# create DAG + +my_dag = DAG( + "sf_dag1", + start_date=datetime.datetime.utcnow(), + default_args={"snowflake_conn_id": conn_id}, + catchup=False, +) + +# add Snowflake operator to DAG + +sf_task_1 = SnowflakeOperator( + task_id="sf_query_1", + dag=my_dag, + sql=""" + CREATE TABLE IF NOT EXISTS test(id INT); + COPY INTO test (id) VALUES (1), (2), (3); + SELECT * FROM test + """, +) +``` + +### Patching the `SnowflakeOperator` in the DAG script + +In order to use the `SnowflakeOperator` in your Airflow DAG, a small patch is required in the code. +The code listings below contain the patch for different Airflow versions - simply copy the relevant snippet and paste it into the top of your DAG script (e.g., `my_dag.py`). + +**Airflow version 2.6.3 and above**: +```python +# --- +# patch for local Snowflake connection, for Airflow 2.6.3 and above +from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook + +def _get_conn_params(self): + result = self._get_conn_params_orig() + conn = self.get_connection(self.snowflake_conn_id) + extra_dict = conn.extra_dejson + if extra_dict.get("host"): + result["host"] = extra_dict["host"] + if extra_dict.get("port"): + result["port"] = extra_dict["port"] + return result + +SnowflakeHook._get_conn_params_orig = SnowflakeHook._get_conn_params +SnowflakeHook._get_conn_params = _get_conn_params +# --- + +# ... rest of your DAG script below ... +``` + +**Airflow version 2.9.2 and above**: +```python +# --- +# patch for local Snowflake connection, for Airflow 2.9.2 / 2.10.1 +from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook + +@property +def _get_conn_params(self): + result = self._get_conn_params_orig + conn = self.get_connection(self.snowflake_conn_id) + extra_dict = conn.extra_dejson + if extra_dict.get("host"): + result["host"] = extra_dict["host"] + if extra_dict.get("port"): + result["port"] = extra_dict["port"] + return result + +SnowflakeHook._get_conn_params_orig = SnowflakeHook._get_conn_params +SnowflakeHook._get_conn_params = _get_conn_params +# --- + +# ... rest of your DAG script below ... +``` + +{{< alert type="info" title="Note" >}} +In a future release, we're looking to integrate these patches directly into the LocalStack environment, such that users do not need to apply these patches in DAG scripts manually. +{{< /alert >}} + +## Deploying the DAG to Airflow + +Next, we copy the `my_dag.py` file to the `/dags` folder within the `my-mwaa-bucket` S3 bucket, to trigger the deployment of the DAG in Airflow: +{{< command>}} +$ awslocal s3 cp my_dag.py s3://my-mwaa-bucket/dags/ +{{< /command >}} + +You should then be able to open the Airflow UI (e.g., http://localhost.localstack.cloud:4510/dags) to view the status of the DAG and trigger a DAG run. \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/continuous-integration.md b/src/content/docs/snowflake/integrations/continuous-integration.md new file mode 100644 index 00000000..a49a8ab9 --- /dev/null +++ b/src/content/docs/snowflake/integrations/continuous-integration.md @@ -0,0 +1,112 @@ +--- +title: Continuous Integration +description: Get started with Snowflake emulator in continuous integration (CI) environments. +--- + +{{< preview-notice >}} + +## Introduction + +This guide explains how to set up the Snowflake emulator in a continuous integration (CI) environment. You can use the emulator to test your Snowflake integration without connecting to the real Snowflake instance. + +To install the Snowflake emulator, you need to set up the LocalStack CLI and pull the Docker image. After starting the Docker container, an endpoint (`snowflake.localhost.localstack.cloud`) is made available to connect to the emulated Snowflake instance, which you can use to test your data integrations in CI environments. + +## Configure the LocalStack CI Key + +Before you begin, set up the LocalStack CI key to activate the Snowflake emulator in your CI environment. LocalStack requires a CI Key for use in continuous integration (CI) or similar machine environments. Each instance startup in a CI or comparable environment consumes one CI token. + +To create a CI key, follow these steps: + +1. Open the [LocalStack Web Application](https://app.localstack.cloud). +2. Click the [**CI Keys**](https://app.localstack.cloud/workspace/ci-keys) on the left navigation pane. +3. Navigate to the **Generate CI Key** tab, enter a description, and click **Generate CI Key**. +4. Copy the generated CI key and set it as the `LOCALSTACK_API_KEY` environment variable in your CI environment. + +## Setup the CI configuration + +The following examples demonstrate how to set up the emulator in GitHub Actions, CircleCI, and GitLab CI. + +{{< tabpane >}} +{{< tab header="GitHub Actions" lang="yaml" >}} +name: LocalStack Test +on: [ push, pull_request ] + +jobs: + localstack-action-test: + name: 'Test LocalStack GitHub Action' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Start Snowflake emulator + run: + docker pull localstack/snowflake:latest & + pip install localstack + IMAGE_NAME=localstack/snowflake localstack start -d + echo "Waiting for LocalStack startup..." + localstack wait -t 15 + echo "Startup complete" + env: + LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }} +{{< /tab >}} +{{< tab header="CircleCI" lang="yaml" >}} +version: 2.1 + +orbs: + python: circleci/python@2.0.3 + +jobs: + example-job: + machine: + image: ubuntu-2004:2022.04.1 + + steps: + - checkout + + - run: + name: Start LocalStack + command: | + pip3 install localstack + docker pull localstack/snowflake + IMAGE_NAME=localstack/snowflake localstack start -d + + echo "Waiting for LocalStack startup..." + localstack wait -t 30 + echo "Startup complete" + +workflows: + version: 2 + build: + jobs: + - example-job +{{< /tab >}} +{{< tab header="GitLab CI" lang="yaml" >}} +image: docker:20.10.16 + +stages: + - test + +test: + stage: test + variables: + DOCKER_HOST: tcp://docker:2375 + DOCKER_TLS_CERTDIR: "" + LOCALSTACK_API_KEY: $LOCALSTACK_API_KEY + + services: + - name: docker:20.10.16-dind + alias: docker + command: ["--tls=false"] + + before_script: + - apk update + - apk add gcc musl-dev linux-headers py3-pip python3 python3-dev + - python3 -m pip install localstack + script: + - docker pull localstack/snowflake:latest + - dind_ip="$(getent hosts docker | cut -d' ' -f1)" + - echo "${dind_ip} localhost.localstack.cloud " >> /etc/hosts + - DOCKER_HOST="tcp://${dind_ip}:2375" IMAGE_NAME=localstack/snowflake localstack start -d + - localstack wait -t 15 +{{< /tab >}} +{{< /tabpane >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/dbeaver.md b/src/content/docs/snowflake/integrations/dbeaver.md new file mode 100644 index 00000000..381bd7cd --- /dev/null +++ b/src/content/docs/snowflake/integrations/dbeaver.md @@ -0,0 +1,37 @@ +--- +title: DBeaver +description: Use DBeaver to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[DBeaver](https://dbeaver.io/) is a free and open-source universal database tool for developers, database administrators, and analysts. DBeaver provides a wide range of features, such as executing SQL statements, viewing and editing data, managing database objects, and more. + +You can use DBeaver to interact with the Snowflake emulator using the same commands and syntax as the Snowflake service. With DBeaver, you can manage Snowflake resources locally, such as databases, schemas, tables, stages, and more. + +## Configuring DBeaver + +In this guide, you will learn how to configure DBeaver to interact with the Snowflake emulator. + +### Install DBeaver + +To install DBeaver, follow the instructions in the [official DBeaver documentation](https://dbeaver.io/download/). + +### Create a new connection + +To create a new connection in DBeaver, follow these steps: + +- Open DBeaver. Go to the top menu, select **Database**, and choose **New Database Connection**. In the **Connect to database** window, pick **All** databases and search for **Snowflake**, then click **Next**. +- In the **Connect to database** window, switch to the **Main** tab. Enter your Snowflake user details: + - **Host**: `snowflake.localhost.localstack.cloud` + - **User**: `test` + - **Password**: `test` + New connection in DBeaver +

+ +- Click **Test Connection**. +- If the connection test succeeds, click **Finish**. The Snowflake database will appear in DBeaver's Database Navigator. + +You can verify the connection by running a query to check the Snowflake version: `SELECT CURRENT_VERSION();` \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/dbt.md b/src/content/docs/snowflake/integrations/dbt.md new file mode 100644 index 00000000..68572c3e --- /dev/null +++ b/src/content/docs/snowflake/integrations/dbt.md @@ -0,0 +1,183 @@ +--- +title: dbt +description: Use dbt to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[dbt (data build tool)](https://www.getdbt.com/) is a transformation workflow tool that enables data analysts and engineers to transform data in their warehouses by writing modular SQL. dbt handles version control, documentation, and modularity for data transformations. + +The Snowflake emulator supports dbt, allowing you to develop and test your dbt models locally without connecting to a production Snowflake instance. + +## Configuring dbt + +In this guide, you will learn how to configure dbt to interact with the Snowflake emulator. + +### Install dbt + +First, install dbt with the Snowflake adapter: + +{{< command >}} +$ pip install dbt-snowflake +{{< /command >}} + +### Configure dbt Profile + +Create or modify your `profiles.yml` file (typically located in `~/.dbt/profiles.yml`) to include the connection details for the Snowflake emulator: + +```yaml +localstack_snowflake: + outputs: + dev: + account: localstack + host: snowflake.localhost.localstack.cloud + port: 4566 + database: test + password: test + role: test + schema: public + threads: 1 + type: snowflake + user: test + warehouse: test + target: dev +``` + +### Test the Connection + +To verify your dbt configuration is working correctly with the Snowflake emulator, run: + +{{< command >}} +$ dbt debug --profile localstack_snowflake +{{< /command >}} + +You should see output indicating a successful connection to the Snowflake emulator. + +### Running dbt Commands + +Once configured, you can run standard dbt commands against the Snowflake emulator. + +### Example dbt Model + +Here's a simple example of a dbt model which creates a table with a single row that you can use to test the integration: + +```sql +-- models/example_model.sql +{{ config(materialized='table') }} + +SELECT + 1 as id, + 'test' as name +``` + +### Example Tests + +You can test your models using dbt's generic tests. + +Add the following to your `models/schema.yml`: + +```yaml +version: 2 + +models: + - name: example_model + description: "A simple example model with generic tests" + columns: + - name: id + description: "The primary key for this table" + tests: + - unique + - not_null + - name: name + description: "The name field" + tests: + - not_null +``` + +{{< command >}} +# Run all models +$ dbt run --profile localstack_snowflake + +# Run tests +$ dbt test --profile localstack_snowflake + +{{< /command >}} + +### Project Structure + +A typical dbt project structure when working with the Snowflake emulator might look like this: + +``` +my_dbt_project/ +├── dbt_project.yml +├── models/ +│ ├── example_model.sql +│ └── schema.yml +└── README.md +``` + +Example `dbt_project.yml`: + +```yaml +name: 'my_dbt_project' +version: '1.0.0' +config-version: 2 + +profile: 'localstack_snowflake' + +model-paths: ["models"] +test-paths: ["tests"] +analysis-paths: ["analyses"] +macro-paths: ["macros"] + +target-path: "target" +clean-targets: + - "target" + - "dbt_packages" + +models: + my_dbt_project: + materialized: table +``` + +### Connect to dbt Cloud + +To use dbt Cloud with the Snowflake emulator, you'll first need to configure an ephemeral instance. Ephemeral instances provide isolated environments for testing and development. For more information, see the [Ephemeral Instances documentation]({{< ref "user-guide/ephemeral-instances" >}}). + +1. First, create an ephemeral instance for Snowflake in the LocalStack Web application. + +2. Once your ephemeral instance is running, note the host URL (e.g., `sf-v09rkl9fcjs21.sandbox.localstack.cloud`). + +3. In the dbt Cloud interface: + - Navigate to **Deploy** > **Environments** + - Create a new environment or edit an existing one + - Under **Connection Settings**, select **Snowflake** + - Fill in the basic credentials: + * Username: `test` + * Password: `test` + * Schema: `public` + +4. In the **Extended attributes** section, specify the host from your ephemeral instance: + +```yaml +host: sf-v09rkl9fcjs21.sandbox.localstack.cloud +``` + +![dbt Cloud Configuration](dbt-cloud-config.png) + +{{< alert type="info" >}} +The host value must match your ephemeral instance URL. This overrides the default Snowflake host and directs connections to your LocalStack environment. +{{< /alert >}} + +## Best Practices + +1. **Version Control**: Keep your dbt models and configurations in version control +2. **Testing**: Write tests for your models to ensure data quality +3. **Documentation**: Document your models using dbt's built-in documentation features +4. **Modularity**: Break down complex transformations into smaller, reusable models + +{{< alert type="info" >}} +It's a good practice to always test your dbt models locally with the Snowflake emulator before deploying to production, to save time and resources. +{{< /alert >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/flyway.md b/src/content/docs/snowflake/integrations/flyway.md new file mode 100644 index 00000000..6afd513e --- /dev/null +++ b/src/content/docs/snowflake/integrations/flyway.md @@ -0,0 +1,40 @@ +--- +title: Flyway +description: Use Flyway to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[Flyway](https://flywaydb.org/) is an open-source database migration tool that simplifies the process of managing and applying database migrations. Flyway supports various databases, including Snowflake, allowing you to manage database schema changes, version control, and data migration in a structured and automated way. + +The Snowflake emulator can connect with Flyway, to apply database migrations, and manage database schema changes locally. + +## Configuring Flyway + +In this guide, you will learn how to configure Flyway to interact with the Snowflake emulator. + +### Install Flyway + +You can install Flyway on the [official Flyway website](https://flywaydb.org/download/). Download the Flyway Desktop & command-line tool and install it on your local machine. + +### Create a new Flyway project + +To create a new Flyway project, follow these steps: + +* Open the Desktop application. +* Click **New Project**. +* Enter the project name and select the database type as **Snowflake**. +* Click **Create project**. + +### Connect Flyway to Snowflake + +To connect Flyway to the Snowflake emulator, follow these steps: + +* Click **Add target database +**. +* Enter username as `test` and password as `test`. +* Enter JDBC URL as `jdbc:snowflake://http://snowflake.localhost.localstack.cloud:4566/?db=test&schema=PUBLIC&JDBC_QUERY_RESULT_FORMAT=JSON`. +* Click on **Test connection**. + +If the connection test succeeds, you can start applying database migrations using Flyway. \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/index.md b/src/content/docs/snowflake/integrations/index.md new file mode 100644 index 00000000..4b9f3e0d --- /dev/null +++ b/src/content/docs/snowflake/integrations/index.md @@ -0,0 +1,8 @@ +--- +title: Integrations +description: +--- + +# Integrations + +Develop your understanding of various LocalStack for Snowflake integrations and use cases. \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/pulumi.md b/src/content/docs/snowflake/integrations/pulumi.md new file mode 100644 index 00000000..e3e5417d --- /dev/null +++ b/src/content/docs/snowflake/integrations/pulumi.md @@ -0,0 +1,78 @@ +--- +title: Pulumi +description: Use Pulumi to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[Pulumi](https://pulumi.com/) is an Infrastructure-as-Code (IaC) framework that allows you to define and provision infrastructure using familiar programming languages. Pulumi supports a wide range of cloud providers and services, including AWS, Azure, Google Cloud, and more. + +The Snowflake emulator supports Pulumi, allowing you to define and provision Snowflake resources using the same commands and syntax as the Snowflake service. You can use Pulumi to create, update, and delete Snowflake resources locally, such as databases, schemas, tables, stages, and more. + +## Configuring Pulumi + +In this guide, you will learn how to configure Pulumi to interact with the Snowflake emulator. + +### Set up Snowflake provider + +To use Pulumi with the Snowflake emulator, you need to configure the Snowflake provider in your Pulumi configuration file. Create a blank Pulumi project, and add the following environment variables to your Pulumi stack: + +{{< command>}} +$ pulumi config set snowflake:account test +$ pulumi config set snowflake:region test +$ pulumi config set snowflake:username test +$ pulumi config set snowflake:password test +$ pulumi config set snowflake:host snowflake.localhost.localstack.cloud +{{< /command >}} + +You can install the Snowflake provider in any of the programming languages supported by Pulumi, such as Python, JavaScript, TypeScript, and Go. The following example shows how to install the Snowflake provider for your TypeScript project: + +{{< command >}} +$ npm install @pulumi/snowflake +{{< /command >}} + +### Create Snowflake resources + +You can now use Pulumi to create Snowflake resources using the Snowflake provider. The following example shows how to create a Snowflake database using Pulumi: + +```typescript +import * as snowflake from "@pulumi/snowflake"; + +const simple = new snowflake.Database("simple", { + comment: "test comment", + dataRetentionTimeInDays: 3, +}); +``` + +### Deploy the Pulumi configuration + +You can now deploy the Pulumi configuration to create the Snowflake resources locally. Run the following command to deploy the Pulumi configuration: + +{{< command >}} +$ pulumi up +{{< /command >}} + +The expected output should show the resources being created in the Snowflake emulator: + +```bash +Enter your passphrase to unlock config/secrets +Previewing update (snowflake): + Type Name Plan Info + + pulumi:pulumi:Stack pulumi-snowflake-sample-snowflake create + + └─ snowflake:index:Database simple create + +Resources: + + 2 to create + +Do you want to perform this update? yes +Updating (snowflake): + Type Name Status Info + + pulumi:pulumi:Stack pulumi-snowflake-sample-snowflake created (0.48s) 2 + +Resources: + + 2 created + +Duration: 5s +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/snow-cli.md b/src/content/docs/snowflake/integrations/snow-cli.md new file mode 100644 index 00000000..b6a4a956 --- /dev/null +++ b/src/content/docs/snowflake/integrations/snow-cli.md @@ -0,0 +1,114 @@ +--- +title: Snow CLI +description: Use Snow CLI to interact with the Snowflake emulator. +--- + +{{< preview-notice >}} + +## Introduction + +Snow CLI is a command-line interface (CLI) for Snowflake. You can use Snow CLI to interact with the Snowflake emulator. Snow CLI provides a set of commands to manage and interact with Snowflake accounts, databases, warehouses, and more. + +You can connect Snow CLI to the Snowflake emulator using a connection profile. A connection profile is a set of parameters that define the connection to a Snowflake account. You can create, list, and test connection profiles using Snow CLI. + +{{}} +Snow CLI is still under [active development](https://docs.snowflake.com/LIMITEDACCESS/snowcli/snowcli-guide), hence the commands and features might change in future releases. +{{}} + +## Installation + +You can install Snow CLI using the following methods: + +{{< tabpane >}} +{{< tab header="PyPI" lang="bash" >}} +pip install snowflake-cli-labs +snow --help +{{< /tab >}} +{{< tab header="Homebrew" lang="bash" >}} +brew tap Snowflake-Labs/snowflake-cli +brew install snowcli +snow --help +{{< /tab >}} +{{< /tabpane >}} + +## Configuring Snow CLI + +In this guide, you will learn how to configure Snow CLI to interact with the Snowflake emulator using a `localstack` connection profile. + +### Create a connection profile + +To configure Snow CLI to interact with the Snowflake emulator, create a connection profile using the following command: + +{{< command >}} +$ snow connection add \ + --connection-name localstack \ + --user test \ + --password test \ + --account test \ + --host snowflake.localhost.localstack.cloud +{{< / command >}} + +You might be prompted to enter more optional parameters, such as the connection port, database name, warehouse name, authentication method, and more. These are however optional and can be skipped. + +After a successful configuration, you can the `localstack` connection profile is ready to use. + +### List your connection profiles + +To list all the connection profiles configured in Snow CLI, execute the following command: + +{{< command >}} +$ snow connection list +{{< / command >}} + +The output should be: + +```bash ++-----------------------------------------------------------------------------------+ +| connection_name | parameters | +|-----------------+-----------------------------------------------------------------| +| localstack | {'account': 'test', 'user': 'test', 'password': '****', | +| | 'host': 'snowflake.localhost.localstack.cloud'} | ++-----------------------------------------------------------------------------------+ +``` + +### Test the connection + +To test the connection to the Snowflake emulator, execute the following command: + +{{< command >}} +$ snow connection test --connection localstack +{{< / command >}} + +The output should be: + +```bash ++--------------------------------------------------------+ +| key | value | +|-----------------+--------------------------------------| +| Connection name | localstack | +| Status | OK | +| Host | snowflake.localhost.localstack.cloud | +| Account | test | +| User | test | ++--------------------------------------------------------+ +``` + +### Run a query using the connection profile + +To run a query using the connection profile, execute the following command: + +{{< command >}} +$ snow sql --query "CREATE DATABASE mytestdb;" --connection localstack +{{< / command >}} + +You can see all the databases in your Snowflake emulator using the following command: + +{{< command >}} +$ snow sql --query "SHOW DATABASES;" --connection localstack +{{< / command >}} + +You can create a schema using the following commands: + +{{< command >}} +$ snow sql --query "CREATE SCHEMA mytestdb.mytestschema;" --connection localstack +{{< / command >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/snow-sql.md b/src/content/docs/snowflake/integrations/snow-sql.md new file mode 100644 index 00000000..c3f099c0 --- /dev/null +++ b/src/content/docs/snowflake/integrations/snow-sql.md @@ -0,0 +1,72 @@ +--- +title: SnowSQL +description: Use SnowSQL to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[SnowSQL](https://docs.snowflake.com/en/user-guide/snowsql.html) is a command-line client for Snowflake that allows you to interact with the Snowflake service using SQL commands. SnowSQL provides a wide range of features, such as executing SQL statements, loading data, unloading data, and more. + +The Snowflake emulator supports SnowSQL, allowing you to interact with the Snowflake emulator using the same commands and syntax as the Snowflake service. You can use SnowSQL to connect to the Snowflake emulator, execute SQL commands, and manage Snowflake resources locally, such as databases, schemas, tables, stages, and more. + +## Configuring SnowSQL + +In this guide, you will learn how to configure SnowSQL to interact with the Snowflake emulator. + +### Install SnowSQL + +To install SnowSQL, follow the instructions in the [official SnowSQL documentation](https://docs.snowflake.com/en/user-guide/snowsql-install-config.html). + +### Start SnowSQL + +To start SnowSQL, execute the following command: + +{{< command >}} +$ export SNOWSQL_PWD=test +$ snowsql \ + -a test \ + -u test \ + -h snowflake.localhost.localstack.cloud \ + -p 4566 \ + -d test \ + -w test \ + -r test \ + -s test +{{< / command >}} + +In the above command: + +- `-a` specifies the account name. +- `-u` specifies the username. +- `-h` specifies the host name. +- `-p` specifies the port number. +- `-d` specifies the database name. +- `-w` specifies the warehouse name. +- `-r` specifies the role name. +- `-s` specifies the schema name. + +After a successful configuration, you can use SnowSQL to interact with the Snowflake emulator. + +```bash +* SnowSQL * v1.2.32 +Type SQL statements or !help +test#test@test.test> +``` + +### Run SQL commands + +You can execute SQL commands using SnowSQL. For example, to create a new database, execute the following command: + +{{< command >}} +$ CREATE DATABASE test_db; + ++----------------------------------------+ +| status | +|----------------------------------------| +| Database TEST_DB successfully created. | ++----------------------------------------+ +0 Row(s) produced. Time Elapsed: 0.198s + +{{< / command >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/integrations/terraform.md b/src/content/docs/snowflake/integrations/terraform.md new file mode 100644 index 00000000..f3372cb9 --- /dev/null +++ b/src/content/docs/snowflake/integrations/terraform.md @@ -0,0 +1,69 @@ +--- +title: Terraform +description: Use Terraform to interact with the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +[Terraform](https://terraform.io/) is an Infrastructure-as-Code (IaC) framework developed by HashiCorp. It enables users to define and provision infrastructure using a high-level configuration language. Terraform uses HashiCorp Configuration Language (HCL) as its configuration syntax. + +The Snowflake emulator supports Terraform, allowing you to define and provision Snowflake resources using the same commands and syntax as the Snowflake service. You can use Terraform to create, update, and delete Snowflake resources locally, such as databases, schemas, tables, and stages. + +## Configuring Terraform + +In this guide, you will learn how to configure Terraform to interact with the Snowflake emulator. + +### Setup Snowflake provider + +To use Terraform with the Snowflake emulator, you need to configure the Snowflake provider in your Terraform configuration file. The following example shows how to configure the Snowflake provider: + +```hcl +terraform { + required_providers { + snowflake = { + source = "Snowflake-Labs/snowflake" + version = "= 0.92" + } + } +} + +provider "snowflake" { + account = "test" + user = "test" + password = "test" + role = "test" + host = "snowflake.localhost.localstack.cloud" +} +``` + +{{< alert type="info" title="Note" >}} +Instead of manually specifying the `host`, you can export the `SNOWFLAKE_HOST` environment variable to set the Snowflake host. Here is an example: +{{< command >}} +$ export SNOWFLAKE_HOST=snowflake.localhost.localstack.cloud +{{< /command >}} +{{< /alert >}} + +### Create Snowflake resources + +You can now use Terraform to create Snowflake resources using the Snowflake provider. The following example shows how to create a Snowflake database using Terraform: + +```hcl +resource "snowflake_database" "example" { + name = "example" + comment = "example database" + data_retention_time_in_days = 3 +} +``` + +### Deploy the Terraform configuration + +You can now deploy the Terraform configuration using the following command: + +{{< command >}} +$ terraform init +$ terraform apply +{{< /command >}} + +The `terraform init` command initializes the Terraform configuration, and the `terraform apply` command creates the Snowflake database. \ No newline at end of file diff --git a/src/content/docs/snowflake/services/cross-database-resource-sharing.md b/src/content/docs/snowflake/services/cross-database-resource-sharing.md new file mode 100644 index 00000000..a912e425 --- /dev/null +++ b/src/content/docs/snowflake/services/cross-database-resource-sharing.md @@ -0,0 +1,100 @@ +--- +title: Cross-Database Resource Sharing +description: Get started with cross-database resource sharing in the Snowflake emulator +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake data providers can easily share data from various databases using secure views. These views can include schemas, tables, and other views from one or more databases, as long as they're part of the same account. + +The Snowflake emulator supports cross-database resource sharing, allowing you to share a secure view that references objects from multiple databases. This guide walks you through the process of creating databases, schemas, tables, and views, and sharing them with other databases. + +## Getting started + +This guide is designed for users new to cross-database resource sharing and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to the Snowflake emulator using an SQL client. + +In this guide, we'll walk through a series of Snowflake SQL statements to create databases, schemas, tables, views, and a share. + +### Create databases + +Create three databases to represent the three different organizations that will share resources. In this example, we'll create databases for `db_name1`, `db_name2`, and `db_name3`. + +```sql +CREATE DATABASE db_name1_actual; +CREATE DATABASE db_name2_actual; +CREATE DATABASE db_name3_actual; +``` + +### Create schemas + +Create a schema in each database to represent the shared resources. In this example, you can create a schema called `sch` in each database. + +```sql +CREATE SCHEMA db_name1_actual.sch; +CREATE SCHEMA db_name2_actual.sch; +CREATE SCHEMA db_name3_actual.sch; +``` + +### Create tables + +Create a table in each schema to represent the shared resources. In this example, you can create a table called `table1` in `db_name1_actual.sch`, `table2` in `db_name2_actual.sch`, and `table3` in `db_name3_actual.sch`. + +```sql +CREATE TABLE db_name1_actual.sch.table1 (id INT); +CREATE TABLE db_name2_actual.sch.table2 (id INT); +CREATE TABLE db_name3_actual.sch.table3 (id INT); +``` + +### Insert Data into Tables + +You can now insert data into the tables to represent the shared resources. In this example, we'll insert a single row into each table. + +```sql +INSERT INTO db_name1_actual.sch.table1 (id) VALUES (1); +INSERT INTO db_name2_actual.sch.table2 (id) VALUES (2); +INSERT INTO db_name3_actual.sch.table3 (id) VALUES (3); +``` + +### Create Views + +You can create a view `view1` based on `table1` in `db_name1_actual`. + +```sql +CREATE VIEW db_name1_actual.sch.view1 AS SELECT * FROM db_name1_actual.sch.table1; +``` + +### Create Secure View + +You can creates a secure view `view3` in `db_name3_actual.sch` by joining data from different tables. + +```sql +CREATE SECURE VIEW db_name3_actual.sch.view3 AS +SELECT view1.id AS View1Id, table2.id AS table2id, table3.id AS table3id +FROM db_name1_actual.sch.view1 view1, db_name2_actual.sch.table2 table2, db_name3_actual.sch.table3 table3; +``` + +### Create Share and Grant Permissions + +You can create a share `s_actual` and grant usage permissions on the `db_name3_actual` database and its schema. + +```sql +CREATE SHARE s_actual; +GRANT USAGE ON DATABASE db_name3_actual TO SHARE s_actual; +GRANT USAGE ON SCHEMA db_name3_actual.sch TO SHARE s_actual; +``` + +### Query Data from Secure View + +You can now query data from the secure view `view3` in `db_name3_actual.sch`. + +```sql +SELECT * FROM db_name3_actual.sch.view3; +``` + +The expected output is: + +```plaintext +(1, 2, 3) +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/dynamic-tables.md b/src/content/docs/snowflake/services/dynamic-tables.md new file mode 100644 index 00000000..d8b21cfb --- /dev/null +++ b/src/content/docs/snowflake/services/dynamic-tables.md @@ -0,0 +1,94 @@ +--- +title: Dynamic Tables +description: Get started with Dynamic Tables in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake Dynamic Tables enable a background process to continuously load new data from sources into the table, supporting both delta and full load operations. A dynamic table automatically updates to reflect query results, removing the need for a separate target table and custom code for data transformation. This table is kept current through regularly scheduled refreshes by an automated process. + +The Snowflake emulator supports Dynamic tables, allowing you to create and manage Dynamic tables locally. The following operations are supported: + +* [`CREATE DYNAMIC TABLE`](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table) +* [`DROP DYNAMIC TABLE`](https://docs.snowflake.com/en/sql-reference/sql/drop-dynamic-table) + +## Getting started + +This guide is designed for users new to Dynamic tables and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a table, create a dynamic table, insert data into the table, and query the dynamic table. + +### Create a table + +You can create a table using the `CREATE TABLE` statement. Run the following query to create a table: + +```sql +CREATE TABLE example_table_name (id int, name text); +``` + +The output should be: + +```sql ++-----------------------------------------------+ +| status | +| ----------------------------------------------+ +| Table EXAMPLE_TABLE_NAME successfully created.| ++-----------------------------------------------+ +``` + +### Create a dynamic table + +You can create a dynamic table using the `CREATE DYNAMIC TABLE` statement. Run the following query to create a dynamic table: + +```sql +CREATE OR REPLACE DYNAMIC TABLE t_12345 + TARGET_LAG = '1 minute' WAREHOUSE = 'test' REFRESH_MODE = auto INITIALIZE = on_create + AS SELECT id, name FROM example_table_name; +``` + +The output should be: + +```sql ++-----------------------------------------------+ +| result | +| ----------------------------------------------+ +Dynamic table T_12345 successfully created. | ++-----------------------------------------------+ +``` + +### Insert data into the table + +You can insert data into the table using the `INSERT INTO` statement. Run the following query to insert data into the table: + +```sql +INSERT INTO example_table_name(id, name) VALUES (1, 'foo'), (2, 'bar'); +``` + +The output should be: + +```sql +| count | +| -----+ | +| 2 | +``` + +### Query the dynamic table + +You can query the dynamic table using the `SELECT` statement. Run the following query to query the dynamic table: + +```sql +SELECT * FROM t_12345; +``` + +The output should be: + +```sql ++----+------+ +| ID | NAME | +| ---+------+ +| 1 | foo | +| 2 | bar | ++----+------+ +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/hybrid-tables.md b/src/content/docs/snowflake/services/hybrid-tables.md new file mode 100644 index 00000000..c6c87173 --- /dev/null +++ b/src/content/docs/snowflake/services/hybrid-tables.md @@ -0,0 +1,76 @@ +--- +title: Hybrid Tables +description: Get started with Hybrid Tables in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake Hybrid tables, also known as Unistore hybrid tables, support fast, single-row operations by enforcing unique constraints for required primary keys and including indexes to speed up data retrieval. These tables are designed to optimize support for both analytical and transactional workloads simultaneously, underpinning Snowflake's Unistore architecture. + +The Snowflake emulator supports Hybrid tables, allowing you to create and manage Hybrid tables locally. The following operations are supported: + +* [`CREATE HYBRID TABLE`](https://docs.snowflake.com/en/sql-reference/sql/create-hybrid-table) +* [`DROP HYBRID TABLE`](https://docs.snowflake.com/en/sql-reference/sql/drop-hybrid-table) +* [`SHOW HYBRID TABLES`](https://docs.snowflake.com/en/sql-reference/sql/show-hybrid-tables) + +## Getting started + +This guide is designed for users new to Hybird tables and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a Hybrid table, display the Hybrid tables, and drop the Hybrid table. + +### Create a Hybrid table + +You can create a Hybrid table using the `CREATE HYBRID TABLE` statement. In this example, you can create a Hybrid table called `test-table`: + +```sql +CREATE HYBRID TABLE "test-table"(id int, name TEXT, PRIMARY KEY(id)); +``` + +The output should be: + +```sql ++------------------------------------------+ +| status | +|------------------------------------------| +| Table "test-table" successfully created. | ++------------------------------------------+ +``` + +### Show Hybrid tables + +You can display the Hybrid tables using the `SHOW HYBRID TABLES` statement: + +```sql +SHOW HYBRID TABLES LIKE 'test-table'; +``` + +The output should be: + +```sql ++-----------------------------------------------------------------------------------------------------+ +created_on |name |database_name|schema_name|comment|rows|bytes|owner |owner_role_type| +-----------------------+----------+-------------+-----------+-------+----+-----+------+---------------+ +1970-01-01 05:30:00.000|test-table|TEST |PUBLIC | |1000| 1000|PUBLIC|ROLE | ++-----------------------------------------------------------------------------------------------------+ +``` + +### Drop Hybrid table + +You can drop the Hybrid table using the `DROP HYBRID TABLE` statement: + +```sql +DROP TABLE "test-table"; +``` + +The output should be: + +```sql ++------------------------------------------+ +| status | +| -----------------------------------------+ +| TEST-TABLE successfully dropped. | ++------------------------------------------+ +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/iceberg-tables.md b/src/content/docs/snowflake/services/iceberg-tables.md new file mode 100644 index 00000000..29e0bacc --- /dev/null +++ b/src/content/docs/snowflake/services/iceberg-tables.md @@ -0,0 +1,86 @@ +--- +title: Iceberg Tables +description: +--- + +{{< preview-notice >}} + +## Introduction + +Iceberg tables uses [Apache Iceberg](https://iceberg.apache.org/) open table format specification to provide an abstraction layer on data files stored in open formats. Iceberg tables for Snowflake offer schema evolution, partitioning, and snapshot isolation to manage the table data efficiently. + +The Snowflake emulator supports Iceberg tables, allowing you to create and manage Iceberg tables locally. You can use Iceberg tables to query data in Snowflake tables using the Iceberg table format by using external volumes, with data stored in local/remote S3 buckets. + +## Getting started + +This guide is designed for users new to Iceberg tables and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create an external volume, and an Iceberg table to store and query data using the Iceberg table format. + +### Create an S3 bucket + +You can create a local S3 bucket using the `mb` command with the `awslocal` CLI. + +```bash +$ awslocal s3 mb s3://test-bucket +``` + +### Create an external volume + +You can create an external volume using the `CREATE OR REPLACE EXTERNAL VOLUME` statement. The external volume is used to define the location of the files that Iceberg will use to store the table data. + +```sql +CREATE OR REPLACE EXTERNAL VOLUME test_volume + STORAGE_LOCATIONS = ( + ( + NAME = 'aws-s3-test' + STORAGE_PROVIDER = 'S3' + STORAGE_BASE_URL = 's3://test-bucket/' + STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::000000000000:role/s3-role' + ENCRYPTION=(TYPE='AWS_SSE_S3') + ) +) +``` + +### Grant access to Snowflake role + +You can grant access to the Snowflake role using the `GRANT USAGE ON EXTERNAL VOLUME` statement. + +```sql +GRANT USAGE ON EXTERNAL VOLUME test_volume TO ROLE PUBLIC +``` + +### Create Iceberg table + +You can create an Iceberg table using the `CREATE ICEBERG TABLE` statement. The Iceberg table is used to define the schema and location of the table data. + +```sql +CREATE ICEBERG TABLE test_table (c1 TEXT) +CATALOG='SNOWFLAKE', EXTERNAL_VOLUME='test_volume', BASE_LOCATION='test' +``` + +### Insert and select data + +You can insert and select data from the Iceberg table using the `INSERT INTO` and `SELECT` statements. + +```sql +INSERT INTO test_table(c1) VALUES ('test'), ('foobar') +SELECT * FROM test_table +``` + +The output should be: + +```plaintext ++------+ +| C1 | +|------| +| test | +| foobar | ++------+ +``` + +You can also list the content of the S3 bucket: + +{{< command >}} +$ awslocal s3 ls --recursive s3://test-bucket/ +{{< / command >}} \ No newline at end of file diff --git a/src/content/docs/snowflake/services/snowpipe.md b/src/content/docs/snowflake/services/snowpipe.md new file mode 100644 index 00000000..f3667d7a --- /dev/null +++ b/src/content/docs/snowflake/services/snowpipe.md @@ -0,0 +1,122 @@ +--- +title: Snowpipe +description: Get started with Snowpipe in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowpipe allows you to load data into Snowflake tables from files stored in an external stage. Snowpipe continuously loads data from files in a stage into a table as soon as the files are available. Snowpipe uses a queue to manage the data loading process, which allows you to load data into Snowflake tables in near real-time. + +The Snowflake emulator supports Snowpipe, allowing you to create and manage Snowpipe objects in the emulator. You can use Snowpipe to load data into Snowflake tables from files stored in a local directory or a local/remote S3 bucket. The following operations are supported: + +* `CREATE PIPE` +* `DESCRIBE PIPE` +* `DROP PIPE` +* `SHOW PIPES` + +## Getting started + +This guide is designed for users new to Snowpipe and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a stage, and a pipe to load data from a local S3 bucket into a Snowflake table. + +### Create an S3 bucket + +You can create a local S3 bucket using the `mb` command with the `awslocal` CLI. + +{{< command >}} +$ awslocal s3 mb s3://test-bucket +{{< / command >}} + +### Create a stage + +You can create a stage using the `CREATE STAGE` command. The stage is used to define the location of the files that Snowpipe will load into the table. + +```sql +CREATE STAGE test_stage + URL='s3://test-bucket' + CREDENTIALS = ( + aws_key_id='test' + aws_secret_key='test' + ) + FILE_FORMAT = (TYPE = 'JSON') +``` + +### Create a table + +You can create a table using the `CREATE TABLE` command. The table is used to store the data that Snowpipe loads from the stage. + +```sql +CREATE TABLE my_test_table(record VARIANT) +``` + +### Create a pipe + +You can create a pipe using the `CREATE PIPE` command. The pipe is used to define the data loading process from the stage to the table. + +```sql +CREATE PIPE test_pipe AUTO_INGEST = TRUE + AS COPY INTO my_test_table FROM @test_stage/ FILE_FORMAT = (TYPE = 'JSON') +``` + +### Get pipe details + +You can use the `DESC PIPE` command to get the details of the pipe you created. + +```sql +DESC PIPE test_pipe +``` + +Retrieve the `notification_channel` value from the output of the `DESC PIPE` query. You will use this value to add a notification configuration to the S3 bucket. + +### Create bucket notification + +You can use the [`PutBucketNotificationConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketNotificationConfiguration.html) API to create a bucket notification configuration that sends notifications to Snowflake when new files are uploaded to the S3 bucket. + +{{< command >}} +$ awslocal s3api put-bucket-notification-configuration \ + --bucket test-bucket \ + --notification-configuration file://notification.json +{{< / command >}} + +The `notification.json` file should contain the following configuration: + +```json +{ + "QueueConfigurations": [ + { + "Id": "test-queue", + "QueueArn": "arn:aws:sqs:us-east-1:000000000000:sf-snowpipe-test", + "Events": ["s3:ObjectCreated:*"] + } + ], + "EventBridgeConfiguration": {} +} +``` + +Replace the `QueueArn` with the ARN of the queue provided in the `SHOW PIPE` query output. + +### Copy data to the stage + +Copy a JSON file to the S3 bucket to trigger the pipe to load the data into the table. Create a JSON file named `test.json` with the following content: + +```json +{"name": "Alice", "age": 30} +{"name": "Bob", "age": 42} +``` + +Upload the file to the S3 bucket: + +{{< command >}} +$ awslocal s3 cp test.json s3://test-bucket/ +{{< / command >}} + +### Check the data + +After uploading the file to the S3 bucket in the previous step, the contents of the JSON file should get inserted into the table automatically by the `test_pipe` pipe. You can check the data in the table using the following query: + +```sql +SELECT * FROM my_test_table +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/stages.md b/src/content/docs/snowflake/services/stages.md new file mode 100644 index 00000000..f17e7f7d --- /dev/null +++ b/src/content/docs/snowflake/services/stages.md @@ -0,0 +1,110 @@ +--- +title: Stages +description: Get started with Stages in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Stages are a way to load data into Snowflake. You can use stages to load data from files in a variety of formats, including CSV, JSON, and Parquet. You can also use stages to load data from external cloud storage services, such as Amazon S3, Google Cloud Storage, and Microsoft Azure Blob Storage. + +The Snowflake emulator supports stages, allowing you to load data into Snowflake using the same commands and syntax as the Snowflake service. The following operations are supported: + +- [`CREATE STAGE`](https://docs.snowflake.com/en/sql-reference/sql/create-stage.html) +- [`DESCRIBE STAGE`](https://docs.snowflake.com/en/sql-reference/sql/desc-stage) +- [`DROP STAGE`](https://docs.snowflake.com/en/sql-reference/sql/drop-stage.html) +- [`SHOW STAGES`](https://docs.snowflake.com/en/sql-reference/sql/show-stages) + +## Getting started + +This guide is designed for users new to Stages and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a database and a table for storing data. You will then create a stage to load data into the table. + +### Download the sample data + +You can download the sample data by [right-clicking on this link](./getting-started.zip) and downloading this in your machine. Unzip the file and save the contents to a directory on your local machine. + +### Create a database & table + +You can create a database using the `CREATE DATABASE` command. In this example, you can create a database called `snowflake_tutorials`: + +```sql +CREATE OR REPLACE DATABASE snowflake_tutorials; +``` + +Similarly, you can create a table using the `CREATE TABLE` command. In this example, you can create a table called `employees` in `snowflake_tutorials.public`: + +```sql +CREATE OR REPLACE TABLE employees ( + first_name STRING , + last_name STRING , + email STRING , + streetaddress STRING , + city STRING , + start_date DATE + ); +``` + +### Create a stage + +You can now create a stage using the `CREATE STAGE` command. In this example, you can create a stage called `employees_stage`: + +```sql +CREATE OR REPLACE STAGE employees_stage +FILE_FORMAT = csv; +``` + +### Upload data to the stage + +In this example, you can upload the CSV files to the table stage provided for `employees` table. + +{{< tabpane >}} +{{< tab header="Linux/macOS" lang="sql" >}} +PUT file://./employees0*.csv @@employees_stage AUTO_COMPRESS=TRUE; +{{< /tab >}} +{{< tab header="Windows" lang="sql" >}} +PUT file://C:\temp\employees0*.csv @@employees_stage AUTO_COMPRESS=TRUE; +{{< /tab >}} +{{< /tabpane >}} + +The expected output is: + +```sql ++-----------------+--------------------+-------------+-------------+--------------------+--------------------+----------+---------+ +| source | target | source_size | target_size | source_compression | target_compression | status | message | +|-----------------+--------------------+-------------+-------------+--------------------+--------------------+----------+---------| +| employees01.csv | employees01.csv.gz | 370 | 0 | NONE | GZIP | SKIPPED | | +| employees02.csv | employees02.csv.gz | 364 | 0 | NONE | GZIP | SKIPPED | | +| employees03.csv | employees03.csv.gz | 407 | 0 | NONE | GZIP | SKIPPED | | +| employees04.csv | employees04.csv.gz | 375 | 0 | NONE | GZIP | SKIPPED | | +| employees05.csv | employees05.csv.gz | 404 | 0 | NONE | GZIP | SKIPPED | | ++-----------------+--------------------+-------------+-------------+--------------------+--------------------+----------+---------+ +``` + +## Loading files from S3 + +You can also load data from an S3 bucket using the `CREATE STAGE` command. Create a new S3 bucket named `testbucket` and upload the [employees CSV files](./getting-started.zip) to the bucket. You can use LocalStack's `awslocal` CLI to create the S3 bucket and upload the files. + +{{< command >}} +awslocal s3 mb s3://testbucket +awslocal s3 cp employees0*.csv s3://testbucket +{{< /command >}} + +In this example, you can create a stage called `my_s3_stage` to load data from an S3 bucket: + +```sql +CREATE STAGE my_s3_stage +STORAGE_INTEGRATION = s3_int +URL = 's3://testbucket/' +FILE_FORMAT = csv; +``` + +You can further copy data from the S3 stage to the table using the `COPY INTO` command: + +```sql +COPY INTO mytable +FROM @my_s3_stage +PATTERN='.*employees.*.csv'; +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/streamlit.md b/src/content/docs/snowflake/services/streamlit.md new file mode 100644 index 00000000..876377b7 --- /dev/null +++ b/src/content/docs/snowflake/services/streamlit.md @@ -0,0 +1,101 @@ +--- +title: Streamlit +description: Get started with Streamlit in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake provides SQL commands to create and modify a `STREAMLIT` object. Streamlit is a Python library that allows you to create web applications with simple Python scripts. With Streamlit, you can create interactive web applications without having to learn complex web development technologies. + +The Snowflake emulator supports Streamlit, allowing you to create Streamlit applications using the same commands and syntax as the Snowflake service. The following operations are supported: + +- `CREATE STREAMLIT` +- `SHOW STREAMLITS` +- `DESCRIBE STREAMLIT` +- `ALTER STREAMLIT` +- `DROP STREAMLIT` + +## Getting started + +This guide is designed for users new to Streamlit and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client in order to execute the queries further below. + +In this guide, you will create a Streamlit application, describe the Streamlit application, and drop the Streamlit application. + +### Create an application + +You can create a Streamlit application using the `CREATE STREAMLIT` command. In this example, you can create a Streamlit application called `testapp` with a `@teststage` stage and a `test.py` script: + +```sql +CREATE STREAMLIT TESTAPP ROOT_LOCATION = '@teststage' MAIN_FILE = 'test.py'; +``` + +The output shows that the Streamlit application `testapp` was successfully created. + +```sql ++-----------------------------------------+ +| ?COLUMN? | +|-----------------------------------------| +| Streamlit TESTAPP successfully created. | ++-----------------------------------------+ +``` + +### Describe the application + +You can show the Streamlit application using the `DESCRIBE STREAMLIT` command. In this example, you can describe the Streamlit application `testapp`: + +```sql +DESCRIBE STREAMLIT testapp; +``` + +The output shows the details of the Streamlit application `testapp`. + +```sql ++---------+-------+---------------+-----------+-----------------+---------+------------------+---------------+ +| name | title | root_location | main_file | query_warehouse | url_id | default_packages | user_packages | +|---------+-------+---------------+-----------+-----------------+---------+------------------+---------------| +| TESTAPP | NULL | @teststage | test.py | NULL | testurl | ... | | +``` + +### Drop the application + +You can drop the Streamlit application using the `DROP STREAMLIT` command. In this example, you can drop the Streamlit application `testapp`: + +```sql +DROP STREAMLIT testapp; +``` + +The output shows that the Streamlit application `testapp` was successfully dropped. + +```sql ++-----------------------------------------+ +| ?COLUMN? | +|-----------------------------------------| +| Streamlit TESTAPP successfully dropped. | ++-----------------------------------------+ +``` + +## Connecting Streamlit to Snowflake emulator + +To connect to the Snowflake emulator while developing locally, Streamlit provides a way to store secrets and connection details in the project. + +To run the sample against Snowflake emulator, your local `~/.streamlit/secrets.toml` should look like this: + +```toml +[snowpark] +user = "test" +password = "test" +account = "test" +warehouse = "test" +database = "STREAMLIT_DEMO" +schema = "STREAMLIT_USER_PROFILER" +role = "test" +host = "snowflake.localhost.localstack.cloud" +``` + +## Limitations + +Currently, the Snowflake emulator supports CRUD operations to create Streamlit application entries in the Snowflake emulator, but support for hosting the Web UIs of these Streamlit apps is not yet available. + +Users can run Streamlit apps locally by using the `streamlit run main.py` command and connecting to the local Snowflake instance. \ No newline at end of file diff --git a/src/content/docs/snowflake/services/streams.md b/src/content/docs/snowflake/services/streams.md new file mode 100644 index 00000000..60299468 --- /dev/null +++ b/src/content/docs/snowflake/services/streams.md @@ -0,0 +1,89 @@ +--- +title: Streams +description: Get started with Streams in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Streams allow you to track changes made to a table. Streams capture changes made to a table, such as inserts, updates, and deletes, and store the changes in a log that you can query to see what changes have been made. + +The Snowflake emulator supports managed streams. You can create a stream locally to track changes made to an emulated Snowflake table and query the stream to see what changes have been made. + +## Getting started + +This guide is designed for users new to Streams and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to it using an SQL client to execute the queries below. + +The following sections guide you through a simple example of using Streams to track changes in a table that stores information about gym members. We will create tables to store member information and their signup dates, and then use a stream to capture changes made to the members' table. + +### Create tables + +The following SQL snippet demonstrates how to create a table named `members` to store the names and fees paid by members of a gym, and a table named `signup` to store the dates when gym members joined. + +```sql +-- Create a table to store the names and fees paid by members of a gym +CREATE TABLE IF NOT EXISTS members ( + id NUMBER(8) NOT NULL, + name VARCHAR(255) DEFAULT NULL, + fee NUMBER(3) NULL +); + +-- Create a table to store the dates when gym members joined +CREATE TABLE IF NOT EXISTS signup ( + id NUMBER(8), + dt DATE +); +``` + +### Create a Stream + +To create a stream, use the `CREATE STREAM` statement. The following example demonstrates how to create a stream named `member_check` to track changes made to the `members` table. + +```sql +CREATE STREAM IF NOT EXISTS member_check ON TABLE members; +``` + +### Insert Data + +To insert data into the `members` and `signup` tables, use the `INSERT INTO` statement. The following example demonstrates how to insert data into the `members` and `signup` tables. + +```sql +INSERT INTO members (id,name,fee) +VALUES +(1,'Joe',0), +(2,'Jane',0), +(3,'George',0), +(4,'Betty',0), +(5,'Sally',0); + +INSERT INTO signup +VALUES +(1,'2018-01-01'), +(2,'2018-02-15'), +(3,'2018-05-01'), +(4,'2018-07-16'), +(5,'2018-08-21'); +``` + +### Query Stream for Changes + +To query the stream for changes, use the `SELECT` statement. The following example demonstrates how to query the `member_check` stream for changes. + +```sql +SELECT * FROM member_check; +``` + +The expected output is: + +```sql ++----+--------+-----+-----------------+-------------------+---------------------+ +| ID | NAME | FEE | METADATA$ACTION | METADATA$ISUPDATE | METADATA$ROW_ID | +|+----+--------+-----+-----------------+-------------------+--------------------| +| 1 | Joe | 0 | INSERT | False | f05ac800-394b-4007-ab6b-28e1a915769e | +| 2 | Jane | 0 | INSERT | False | ab54a93e-3eb5-45fb-85f9-0e5f208e02dc | +| 3 | George | 0 | INSERT | False | 0e061182-fb1b-4a54-b018-61ada3feba35 | +| 4 | Betty | 0 | INSERT | False | 4dcf24c3-c25e-4e89-b0ec-cb20fbf1275c | +| 5 | Sally | 0 | INSERT | False | 1e3abb7e-f3f0-4a78-8fc1-d80e2dfdaaf7 | ++----+--------+-----+-----------------+-------------------+---------------------+ +``` diff --git a/src/content/docs/snowflake/services/tasks.md b/src/content/docs/snowflake/services/tasks.md new file mode 100644 index 00000000..b87911a4 --- /dev/null +++ b/src/content/docs/snowflake/services/tasks.md @@ -0,0 +1,63 @@ +--- +title: Tasks +description: Get started with Tasks in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Tasks are user-defined objects that enable the automation of repetitive SQL operations in Snowflake. You can use tasks to schedule SQL statements, such as queries, DDL, and DML operations, to run at a specific time or at regular intervals. + +The Snowflake emulator provides a CRUD (Create, Read, Update, Delete) interface to manage tasks. The following operations are supported: + +- [`CREATE TASK`](https://docs.snowflake.com/en/sql-reference/sql/create-task) +- [`DESCRIBE TASK`](https://docs.snowflake.com/en/sql-reference/sql/desc-task) +- [`DROP TASK`](https://docs.snowflake.com/en/sql-reference/sql/drop-task) +- [`SHOW TASKS`](https://docs.snowflake.com/en/sql-reference/sql/show-tasks) + +## Getting started + +This guide is designed for users new to Tasks and assumes basic knowledge of SQL and Snowflake. Start your Snowflake emulator and connect to the Snowflake emulator using an SQL client. + +### Create a Task + +To create a task, use the `CREATE TASK` statement. The following example demonstrates how to create a task named `test_task` that inserts a record into a table named `sample_table` every minute. + +```sql +CREATE TASK test_task +WAREHOUSE = 'test' +SCHEDULE = '1 MINUTE' +AS +INSERT INTO sample_table(ID) VALUES (123); +``` + +### Drop a Task + +To drop a task, use the `DROP TASK` statement. The following example demonstrates how to drop the `test_task` task. + +```sql +DROP TASK IF EXISTS test_task; +``` + +### Resume a Task + +To start or resume a task, use the `ALTER TASK` statement. The following example demonstrates how to resume the `test_task` task. + +```sql +ALTER TASK test_task RESUME; +``` + +### Query the table + +To query the table, use the `SELECT` statement. The following example demonstrates how to query the `sample_table` table. + +```sql +SELECT * FROM sample_table; +``` + +The expected output is: + +```plaintext +123 +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/services/user-defined-functions.md b/src/content/docs/snowflake/services/user-defined-functions.md new file mode 100644 index 00000000..aa57c909 --- /dev/null +++ b/src/content/docs/snowflake/services/user-defined-functions.md @@ -0,0 +1,65 @@ +--- +title: User-Defined Functions +description: Get started with User-Defined Functions in Node.js & Python with LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +User-Defined Functions (UDFs) are functions that you can create to extend the functionality of your SQL queries. Snowflake supports UDFs in different programming languages, including JavaScript, Python, Java, Scala, and SQL. + +The Snowflake emulator supports User-Defined Functions (UDFs) in JavaScript and Python. You can create UDFs to extend the functionality of your SQL queries. This guide demonstrates how to create and execute UDFs in JavaScript and Python. + +## JavaScript + +In the Snowflake emulator, you can create JavaScript UDFs to extend the functionality of your SQL queries. Start your Snowflake emulator and connect to it using a SQL client to execute the queries below. + +### Create a JavaScript UDF + +You can create a JavaScript UDF using the `CREATE FUNCTION` statement. The following example creates a JavaScript UDF that receives a number as input and adds 5 to it. + +```sql +CREATE OR REPLACE FUNCTION add5(n double) + RETURNS double + LANGUAGE JAVASCRIPT + AS 'return N + 5;'; +``` + +### Execute a JavaScript UDF + +You can execute a JavaScript UDF using the `SELECT` statement. The following example executes the UDF created in the previous step. + +```sql +SELECT add5(10); +``` + +The result of the query is `15`. + +## Python + +In the Snowflake emulator, you can create User-Defined Functions (UDFs) in Python to extend the functionality of your SQL queries. Start your Snowflake emulator and connect to it using a SQL client to execute the queries below. + +### Create a Python UDF + +You can create a Python UDF using the `CREATE FUNCTION` statement. The following example creates a Python UDF that takes a string as input and returns the string with a prefix. + +```sql +CREATE OR REPLACE FUNCTION sample_func(sample_arg TEXT) +RETURNS VARCHAR LANGUAGE PYTHON +RUNTIME_VERSION='3.8' HANDLER='sample_func' +AS $$ +def sample_func(i): + return 'echo: ' + i +$$; +``` + +### Execute a Python UDF + +You can execute a Python UDF using the `SELECT` statement. The following example executes the Python UDF created in the previous step. + +```sql +SELECT sample_func('foobar'); +``` + +The result of the query is `echo: foobar`. \ No newline at end of file diff --git a/src/content/docs/snowflake/snowflake-services.md b/src/content/docs/snowflake/snowflake-services.md new file mode 100644 index 00000000..45f2eb9b --- /dev/null +++ b/src/content/docs/snowflake/snowflake-services.md @@ -0,0 +1,10 @@ +--- +title: Snowflake Services +description: This is a dummy description +template: doc +sidebar: +--- + +# Snowflake Services + +Browse implemented Snowflake services and check their feature coverage. diff --git a/src/content/docs/snowflake/sql-functions.md b/src/content/docs/snowflake/sql-functions.md new file mode 100644 index 00000000..af259a4b --- /dev/null +++ b/src/content/docs/snowflake/sql-functions.md @@ -0,0 +1,12 @@ +--- +title: SQL Functions Coverage +description: This is a dummy description +template: doc +sidebar: +--- + +# SQL Functions Coverage + +Browse implemented SQL functions coverage in the following table. + +Look up all Snowflake system-defined SQL functions, scalar or table, emulated by LocalStack. \ No newline at end of file diff --git a/src/content/docs/snowflake/test/index.md b/src/content/docs/snowflake/test/index.md deleted file mode 100644 index 757864e3..00000000 --- a/src/content/docs/snowflake/test/index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: A Test page -description: Get started with LocalStack Docs. -template: doc ---- - -This is stuff \ No newline at end of file diff --git a/src/content/docs/snowflake/tooling/index.md b/src/content/docs/snowflake/tooling/index.md new file mode 100644 index 00000000..7ce33781 --- /dev/null +++ b/src/content/docs/snowflake/tooling/index.md @@ -0,0 +1,7 @@ +--- +title: Tooling +description: +--- + + +# Tooling \ No newline at end of file diff --git a/src/content/docs/snowflake/tooling/snowflake-drivers.md b/src/content/docs/snowflake/tooling/snowflake-drivers.md new file mode 100644 index 00000000..b50b5224 --- /dev/null +++ b/src/content/docs/snowflake/tooling/snowflake-drivers.md @@ -0,0 +1,155 @@ +--- +title: Snowflake Drivers +description: Get started with Snowflake Drivers in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowflake Drivers enable the use of programming languages like Go, C#, and Python for developing applications that interact with Snowflake. The Snowflake emulator facilitates testing Snowflake integration without connecting to the actual Snowflake instance. This guide provides instructions on connecting the Snowflake emulator with various drivers. + +## Snowflake Connector for Python + +The Snowflake Connector for Python (`snowflake-connector-python`) is a Python library that facilitates connecting Python programs to Snowflake databases and executing operations. Utilize this connector to link to the Snowflake emulator for testing your Snowflake integrations in Python. + +To install the Snowflake Connector for Python, execute the following command: + +{{< command >}} +$ pip install snowflake-connector-python +{{< /command >}} + +The Snowflake emulator operates on `snowflake.localhost.localstack.cloud` - note that this is a DNS name that resolves to a local IP address (`127.0.0.1`) to make sure the connector interacts with the local APIs. Connect to the emulator using the following Python code: + +```python +import snowflake.connector as sf + +conn = sf.connect( + user="test", + password="test", + account="test", + database="test", + host="snowflake.localhost.localstack.cloud", +) +``` + +Subsequently, create a warehouse named `test_warehouse`, a database named `testdb`, and a schema named `testschema` using the Snowflake Connector for Python: + +```python +conn.cursor().execute("CREATE WAREHOUSE IF NOT EXISTS test_warehouse") +conn.cursor().execute("CREATE DATABASE IF NOT EXISTS testdb") +conn.cursor().execute("USE DATABASE testdb") +conn.cursor().execute("CREATE SCHEMA IF NOT EXISTS testschema") +``` + +## Node.js Driver + +The Snowflake Node.js driver facilitates connecting to Snowflake and executing operations on Snowflake databases using Node.js. Use this driver to link to the Snowflake emulator for testing Snowflake integration. + +To install the Snowflake Node.js driver, execute the following command: + +{{< command >}} +$ npm install snowflake-sdk +{{< /command >}} + +The Snowflake emulator runs on `snowflake.localhost.localstack.cloud`. Connect to the emulator using the following JavaScript code: + +```javascript +var snowflake = require('snowflake-sdk'); +var connection = snowflake.createConnection({ + username: 'test', + password: 'test', + account: 'test', + database: 'test', + // snowflake-sdk version 1.9.3 and later supports host property and can be used instead of accessUrl like: + // host: 'snowflake.localhost.localstack.cloud', + accessUrl: 'https://snowflake.localhost.localstack.cloud', +}); +connection.connect(function(err, conn) { + if (err) { + console.error('Unable to connect: ' + err.message); + } else { + console.log('Successfully connected as id: ' + connection.getId()); + } +}); +``` + +Execute a query to create a database named `testdb` and verify the results using the following JavaScript code: + +```javascript +connection.execute({ + sqlText: 'CREATE DATABASE testdb', + complete: function(err, stmt, rows) { + if (err) { + console.error('Failed to execute statement due to the following error: ' + err.message); + } else { + console.log('Successfully executed statement: ' + stmt.getSqlText()); + } + } + }); +``` + +## Go Driver + +The Go Snowflake driver provides a way to connect to Snowflake and perform database operations using Go. You can use this driver to connect to the Snowflake emulator for testing your Snowflake integrations in Go. + +To install the Go Snowflake driver, execute the following command: + +{{< command >}} +$ go get github.com/snowflakedb/gosnowflake +{{< /command >}} + +The connection string follows the format `username:password@host:port/database?account=account_name`. For the emulator use: +`test:test@snowflake.localhost.localstack.cloud:4566/test?account=test` + +Here's an example of how to connect to the Snowflake emulator using Go: + +```go +package main + +import ( + "database/sql" + "fmt" + "log" + + _ "github.com/snowflakedb/gosnowflake" +) + +func main() { + // Connection string + connectionString := "test:test@snowflake.localhost.localstack.cloud:4566/test?account=test" + + // Connect to LocalStack Snowflake + db, err := sql.Open("snowflake", connectionString) + if err != nil { + log.Fatalf("Failed to connect to Snowflake: %v", err) + } + defer db.Close() + + // Ping the database to verify the connection + if err := db.Ping(); err != nil { + log.Fatalf("Failed to ping Snowflake: %v", err) + } + fmt.Println("Successfully connected to Snowflake!") + + // Execute a simple query + rows, err := db.Query("SELECT 123") + if err != nil { + log.Fatalf("Failed to execute query: %v", err) + } + defer rows.Close() + + // Process the result + var version string + for rows.Next() { + if err := rows.Scan(&version); err != nil { + log.Fatalf("Failed to scan row: %v", err) + } + fmt.Printf("Query result: %s\n", version) + } + + if err := rows.Err(); err != nil { + log.Fatalf("Error iterating rows: %v", err) + } +} +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/tooling/snowpark.md b/src/content/docs/snowflake/tooling/snowpark.md new file mode 100644 index 00000000..2ccd9bce --- /dev/null +++ b/src/content/docs/snowflake/tooling/snowpark.md @@ -0,0 +1,153 @@ +--- +title: Snowpark +description: Get started with Snowpark in LocalStack for Snowflake +--- + +{{< preview-notice >}} + +## Introduction + +Snowpark library is a developer library for querying and processing data at scale in Snowflake. Snowflake currently provides Snowpark libraries for three languages: Java, Python, and Scala. The Snowflake emulator facilitates testing Snowpark queries without connecting to the actual Snowflake instance. This guide provides instructions on using the Snowflake emulator in conjunction with Snowpark. + +## Snowpark for Python + +The Snowflake emulator supports the development and testing of Snowpark Python code in a local development environment. You can install the Snowpark Python library using the following command: + +{{< command >}} +$ pip install snowflake-snowpark-python +{{< /command >}} + +In this getting started guide, we'll use the Snowpark Python library to establish a connection to the Snowflake emulator and employ a DataFrame to query a table named `sample_product_data`. + +### Create a Snowpark Session + +The Snowflake emulator operates on `snowflake.localhost.localstack.cloud`. To create a Snowpark session in Python, use the following code: + +```python +from snowflake.snowpark import * +from snowflake.snowpark.functions import * + +connection_parameters = { + "user": "test", + "password": "test", + "account": "test", + "warehouse": "test", + "host": "snowflake.localhost.localstack.cloud", +} + +session = Session.builder.configs(connection_parameters).create() +``` + +### Create a table + +You can create a table named `sample_product_data` and fill the table with some data by executing SQL statements. Add the following Python code to create the table: + +```python +session.sql('CREATE OR REPLACE TABLE sample_product_data (id INT, parent_id INT, category_id INT, name VARCHAR, serial_number VARCHAR, key INT, "3rd" INT)').collect() +[Row(status='Table SAMPLE_PRODUCT_DATA successfully created.')] +session.sql(""" + INSERT INTO sample_product_data VALUES + (1, 0, 5, 'Product 1', 'prod-1', 1, 10), + (2, 1, 5, 'Product 1A', 'prod-1-A', 1, 20), + (3, 1, 5, 'Product 1B', 'prod-1-B', 1, 30), + (4, 0, 10, 'Product 2', 'prod-2', 2, 40), + (5, 4, 10, 'Product 2A', 'prod-2-A', 2, 50), + (6, 4, 10, 'Product 2B', 'prod-2-B', 2, 60), + (7, 0, 20, 'Product 3', 'prod-3', 3, 70), + (8, 7, 20, 'Product 3A', 'prod-3-A', 3, 80), + (9, 7, 20, 'Product 3B', 'prod-3-B', 3, 90), + (10, 0, 50, 'Product 4', 'prod-4', 4, 100), + (11, 10, 50, 'Product 4A', 'prod-4-A', 4, 100), + (12, 10, 50, 'Product 4B', 'prod-4-B', 4, 100) +""").collect() +``` + +You can verify that the table was created and the data was inserted by executing the following Python code: + +```python +session.sql("SELECT count(*) FROM sample_product_data").collect() +``` + +The following output should be displayed: + +```bash +[Row(COUNT=12)] +``` + +### Construct a DataFrame + +You can now construct a DataFrame from the data in the `sample_product_data` table using the following Python code: + +```python +df_table = session.table("sample_product_data") +``` + +You can see the first 10 rows of the DataFrame by executing the following Python code: + +```python +df_table.show(10) +``` + +The following output should be displayed: + +```bash +------------------------------------------------------------------------------------- +|"ID" |"PARENT_ID" |"CATEGORY_ID" |"NAME" |"SERIAL_NUMBER" |"KEY" |"3RD" | +------------------------------------------------------------------------------------- +|1 |0 |5 |Product 1 |prod-1 |1 |10 | +|2 |1 |5 |Product 1A |prod-1-A |1 |20 | +|3 |1 |5 |Product 1B |prod-1-B |1 |30 | +|4 |0 |10 |Product 2 |prod-2 |2 |40 | +|5 |4 |10 |Product 2A |prod-2-A |2 |50 | +|6 |4 |10 |Product 2B |prod-2-B |2 |60 | +|7 |0 |20 |Product 3 |prod-3 |3 |70 | +|8 |7 |20 |Product 3A |prod-3-A |3 |80 | +|9 |7 |20 |Product 3B |prod-3-B |3 |90 | +|10 |0 |50 |Product 4 |prod-4 |4 |100 | +------------------------------------------------------------------------------------- +``` + +### Transform the DataFrame + +You can perform local transformations on the DataFrame. For example, you can filter rows with the value of 'id' equal to 1: + +```python +df = session.table("sample_product_data").filter(col("id") == 1) +df.show() +``` + +The following output should be displayed: + +```bash +------------------------------------------------------------------------------------ +|"ID" |"PARENT_ID" |"CATEGORY_ID" |"NAME" |"SERIAL_NUMBER" |"KEY" |"3RD" | +------------------------------------------------------------------------------------ +|1 |0 |5 |Product 1 |prod-1 |1 |10 | +------------------------------------------------------------------------------------ +``` + +Furthermore, you can also select specific columns: + +```python +df = session.table("sample_product_data").select(col("id"), col("name"), col("serial_number")) +df.show() +``` + +The following output should be displayed: + +```bash +--------------------------------------- +|"ID" |"NAME" |"SERIAL_NUMBER" | +--------------------------------------- +|1 |Product 1 |prod-1 | +|2 |Product 1A |prod-1-A | +|3 |Product 1B |prod-1-B | +|4 |Product 2 |prod-2 | +|5 |Product 2A |prod-2-A | +|6 |Product 2B |prod-2-B | +|7 |Product 3 |prod-3 | +|8 |Product 3A |prod-3-A | +|9 |Product 3B |prod-3-B | +|10 |Product 4 |prod-4 | +--------------------------------------- +``` \ No newline at end of file diff --git a/src/content/docs/snowflake/tooling/web-user-interface.md b/src/content/docs/snowflake/tooling/web-user-interface.md new file mode 100644 index 00000000..18343b63 --- /dev/null +++ b/src/content/docs/snowflake/tooling/web-user-interface.md @@ -0,0 +1,55 @@ +--- +title: Web UI +description: Get started with LocalStack for Snowflake Web User Interface +--- + +{{< preview-notice >}} + +## Introduction + +The Snowflake emulator provides a simple web user interface (UI) accessible via a web browser. The Web UI allows you to: + +* Run SQL queries and view results using a Query Editor. +* View detailed request/response traces of API calls. +* Forward queries to a real Snowflake instance using a proxy. + +Access the Web UI at [https://snowflake.localhost.localstack.cloud/](https://snowflake.localhost.localstack.cloud/). The Web UI is available only when the Snowflake emulator is running. It does not connect to the Snowflake service (except during a proxy connection) or any other external service on the Internet. + +{{< alert title="Info" >}} +Please note that the LocalStack Snowflake Web UI is still experimental and under active development. +{{< /alert >}} + +## Getting started + +This guide is designed for users new to the Snowflake emulator Web UI. Start your Snowflake emulator using the following command: + +```bash +EXTRA_CORS_ALLOWED_ORIGINS='*' IMAGE_NAME=localstack/snowflake DEBUG=1 localstack start +``` + +The `EXTRA_CORS_ALLOWED_ORIGINS` environment variable is used to allow CORS requests from the Web UI. Navigate to [https://snowflake.localhost.localstack.cloud/](https://snowflake.localhost.localstack.cloud/) to access the Web UI. + +### Run SQL queries + +The Web UI provides a Query Editor that allows you to run SQL queries and view the results. + +Running SQL queries +

+ +You can click on **Warehouses** and **Databases** on the left side of the screen to view the available warehouses and databases. + +### View request/response traces + +The Web UI provides a detailed view of request/response traces of API calls. You can view the request and response headers, body, and status code. Click on the **Request Logs** tab in the Web UI to view the request/response traces. + +View request/response traces + +### Forward queries to a real Snowflake instance + +You can forward queries from the Snowflake emulator to a real Snowflake instance using a proxy. + +To forward queries, click on the **Proxy** tab in the Web UI and enter the Snowflake account username & password. Click on the **Save** button to save the credentials. You can now run queries in the Query Editor, and they will be forwarded to the real Snowflake instance. +{{< alert title="Important" color="danger" >}} +Be careful when operating the proxy, as it can incur costs and access data in your real Snowflake account. For security reasons, please make sure to use scoped credentials with the least set of required permissions (ideally read-only). Only run the proxy against test/staging environments, and never against a production database. +{{< /alert >}} +Forward queries to a real Snowflake instance \ No newline at end of file diff --git a/src/content/docs/snowflake/tutorials/aws-lambda-localstack-snowpark.md b/src/content/docs/snowflake/tutorials/aws-lambda-localstack-snowpark.md new file mode 100644 index 00000000..fc717568 --- /dev/null +++ b/src/content/docs/snowflake/tutorials/aws-lambda-localstack-snowpark.md @@ -0,0 +1,182 @@ +--- +title: Connecting Snowpark to Lambda using LocalStack +description: In this tutorial, you will explore how you can use Snowpark to connect to a locally running AWS Lambda using LocalStack. +template: doc +nav: +label: +--- + +## Introduction + +In this tutorial, you will explore how to connect Snowpark to AWS Lambda locally using LocalStack. Snowpark allows you to query, process, and transform data in a variety of ways using Snowpark Python. In this example, we create a Lambda function that uses Snowpark to: + +- Establish a connection to a local Snowflake database provided by LocalStack. +- Create a cursor object and execute a simple query to create a table. +- Insert rows and execute an insert query with executemany and a query to select data from the table. +- Fetch the results and execute a query to get the current timestamp. + +The code in this tutorial is available on [GitHub](https://github.com/localstack-samples/localstack-snowflake-samples/tree/main/lambda-snowpark-connector). + +## Prerequisites + +- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) with a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/) +- [LocalStack for Snowflake]({{< ref "installation" >}}) +- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) & [`awslocal` wrapper](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal) +- Python 3.10 installed locally + +## Create the Lambda function + +Create a new directory for your lambda function and navigate to it: + +{{< command >}} +$ mkdir -p lambda-snowpark +$ cd lambda-snowpark +{{< / command >}} + +Create a new file named `handler.py` and add the following code: + +```python +import snowflake.connector as sf + +def lambda_handler(event, context): + snowflake_params = { + 'user': 'test', + 'password': 'test', + 'account': 'test', + 'warehouse': 'test', + 'database': 'test', + 'schema': 'TEST_SCHEMA', + 'host': 'snowflake.localhost.localstack.cloud' + } + + # Establish a connection + connection = sf.connect(**snowflake_params) + + try: + # Create a cursor object + cursor = connection.cursor() + + # Execute the query to create a table + cursor.execute("create or replace table ability(name string, skill string )") + + # Rows to insert + rows_to_insert = [('John', 'SQL'), ('Alex', 'Java'), ('Pete', 'Snowflake')] + + # Execute the insert query with executemany + cursor.executemany("insert into ability (name, skill) values (%s, %s)", rows_to_insert) + + # Execute a query to select data from the table + cursor.execute("select name, skill from ability") + + # Fetch the results + result = cursor.fetchall() + print("Total # of rows:", len(result)) + print("Row-1 =>", result[0]) + print("Row-2 =>", result[1]) + + # Execute a query to get the current timestamp + cursor.execute("SELECT CURRENT_TIMESTAMP()") + current_timestamp = cursor.fetchone()[0] + print("Current timestamp from Snowflake:", current_timestamp) + + finally: + # Close the cursor + cursor.close() + + # Close the connection + connection.close() + + return { + 'statusCode': 200, + 'body': "Successfully connected to Snowflake and inserted rows!" + } +``` + +In the above code: + +- You import the `snowflake.connector` module to establish a connection to the Snowflake database. +- You define the `lambda_handler` function, which is the entry point for the Lambda function. +- You define the `snowflake_params` dictionary with `snowflake.localhost.localstack.cloud` as the host to connect to the Snowflake emulator. +- You establish a connection to the Snowflake database using the `sf.connect` method. +- You create a cursor object and execute a query to create a table. +- You insert rows and execute an insert query with `executemany` and a query to select data from the table. +- You fetch the results and execute a query to get the current timestamp. +- You close the cursor and the connection. + +## Install the dependencies + +You can now install the dependencies for your Lambda function. These include: + +- `snowflake-connector-python` to connect to the Snowflake database. +- `boto3` and `botocore` to interact with AWS services. + +Run the following command: + +{{< command >}} +$ pip3 install \ + --platform manylinux2010_x86_64 \ + --implementation cp \ + --only-binary=:all: --upgrade \ + --target ./libs \ + snowflake-connector-python==2.7.9 boto3==1.26.153 botocore==1.29.153 +{{< / command >}} + +## Package the Lambda function + +Package the Lambda function and its dependencies into a ZIP file. Run the following command: + +{{< command >}} +$ mkdir -p build +$ cp -r libs/* build/ +$ (cd build && zip -q -r function-py.zip .) +{{< / command >}} + +You have now created a ZIP file named `function-py.zip` that contains the Lambda function and its dependencies. + +## Start the LocalStack container + +Start your LocalStack container in your preferred terminal/shell. + +{{< command >}} +$ export LOCALSTACK_AUTH_TOKEN= +$ DEBUG=1 \ + LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=180 \ + IMAGE_NAME=localstack/snowflake \ + localstack start +{{< / command >}} + +> The `DEBUG=1` environment variable is set to enable debug logs. It would allow you to see the SQL queries executed by the Lambda function. The `LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT` environment variable is set to increase the Lambda function's timeout to 180 seconds. + +## Deploy the Lambda function + +You can now deploy the Lambda function to LocalStack using the `awslocal` CLI. Run the following command: + +{{< command >}} +$ awslocal lambda create-function \ + --function-name localstack-snowflake-lambda-example \ + --runtime python3.10 \ + --timeout 180 \ + --zip-file fileb://build/function-py.zip \ + --handler handler.lambda_handler \ + --role arn:aws:iam::000000000000:role/lambda-role +{{< / command >}} + +After successfully deploying the Lambda function, you will receive a response with the details of the function. You can now invoke the function using the `awslocal` CLI: + +{{< command >}} +$ awslocal lambda invoke --function-name localstack-snowflake-lambda-example \ + --payload '{"body": "test" }' output.txt +{{< / command >}} + +You will receive a response with the details of the invocation. You can view the output in the `output.txt` file. To see the SQL queries executed by the Lambda function, check the logs by navigating to LocalStack logs (`localstack logs`). + +```bash +2024-02-07T17:33:36.763 DEBUG --- [ asgi_gw_3] l.s.l.i.version_manager : [localstack-snowflake-lambda-example-b0813b21-ad5f-4ec7-8fb4-53147df9695e] Total # of rows: 3 +2024-02-07T17:33:36.763 DEBUG --- [ asgi_gw_3] l.s.l.i.version_manager : [localstack-snowflake-lambda-example-b0813b21-ad5f-4ec7-8fb4-53147df9695e] Row-1 => ('John', 'SQL') +2024-02-07T17:33:36.763 DEBUG --- [ asgi_gw_3] l.s.l.i.version_manager : [localstack-snowflake-lambda-example-b0813b21-ad5f-4ec7-8fb4-53147df9695e] Row-2 => ('Alex', 'Java') +2024-02-07T17:33:36.771 DEBUG --- [ asgi_gw_3] l.s.l.i.version_manager : [localstack-snowflake-lambda-example-b0813b21-ad5f-4ec7-8fb4-53147df9695e] Current timestamp from Snowflake: 2024-02-07T17:33:36 +``` + +## Conclusion + +LocalStack's core cloud emulator supports a wide range of AWS services, including Lambda, and provides a local environment for developing and testing your serverless applications. You can now use LocalStack's Snowpark emulator to connect to locally running AWS services, and develop/test your Snowpark & AWS applications locally. diff --git a/src/content/docs/snowflake/tutorials/credit-scoring-with-localstack-snowpark.md b/src/content/docs/snowflake/tutorials/credit-scoring-with-localstack-snowpark.md new file mode 100644 index 00000000..d1a0cba8 --- /dev/null +++ b/src/content/docs/snowflake/tutorials/credit-scoring-with-localstack-snowpark.md @@ -0,0 +1,162 @@ +--- +title: Credit Scoring with Snowpark & LocalStack +description: In this tutorial, you will explore how you can use LocalStack for Snowflake with Snowpark for data analysis and visualization. +template: doc +nav: +label: +--- + +## Introduction + +In this tutorial, you will learn how you can use the Snowflake emulator with Snowpark for Python and your favorite Python libraries for data analysis. + +The Jupyter Notebook and the dataset used in this tutorial are available on [GitHub](https://github.com/localstack-samples/localstack-snowflake-samples/tree/main/credit-scoring-with-snowpark). + +## Prerequisites + +- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) with a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/) +- [LocalStack for Snowflake]({{< ref "installation" >}}) +- [Snowpark]({{< ref "snowpark" >}}) with other Python libraries +- [Jupyter Notebook](https://jupyter.org/install#jupyter-notebook) + +You should also download [`credit_files.csv`](https://github.com/localstack-samples/localstack-snowflake-samples/blob/main/credit-scoring-with-snowpark/credit_files.csv) and [`credit_request.csv`](https://github.com/localstack-samples/localstack-snowflake-samples/blob/main/credit-scoring-with-snowpark/credit_request.csv) files from the LocalStack repository. The files should be present in the same directory as your Jupyter Notebook. + +## Start the Snowflake emulator + +Start your LocalStack container in your preferred terminal/shell. + +{{< command >}} +$ export LOCALSTACK_AUTH_TOKEN= +$ IMAGE_NAME=localstack/snowflake localstack start +{{< / command >}} + +## Create a Snowpark session + +The next step is to configure the Snowflake emulator. The Snowflake emulator runs on `snowflake.localhost.localstack.cloud`. You can use the Snowpark to connect to the locally running Snowflake server. + +Start Jupyter Notebook and create a new notebook. Add the following code to connect to the Snowflake emulator: + +```python +from snowflake.snowpark import * +from snowflake.snowpark.functions import * + +connection_parameters = { + "user": "test", + "password": "test", + "account": "test", + "warehouse": "test", + "host": "snowflake.localhost.localstack.cloud", +} + +session = Session.builder.configs(connection_parameters).create() +``` + +In the above configuration, you can set `user`, `password`, `account`, and `warehouse` as `test` to avoid passing any production values. You can now run Snowflake SQL queries on your local machine. + +```python +session.sql("create or replace database credit_bank").collect() +session.sql("use schema credit_bank.public").collect() +print(session.sql("select current_warehouse(), current_database(), current_schema(), current_user(), current_role()").collect()) +``` + +The following output is displayed: + +```bash +[Row(?COLUMN?='TEST', CURRENT_DATABASE='CREDIT_BANK', CURRENT_SCHEMA='public', ?COLUMN?='TEST', GET_CURRENT_ROLE='PUBLIC')] +``` + +## Create the tables + +You can now create two tables associated with this tutorial: + +- `CREDIT_FILES`: This table contains the credit on files along with the credit standing whether the loan is being repaid or if there are actual issues with reimbursing the credit. +- `CREDIT_REQUESTS`: This table contains the new credit requests that the bank needs to provide approval on. + +Run the following code to create the `credit_df` table: + +```python +import pandas as pd +credit_files = pd.read_csv('credit_files.csv') +session.write_pandas(credit_files,"CREDIT_FILES",auto_create_table='True') +credit_df = session.table("CREDIT_FILES") +credit_df.schema +``` + +The following output is displayed: + +```bash +StructType([StructField('CREDIT_REQUEST_ID', LongType(), nullable=True), StructField('CREDIT_AMOUNT', LongType(), nullable=True), StructField('CREDIT_DURATION', LongType(), nullable=True), StructField('PURPOSE', StringType(), nullable=True), StructField('INSTALLMENT_COMMITMENT', LongType(), nullable=True), StructField('OTHER_PARTIES', StringType(), nullable=True), StructField('CREDIT_STANDING', StringType(), nullable=True), StructField('CREDIT_SCORE', LongType(), nullable=True), StructField('CHECKING_BALANCE', LongType(), nullable=True), StructField('SAVINGS_BALANCE', LongType(), nullable=True), StructField('EXISTING_CREDITS', LongType(), nullable=True), StructField('ASSETS', StringType(), nullable=True), StructField('HOUSING', StringType(), nullable=True), StructField('QUALIFICATION', StringType(), nullable=True), StructField('JOB_HISTORY', LongType(), nullable=True), StructField('AGE', LongType(), nullable=True), StructField('SEX', StringType(), nullable=True), StructField('MARITAL_STATUS', StringType(), nullable=True), StructField('NUM_DEPENDENTS', LongType(), nullable=True), StructField('RESIDENCE_SINCE', LongType(), nullable=True), StructField('OTHER_PAYMENT_PLANS', StringType(), nullable=True)]) +``` + +In a similar fashion, you can create the `credit_req_df` table: + +```python +credit_requests = pd.read_csv('credit_request.csv') +session.write_pandas(credit_requests,"CREDIT_REQUESTS",auto_create_table='True') +credit_req_df = session.table("CREDIT_REQUESTS") +credit_req_df.schema +``` + +The following output is displayed: + +```bash +StructType([StructField('CREDIT_REQUEST_ID', LongType(), nullable=True), StructField('CREDIT_AMOUNT', LongType(), nullable=True), StructField('CREDIT_DURATION', LongType(), nullable=True), StructField('PURPOSE', StringType(), nullable=True), StructField('INSTALLMENT_COMMITMENT', LongType(), nullable=True), StructField('OTHER_PARTIES', StringType(), nullable=True), StructField('CREDIT_SCORE', LongType(), nullable=True), StructField('CHECKING_BALANCE', LongType(), nullable=True), StructField('SAVINGS_BALANCE', LongType(), nullable=True), StructField('EXISTING_CREDITS', LongType(), nullable=True), StructField('ASSETS', StringType(), nullable=True), StructField('HOUSING', StringType(), nullable=True), StructField('QUALIFICATION', StringType(), nullable=True), StructField('JOB_HISTORY', LongType(), nullable=True), StructField('AGE', LongType(), nullable=True), StructField('SEX', StringType(), nullable=True), StructField('MARITAL_STATUS', StringType(), nullable=True), StructField('NUM_DEPENDENTS', LongType(), nullable=True), StructField('RESIDENCE_SINCE', LongType(), nullable=True), StructField('OTHER_PAYMENT_PLANS', StringType(), nullable=True)]) +``` + +## Analyze the data + +You can now analyze the data using Snowpark and your favorite Python libraries. For example, you can fetch the first 5 rows of the `credit_df` table: + +```python +credit_df.toPandas().head() +``` + +You can similarly fetch the first 5 rows of the `credit_req_df` table: + +```python +credit_req_df.toPandas().head() +``` + +You can also visualize the numeric features and categorical features. For example, you can visualize the histogram of the `credit_df` table: + +```python +credit_df.toPandas().hist(figsize=(15,15)) +``` + +The following output is displayed: +

+ +credit_df_hist +

+ +You can also visualize the categorical features of the `credit_df` table: + +```python +import matplotlib.pyplot as plt +import seaborn as sns + +sns.set(style="darkgrid") + +fig, axs = plt.subplots(5, 2, figsize=(15, 30)) +df = credit_df.toPandas() +sns.countplot(data=df, y="PURPOSE", ax=axs[0,0]) +sns.countplot(data=df, x="OTHER_PARTIES", ax=axs[0,1]) +sns.countplot(data=df, x="CREDIT_STANDING", ax=axs[1,0]) +sns.countplot(data=df, x="ASSETS", ax=axs[1,1]) +sns.countplot(data=df, x="HOUSING", ax=axs[2,0]) +sns.countplot(data=df, x="QUALIFICATION", ax=axs[2,1]) +sns.countplot(data=df, x="SEX", ax=axs[3,0]) +sns.countplot(data=df, x="MARITAL_STATUS", ax=axs[3,1]) +sns.countplot(data=df, x="OTHER_PAYMENT_PLANS", ax=axs[4,0]) +sns.stripplot(y="PURPOSE", x="CREDIT_AMOUNT", data=df, hue='CREDIT_STANDING', jitter=True, ax=axs[4,1]) +plt.show() +``` + +The following output is displayed: +

+ +credit_df_cat + +## Conclusion + +You can now perform further experimentations with the Snowflake emulator. For example, you can use the Snowpark API to run queries to get various insights, such as determining the range of loans per different category. \ No newline at end of file diff --git a/src/content/docs/snowflake/tutorials/index.md b/src/content/docs/snowflake/tutorials/index.md new file mode 100644 index 00000000..cb8b2a09 --- /dev/null +++ b/src/content/docs/snowflake/tutorials/index.md @@ -0,0 +1,9 @@ +--- +title: Tutorials +description: +template: doc +nav: +label: +--- + +Welcome to our Snowflake tutorials! \ No newline at end of file