{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# S3-Only Backend Quick Start: Versioned Artifacts Management\n", "\n", "This guide walks you through using the versioned artifacts system's S3-only backend to manage binary artifacts. We'll use simple binary content as examples to demonstrate the concepts, but the system works with any binary files - SQLite databases, machine learning models, configuration archives, documentation snapshots, or any files you need to version.\n", "\n", "## What is the S3-Only Backend?\n", "\n", "The S3-Only Backend is a lightweight artifact management system that uses Amazon S3 as the sole storage solution for both binary content and metadata. Unlike the hybrid S3+DynamoDB approach, this backend stores everything in S3 using a structured folder hierarchy and smart filename encoding.\n", "\n", "### When to Use S3-Only Backend\n", "\n", "**Perfect for any binary artifacts:**\n", "- SQLite databases (1MB - 50MB)\n", "- Machine learning model files\n", "- Configuration archives\n", "- Documentation snapshots\n", "- Small to medium binary assets\n", "- Compiled applications\n", "- Asset bundles\n", "\n", "**Key Benefits:**\n", "- Simpler setup (only S3 required)\n", "- Cost-effective for moderate file sizes\n", "- Built-in versioning and rollback\n", "- No DynamoDB complexity\n", "\n", "## Core Concepts\n", "\n", "### Artifact\n", "\n", "Any binary file that you want to version - SQLite databases, ML models, configuration files, compiled applications, or any other binary content that needs version management.\n", "\n", "### Version\n", "\n", "An immutable snapshot of your artifact:\n", "\n", "- **LATEST**: Mutable development version (always points to newest content)\n", "- **Numbered Versions**: Immutable snapshots (1, 2, 3, ... up to 999,999)\n", "\n", "### Alias\n", "\n", "A named pointer to specific versions that enables deployment patterns:\n", "\n", "- **Simple Alias**: Points to one version (e.g., `prod` โ version 5)\n", "- **Traffic Splitting**: Routes percentage of requests between two versions for canary deployments\n", "\n", "## Example Use Case: Binary Content Versioning\n", "\n", "To demonstrate the versioned artifacts system, we'll use simple binary content as our example. Imagine you're managing application assets, configuration files, or any binary content that needs:\n", "- Version control and history\n", "- Environment-specific deployments\n", "- Rollback capabilities\n", "- Traffic splitting for testing\n", "\n", "The same principles apply to any binary artifacts - ML models, configuration bundles, compiled assets, databases, etc.\n", "\n", "## Setup and Initialization\n", "\n", "Before we can start managing artifacts, we need to set up our development environment and initialize the versioned artifacts repository. In this section, we'll use moto to create a mock AWS environment that simulates S3 services locally - this allows you to experiment with the system without needing real AWS credentials or incurring costs. We'll configure a repository that will store all our binary artifacts with proper organization and naming conventions." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from versioned.api import s3_only_backend\n", "\n", "Artifact = s3_only_backend.Alias\n", "Alias = s3_only_backend.Alias\n", "Repository = s3_only_backend.Repository" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bsm.aws_account_id = '123456789012'\n" ] } ], "source": [ "import moto\n", "from boto_session_manager import BotoSesManager\n", "\n", "# Start mocking AWS services\n", "mock_aws = moto.mock_aws()\n", "mock_aws.start()\n", "\n", "# Configure AWS session (mocked)\n", "bsm = BotoSesManager(region_name=\"us-east-1\")\n", "print(f\"{bsm.aws_account_id = }\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โ Repository initialized successfully!\n" ] } ], "source": [ "# Use a simple bucket name\n", "bucket = \"mybucket\"\n", "\n", "# Create repository for binary artifacts\n", "repo = s3_only_backend.Repository(\n", " aws_region=bsm.aws_region,\n", " s3_bucket=bucket,\n", " s3_prefix=\"artifacts\",\n", " suffix=\".bin\" # All artifacts will have .bin extension\n", ")\n", "\n", "# Initialize S3 bucket\n", "repo.bootstrap(bsm=bsm)\n", "\n", "print(\"โ Repository initialized successfully!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating Your First Artifact\n", "\n", "Now that our repository is set up, let's create and upload our first binary artifact. This section demonstrates the fundamental operation of the versioned artifacts system - taking binary content and storing it as a managed artifact. We'll create simple binary content in memory (avoiding file system operations) and upload it to the repository with metadata. This represents the core workflow you'll use whether you're managing ML models, configuration files, or any other binary content." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ฆ Created binary content: 54 bytes\n", "๐ Content preview: b'Binary content for version 1.0 - timestamp: 2024-0'...\n" ] } ], "source": [ "# Create binary content (could be any binary data)\n", "def create_binary_content(version_tag: str) -> bytes:\n", " \"\"\"Create sample binary content for demonstration\"\"\"\n", " content = f\"Binary content for {version_tag} - timestamp: 2024-01-01\"\n", " return content.encode('utf-8')\n", "\n", "# Create initial binary content\n", "artifact_name = \"my_application\"\n", "content_v1 = create_binary_content(\"version 1.0\")\n", "\n", "print(f\"๐ฆ Created binary content: {len(content_v1)} bytes\")\n", "print(f\"๐ Content preview: {content_v1[:50]}...\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='LATEST',\n", " update_at='2025-06-27T16:53:31+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/000000_LATEST.bin',\n", " sha256='88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'LATEST'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:31+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/000000_LATEST.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐ View in S3: https://console.aws.amazon.com/s3/object/mybucket?prefix=artifacts/my_application/versions/000000_LATEST.bin\n", "โ Successfully uploaded and verified (54 bytes)\n", "๐ Downloaded content matches: True\n" ] } ], "source": [ "from rich import print as rprint\n", "\n", "# Upload binary content as LATEST version\n", "artifact = repo.put_artifact(\n", " bsm=bsm, \n", " name=artifact_name, \n", " content=content_v1,\n", " content_type=\"application/octet-stream\",\n", " metadata={\n", " \"description\": \"Application binary with core functionality\",\n", " \"version\": \"1.0\",\n", " \"created_by\": \"dev_team\"\n", " }\n", ")\n", "\n", "rprint(artifact)\n", "print(f\"๐ View in S3: {artifact.s3path.console_url}\")\n", "\n", "# Verify we can download the content\n", "downloaded_content = artifact.get_content(bsm=bsm)\n", "print(f\"โ Successfully uploaded and verified ({len(downloaded_content)} bytes)\")\n", "print(f\"๐ Downloaded content matches: {downloaded_content == content_v1}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Understanding S3 Folder Structure\n", "\n", "One of the key features of the S3-only backend is its intelligent organization of artifacts in S3 storage. This section explores how the system automatically structures your artifacts using a hierarchical folder system and smart filename encoding. Understanding this structure is important because it enables features like chronological sorting, easy navigation, and efficient artifact discovery. The naming conventions ensure that your artifacts are stored in a predictable, scalable way that works well with S3's native capabilities." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ Artifacts in repository: ['my_application']\n" ] }, { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='LATEST',\n", " update_at='2025-06-27T16:53:31+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/000000_LATEST.bin',\n", " sha256='88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'LATEST'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:31+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/000000_LATEST.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "๐๏ธ S3 Folder Structure:\n", "Repository Root: s3://mybucket/artifacts/\n", "\n", "Your artifact structure:\n", "my_application/\n", "โโโ versions/\n", "โ โโโ 000000_LATEST.bin # โ Your current binary content\n", "โโโ aliases/\n", " โโโ (no aliases yet)\n", "\n", "The filename encoding ensures proper chronological sorting:\n", "- 000000_LATEST.bin: Always appears first (development version)\n", "- 999999_000001.bin: Version 1 (when published)\n", "- 999998_000002.bin: Version 2 (when published)\n", "- etc.\n", "\n" ] } ], "source": [ "# List all artifacts in the repository\n", "artifact_names = repo.list_artifact_names(bsm=bsm)\n", "print(f\"๐ Artifacts in repository: {artifact_names}\")\n", "\n", "# Get detailed information about our artifact\n", "artifact_info = repo.get_artifact_version(bsm=bsm, name=artifact_name)\n", "rprint(artifact_info)\n", "\n", "print(f\"\"\"\n", "๐๏ธ S3 Folder Structure:\n", "Repository Root: s3://{repo.s3_bucket}/{repo.s3_prefix}/\n", "\n", "Your artifact structure:\n", "{artifact_name}/\n", "โโโ versions/\n", "โ โโโ 000000_LATEST.bin # โ Your current binary content\n", "โโโ aliases/\n", " โโโ (no aliases yet)\n", "\n", "The filename encoding ensures proper chronological sorting:\n", "- 000000_LATEST.bin: Always appears first (development version)\n", "- 999999_000001.bin: Version 1 (when published)\n", "- 999998_000002.bin: Version 2 (when published)\n", "- etc.\n", "\"\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating Immutable Versions\n", "\n", "The LATEST version we created is mutable - it can be updated and changed as you develop your artifact. However, for production deployments and stable releases, you need immutable versions that never change. This section demonstrates how to \"publish\" the current LATEST version as a numbered, immutable snapshot. This is a critical concept in artifact management: development happens on LATEST, but deployments use numbered versions that provide consistency and enable reliable rollbacks." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ธ Creating immutable version 1...\n" ] }, { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='1',\n", " update_at='2025-06-27T16:53:31+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/999999_000001.bin',\n", " sha256='88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'1'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:31+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999999_000001.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'88f0dfa970d81e48f19c9dbe60b4e2e4f9cf82bc728dee82e04d13aad5076d0c'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐ Version 1 in S3: https://console.aws.amazon.com/s3/object/mybucket?prefix=artifacts/my_application/versions/999999_000001.bin\n", "\n", "๐ All versions (2 total):\n", " - Version LATEST: 2025-06-27T16:53:31+00:00 (88f0dfa9...)\n", " - Version 1: 2025-06-27T16:53:31+00:00 (88f0dfa9...)\n" ] } ], "source": [ "# Publish the LATEST version as version 1\n", "print(\"๐ธ Creating immutable version 1...\")\n", "published_version = repo.publish_artifact_version(bsm=bsm, name=artifact_name)\n", "\n", "rprint(published_version)\n", "print(f\"๐ Version 1 in S3: {published_version.s3path.console_url}\")\n", "\n", "# List all versions\n", "versions = repo.list_artifact_versions(bsm=bsm, name=artifact_name)\n", "print(f\"\\n๐ All versions ({len(versions)} total):\")\n", "for version in versions:\n", " print(f\" - Version {version.version}: {version.update_at} ({version.sha256[:8]}...)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Updating and Publishing New Versions\n", "\n", "In real development workflows, you'll continuously update your artifacts with new features, bug fixes, or improvements. This section demonstrates the iterative development cycle: updating the LATEST version with new content, then publishing stable snapshots when ready for release. This workflow separates ongoing development (which uses the mutable LATEST version) from production releases (which use immutable numbered versions), allowing you to work safely without affecting deployed systems." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ฆ Updated content size: 74 bytes\n", "๐ Size change: 20 bytes\n", "๐ Content preview: b'Binary content for version 2.0 - enhanced features'...\n" ] } ], "source": [ "# Create updated binary content\n", "content_v2 = create_binary_content(\"version 2.0 - enhanced features\")\n", "\n", "print(f\"๐ฆ Updated content size: {len(content_v2)} bytes\")\n", "print(f\"๐ Size change: {len(content_v2) - len(content_v1)} bytes\")\n", "print(f\"๐ Content preview: {content_v2[:50]}...\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ LATEST version updated\n" ] }, { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='LATEST',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/000000_LATEST.bin',\n", " sha256='5a52fc596380a9f5ea3000a553e67802fd8f43858cea6b9e7bcedd7ffeae5453'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'LATEST'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/000000_LATEST.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'5a52fc596380a9f5ea3000a553e67802fd8f43858cea6b9e7bcedd7ffeae5453'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐ธ Published version 2\n" ] }, { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='2',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/999998_000002.bin',\n", " sha256='5a52fc596380a9f5ea3000a553e67802fd8f43858cea6b9e7bcedd7ffeae5453'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'2'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999998_000002.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'5a52fc596380a9f5ea3000a553e67802fd8f43858cea6b9e7bcedd7ffeae5453'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Update the LATEST version\n", "updated_artifact = repo.put_artifact(\n", " bsm=bsm,\n", " name=artifact_name,\n", " content=content_v2,\n", " content_type=\"application/octet-stream\",\n", " metadata={\n", " \"description\": \"Enhanced application binary with new features\",\n", " \"version\": \"2.0\", \n", " \"created_by\": \"dev_team\",\n", " \"changes\": \"Added enhanced features and optimizations\"\n", " }\n", ")\n", "\n", "print(\"๐ LATEST version updated\")\n", "rprint(updated_artifact)\n", "\n", "# Publish as version 2\n", "version_2 = repo.publish_artifact_version(bsm=bsm, name=artifact_name)\n", "print(f\"๐ธ Published version 2\")\n", "rprint(version_2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with Aliases for Deployment\n", "\n", "While versions provide immutable snapshots of your artifacts, aliases provide a flexible deployment layer that maps environment names to specific versions. This section introduces aliases - named pointers that allow you to manage different environments (development, staging, production) without hardcoding version numbers in your applications. Aliases enable clean environment management and make it easy to promote versions through your deployment pipeline while maintaining clear separation between environments." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ง Development alias created\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='development',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " version='LATEST',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/000000_LATEST.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'development'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'LATEST'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/000000_LATEST.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐ Production alias created\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='production',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " version='1',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999999_000001.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'production'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'1'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999999_000001.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐งช Staging alias created\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='staging',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " version='2',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999998_000002.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'staging'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'2'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999998_000002.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Create development alias pointing to LATEST\n", "dev_alias = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"development\"\n", " # No version specified = points to LATEST\n", ")\n", "\n", "print(\"๐ง Development alias created\")\n", "rprint(dev_alias)\n", "\n", "# Create production alias pointing to stable version 1\n", "prod_alias = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=1\n", ")\n", "\n", "print(\"๐ Production alias created\")\n", "rprint(prod_alias)\n", "\n", "# Create staging alias pointing to version 2\n", "staging_alias = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"staging\", \n", " version=2\n", ")\n", "\n", "print(\"๐งช Staging alias created\")\n", "rprint(staging_alias)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Blue/Green Deployment\n", "\n", "Blue/Green deployment is a powerful deployment strategy that enables zero-downtime releases by maintaining two identical production environments and instantly switching between them. This section demonstrates how aliases make Blue/Green deployments simple and safe. By pointing your production alias from one version to another in a single atomic operation, you can instantly deploy new versions or rollback to previous ones without any service interruption. This approach minimizes deployment risk and provides immediate rollback capabilities." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ต Current production points to version: 1\n", "๐งช Staging is using version: 2\n", "\n", "๐ Performing Blue/Green deployment...\n", "โ Production switched to version 2\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='production',\n", " update_at='2025-06-27T16:53:32+00:00',\n", " version='2',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999998_000002.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'production'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:32+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'2'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999998_000002.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐ฆ Production content size: 74 bytes\n", "๐ Production content: b'Binary content for version 2.0 - enhanced features'...\n" ] } ], "source": [ "# Current production setup\n", "current_prod = repo.get_alias(bsm=bsm, name=artifact_name, alias=\"production\")\n", "print(f\"๐ต Current production points to version: {current_prod.version}\")\n", "\n", "# Simulate testing version 2 in staging\n", "staging = repo.get_alias(bsm=bsm, name=artifact_name, alias=\"staging\")\n", "print(f\"๐งช Staging is using version: {staging.version}\")\n", "\n", "# After testing passes, promote staging to production (Blue/Green switch)\n", "print(\"\\n๐ Performing Blue/Green deployment...\")\n", "new_prod = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=2 # Switch from version 1 to version 2\n", ")\n", "\n", "print(f\"โ Production switched to version {new_prod.version}\")\n", "rprint(new_prod)\n", "\n", "# Applications using the production alias now get version 2 instantly\n", "prod_content = new_prod.get_version_content(bsm=bsm)\n", "print(f\"๐ฆ Production content size: {len(prod_content)} bytes\")\n", "print(f\"๐ Production content: {prod_content[:50]}...\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Canary Deployment\n", "\n", "Canary deployment is a risk-mitigation strategy that gradually rolls out new versions by directing a small percentage of traffic to the new version while most traffic continues using the stable version. This section demonstrates the advanced alias feature of traffic splitting, which allows you to route a controlled percentage of requests between two versions. This approach enables you to test new versions with real production traffic, monitor their performance, and gradually increase adoption while maintaining the ability to quickly abort if issues arise." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ฆ Canary content size: 82 bytes\n", "๐ Content preview: b'Binary content for version 3.0 - performance optim'...\n", "๐ธ Created version 3 with optimizations\n" ] }, { "data": { "text/html": [ "
Artifact(\n", " name='my_application',\n", " version='3',\n", " update_at='2025-06-27T16:53:33+00:00',\n", " s3uri='s3://mybucket/artifacts/my_application/versions/999997_000003.bin',\n", " sha256='8e488d3f32ed90637bfc7658f674739585fe69ec4697d198cc97c25ff4a51ba8'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mArtifact\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'3'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:33+00:00'\u001b[0m,\n", " \u001b[33ms3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999997_000003.bin'\u001b[0m,\n", " \u001b[33msha256\u001b[0m=\u001b[32m'8e488d3f32ed90637bfc7658f674739585fe69ec4697d198cc97c25ff4a51ba8'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Create version 3 for canary testing\n", "content_v3 = create_binary_content(\"version 3.0 - performance optimizations\")\n", "\n", "print(f\"๐ฆ Canary content size: {len(content_v3)} bytes\")\n", "print(f\"๐ Content preview: {content_v3[:50]}...\")\n", "\n", "# Upload and publish version 3\n", "repo.put_artifact(\n", " bsm=bsm, \n", " name=artifact_name, \n", " content=content_v3,\n", " content_type=\"application/octet-stream\",\n", " metadata={\n", " \"description\": \"Optimized application binary\",\n", " \"version\": \"3.0\",\n", " \"changes\": \"Performance optimizations and bug fixes\"\n", " }\n", ")\n", "version_3 = repo.publish_artifact_version(bsm=bsm, name=artifact_name)\n", "\n", "print(f\"๐ธ Created version 3 with optimizations\")\n", "rprint(version_3)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐๏ธ Canary deployment started: 10% traffic to version 3\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='production',\n", " update_at='2025-06-27T16:53:33+00:00',\n", " version='2',\n", " secondary_version='3',\n", " secondary_version_weight=10,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999998_000002.bin',\n", " secondary_version_s3uri='s3://mybucket/artifacts/my_application/versions/999997_000003.bin'\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'production'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:33+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'2'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[32m'3'\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[1;36m10\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999998_000002.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999997_000003.bin'\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "๐ Simulating 20 requests to see traffic distribution:\n", "Version 2 (stable): 896 requests (89.6%)\n", "Version 3 (canary): 104 requests (10.4%)\n" ] } ], "source": [ "# Start canary deployment: 90% traffic to stable v2, 10% to new v3\n", "canary_alias = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=2, # Stable version (90% traffic)\n", " secondary_version=3, # Canary version (10% traffic)\n", " secondary_version_weight=10 # 10% goes to version 3\n", ")\n", "\n", "print(\"๐๏ธ Canary deployment started: 10% traffic to version 3\")\n", "rprint(canary_alias)\n", "\n", "# Simulate traffic distribution\n", "print(\"\\n๐ Simulating 20 requests to see traffic distribution:\")\n", "version_counts = {2: 0, 3: 0}\n", "\n", "for i in range(1000):\n", " selected_uri = canary_alias.random_artifact()\n", " if \"000002\" in selected_uri: # Version 2\n", " version_counts[2] += 1\n", " else: # Version 3\n", " version_counts[3] += 1\n", "\n", "print(f\"Version 2 (stable): {version_counts[2]} requests ({version_counts[2]/1000*100:.1f}%)\")\n", "print(f\"Version 3 (canary): {version_counts[3]} requests ({version_counts[3]/1000*100:.1f}%)\")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "๐ Increasing canary traffic to 50%...\n", "โ Canary successful! Completing rollout...\n", "๐ Production fully migrated to version 3\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='production',\n", " update_at='2025-06-27T16:53:33+00:00',\n", " version='3',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999997_000003.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'production'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:33+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'3'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999997_000003.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Gradually increase canary traffic\n", "print(\"\\n๐ Increasing canary traffic to 50%...\")\n", "canary_50 = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=2,\n", " secondary_version=3,\n", " secondary_version_weight=50 # Now 50% traffic to version 3\n", ")\n", "\n", "# After monitoring shows good results, complete the rollout\n", "print(\"โ Canary successful! Completing rollout...\")\n", "final_prod = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=3 # 100% traffic to version 3\n", ")\n", "\n", "print(f\"๐ Production fully migrated to version {final_prod.version}\")\n", "rprint(final_prod)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Emergency Rollback\n", "\n", "Despite careful testing, production issues can still occur after deployment. When they do, speed of recovery is critical. This section demonstrates emergency rollback procedures using aliases to instantly revert to a previous stable version. The versioned artifacts system makes rollbacks as simple as updating an alias pointer - no complex deployment processes, no waiting for builds, just an immediate switch back to a known-good version. This capability provides confidence to deploy frequently, knowing that rollback is always fast and reliable." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐จ Emergency detected! Rolling back to stable version...\n", "โก Emergency rollback completed to version 2\n" ] }, { "data": { "text/html": [ "
Alias(\n", " name='my_application',\n", " alias='production',\n", " update_at='2025-06-27T16:53:33+00:00',\n", " version='2',\n", " secondary_version=None,\n", " secondary_version_weight=None,\n", " version_s3uri='s3://mybucket/artifacts/my_application/versions/999998_000002.bin',\n", " secondary_version_s3uri=None\n", ")\n", "\n" ], "text/plain": [ "\u001b[1;35mAlias\u001b[0m\u001b[1m(\u001b[0m\n", " \u001b[33mname\u001b[0m=\u001b[32m'my_application'\u001b[0m,\n", " \u001b[33malias\u001b[0m=\u001b[32m'production'\u001b[0m,\n", " \u001b[33mupdate_at\u001b[0m=\u001b[32m'2025-06-27T16:53:33+00:00'\u001b[0m,\n", " \u001b[33mversion\u001b[0m=\u001b[32m'2'\u001b[0m,\n", " \u001b[33msecondary_version\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33msecondary_version_weight\u001b[0m=\u001b[3;35mNone\u001b[0m,\n", " \u001b[33mversion_s3uri\u001b[0m=\u001b[32m's3://mybucket/artifacts/my_application/versions/999998_000002.bin'\u001b[0m,\n", " \u001b[33msecondary_version_s3uri\u001b[0m=\u001b[3;35mNone\u001b[0m\n", "\u001b[1m)\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "โ Production verified at version 2\n", "๐ Rollback content: b'Binary content for version 2.0 - enhanced features'...\n" ] } ], "source": [ "# Simulate an emergency: rollback production to version 2\n", "print(\"๐จ Emergency detected! Rolling back to stable version...\")\n", "\n", "rollback_alias = repo.put_alias(\n", " bsm=bsm,\n", " name=artifact_name,\n", " alias=\"production\",\n", " version=2 # Instant rollback to version 2\n", ")\n", "\n", "print(f\"โก Emergency rollback completed to version {rollback_alias.version}\")\n", "rprint(rollback_alias)\n", "\n", "# Verify rollback\n", "current_prod = repo.get_alias(bsm=bsm, name=artifact_name, alias=\"production\")\n", "print(f\"โ Production verified at version {current_prod.version}\")\n", "\n", "# Check the content is correct\n", "rollback_content = current_prod.get_version_content(bsm=bsm)\n", "print(f\"๐ Rollback content: {rollback_content[:50]}...\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Lifecycle Management\n", "\n", "Over time, you'll accumulate many versions of your artifacts, and storage costs can grow if old versions aren't managed properly. This section demonstrates lifecycle management features that help you automatically clean up old versions while preserving important releases. The system provides flexible policies to keep recent versions, preserve versions based on age, and maintain versions that are still referenced by aliases. This automated cleanup helps control storage costs while ensuring you always retain the versions you need for rollbacks and historical reference." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ Total versions before cleanup: 4\n", " - Version LATEST: 2025-06-27T16:53:33+00:00\n", " - Version 3: 2025-06-27T16:53:33+00:00\n", " - Version 2: 2025-06-27T16:53:32+00:00\n", " - Version 1: 2025-06-27T16:53:31+00:00\n", "\n", "๐งน Cleanup completed at 2025-06-27 16:53:34.333589+00:00\n", "๐๏ธ Deleted 1 old versions:\n", " - Version 1\n", "โ Remaining versions: 3\n" ] } ], "source": [ "# List all current versions\n", "all_versions = repo.list_artifact_versions(bsm=bsm, name=artifact_name)\n", "print(f\"๐ Total versions before cleanup: {len(all_versions)}\")\n", "\n", "for version in all_versions:\n", " print(f\" - Version {version.version}: {version.update_at}\")\n", "\n", "# Clean up old versions (keep last 2, delete versions older than 0 seconds for demo)\n", "purge_time, deleted_versions = repo.purge_artifact_versions(\n", " bsm=bsm,\n", " name=artifact_name,\n", " keep_last_n=2, # Keep latest 2 versions + LATEST\n", " purge_older_than_secs=0 # Delete immediately for demo\n", ")\n", "\n", "print(f\"\\n๐งน Cleanup completed at {purge_time}\")\n", "print(f\"๐๏ธ Deleted {len(deleted_versions)} old versions:\")\n", "for deleted in deleted_versions:\n", " print(f\" - Version {deleted.version}\")\n", "\n", "# Verify remaining versions\n", "remaining_versions = repo.list_artifact_versions(bsm=bsm, name=artifact_name)\n", "print(f\"โ Remaining versions: {len(remaining_versions)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with Binary Content\n", "\n", "The final piece of the artifact management workflow is consuming the stored content in your applications. This section demonstrates how to retrieve binary content from aliases and versions, inspect artifact metadata, and integrate the versioned artifacts system into your application code. Whether you're loading ML models, configuration files, or other binary assets, this shows the practical patterns for accessing your managed artifacts in production systems. The examples also show how to handle different types of binary content and extract useful information for monitoring and debugging." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "๐ Production Binary Stats:\n", " - Version: 2\n", " - Size: 74 bytes\n", " - Content Type: application/octet-stream\n", " - SHA256: N/A\n", "\n", "๐ Content: Binary content for version 2.0 - enhanced features - timestamp: 2024-01-01\n", "\n", "โ Tutorial completed! You've learned to:\n", " - Upload and version binary artifacts\n", " - Create immutable snapshots \n", " - Use aliases for environment management\n", " - Perform Blue/Green deployments\n", " - Execute canary rollouts\n", " - Handle emergency rollbacks\n", " - Manage artifact lifecycle\n", "\n" ] } ], "source": [ "# Get production content\n", "prod_alias = repo.get_alias(bsm=bsm, name=artifact_name, alias=\"production\")\n", "binary_content = prod_alias.get_version_content(bsm=bsm)\n", "\n", "print(f\"\"\"\n", "๐ Production Binary Stats:\n", " - Version: {prod_alias.version}\n", " - Size: {len(binary_content)} bytes\n", " - Content Type: application/octet-stream\n", " - SHA256: {prod_alias.version.sha256 if hasattr(prod_alias.version, 'sha256') else 'N/A'}\n", "\"\"\")\n", "\n", "# Decode and display content (since we know it's text)\n", "try:\n", " decoded_content = binary_content.decode('utf-8')\n", " print(f\"๐ Content: {decoded_content}\")\n", "except UnicodeDecodeError:\n", " print(f\"๐ Binary content (first 50 bytes): {binary_content[:50]}\")\n", "\n", "print(f\"\"\"\n", "โ Tutorial completed! You've learned to:\n", " - Upload and version binary artifacts\n", " - Create immutable snapshots \n", " - Use aliases for environment management\n", " - Perform Blue/Green deployments\n", " - Execute canary rollouts\n", " - Handle emergency rollbacks\n", " - Manage artifact lifecycle\n", "\"\"\")\n", "\n", "# Clean up mocked AWS\n", "mock_aws.stop()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "The S3-Only Backend provides a powerful yet simple solution for managing any binary artifacts. Key takeaways:\n", "\n", "**โ Version Control**: Immutable snapshots ensure reliable rollbacks \n", "**โ Environment Management**: Aliases enable clean dev/staging/prod workflows \n", "**โ Deployment Patterns**: Blue/Green and canary deployments minimize risk \n", "**โ Cost Effective**: S3-only storage optimizes costs for moderate file sizes \n", "**โ Simple Operations**: No DynamoDB complexity, just S3 and smart file organization \n", "**โ Universal**: Works with any binary content - databases, models, assets, configurations\n", "\n", "This approach scales well for artifacts from 1MB to 50MB and integrates seamlessly with CI/CD pipelines for automated deployment workflows of any binary content." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 4 }