Blog Name

Zero-Knowledge Architecture for Your Lab

2026-03-13T00:00:00+00:00

Part 3 of the “Building Zero-Knowledge Biotech Infrastructure” series

In Part 1, we explained why standard cloud encryption leaves biotech data exposed during processing. In Part 2, we showed how AWS Nitro Enclaves provide hardware-enforced isolation with cryptographic attestation.

Now the practical question: how do you actually design a zero-knowledge confidential computing architecture for your biotech organization? The technology is available. The hard part is the architectural decisions that determine whether it actually protects your data or just adds complexity.

This post covers the five decisions that matter most.

Key Terms

Terms used throughout this post:

Attestation: A cryptographic process that proves exactly what code is running inside an enclave. Before releasing a decryption key, the key management service verifies a signed attestation document from the hardware itself.

PCR values: Platform Configuration Register values. Cryptographic hashes that measure the exact software stack inside an enclave: the image, the kernel, the application binary. If anything changes, the hash changes.

vsock: The single local socket channel an enclave uses to communicate with its parent instance. There is no other network interface. All data in and out travels through this channel.

KMS: Key Management Service. A cloud-hosted service for creating and controlling encryption keys. In a zero-knowledge architecture, KMS key policies use attestation values to ensure only verified enclave code can trigger decryption.

Decision 1: Who Owns the Encryption Keys in Your Zero-Knowledge Architecture?

This is the single most important architectural choice. Everything else follows from it.

There are three models:

Platform-Managed Keys

The processing platform generates and stores encryption keys. Your data is encrypted, but the platform operator retains control of the keys. On most platforms, a sufficiently privileged administrator (one with root or hypervisor-level access) can potentially inspect process memory where key material is loaded during decryption operations. They may not exercise that access, and most platforms have policies against it. But the capability exists in the infrastructure, and no cryptographic control prevents it.

What you get: Encryption at rest, encryption in transit, and the operator’s promise not to look.

What you don’t get: Independence from the operator. If the platform is compromised, subpoenaed, or acquired, your data can be decrypted by someone other than you.

When this is acceptable: When the data isn’t sensitive enough to justify the operational overhead of customer-owned keys. De-identified datasets, published reference data, non-proprietary analyses.

Customer-Managed Keys (Same Account)

You create encryption keys in the platform’s cloud account, but only you can access them through IAM policies. The platform operator has administrative access to the account but commits not to use it.

What you get: Logical separation of key access. Audit trails showing who accessed what.

What you don’t get: Mathematical guarantees. An account administrator can always modify IAM policies. Your security depends on the operator’s discipline, not cryptographic enforcement.

When this is acceptable: When you trust the operator and need simpler onboarding. Internal platforms where the “operator” is your own DevOps team.

Customer-Owned Keys (Separate Account)

You create encryption keys in your own cloud account. The processing platform never has administrative access to your keys. Key access is gated by hardware attestation: the key will only decrypt data inside a verified, unmodified enclave.

What you get: Cryptographic proof that only attested code can access your data. The platform operator cannot decrypt your data even if they wanted to, even if they’re compromised, even if they’re compelled by court order. They don’t have the keys.

What you don’t get: Simplicity. Customer-owned keys require cross-account IAM roles, attestation-based key management policies, and a coordination process when the enclave code updates. Your team needs to understand key policies, or work with someone who does.

When this is necessary: When you’re processing data where unauthorized access has existential consequences. Patient genomics, clinical trial data, proprietary drug discovery pipelines, pre-publication research.

Our Recommendation

For biotech data that matters, customer-owned keys in a separate account is the only model that provides guarantees rather than promises. Every other model has an administrator somewhere who can bypass the controls.

This is harder to set up. It creates operational friction when enclave code updates (because attestation hashes change and key policies need updating). But when a customer asks “can you access our data?” the answer is “no, and here’s the key policy that proves it.” That’s a different conversation than “no, we promise.”

Decision 2: Where Are Your Trust Boundaries?

A trust boundary is the line between “we trust the code on this side” and “we don’t trust anything on that side.” In a zero-knowledge architecture, you need to draw these lines explicitly.

The Zero-Knowledge Architecture in Practice

The diagram below shows a complete end-to-end flow for a confidential computing biotech workload: from your machine, through your own key management account, into the processing infrastructure, and back.

┌─────────────────────────────────────────────────────────────────────────────┐
│                            YOUR MACHINE                                     │
│                                                                             │
│  Client CLI                                                                 │
│  1. Encrypts file locally with a data key                                   │
│  2. Uploads ciphertext only (plaintext never leaves this machine)           │
└──────────────────────────────┬──────────────────────────────────────────────┘
                               │ ciphertext only
                               ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          YOUR CLOUD ACCOUNT                                 │
│                                                                             │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │  KMS Key  (you own it and the platform has no administrative access) │   │
│  │                                                                      │   │
│  │  Policy: kms:Decrypt permitted ONLY IF the request carries a valid   │   │
│  │  attestation document with a matching PCR0 hash.                     │   │
│  │  All other callers (including platform operators) are denied.        │   │
│  └─────────────────────────────┬────────────────────────────────────────┘   │
│                                │ attestation-gated decryption               │
│                                │ cross-account IAM role, PCR0 condition     │
└────────────────────────────────┼────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                       PROCESSING INFRASTRUCTURE                             │
│                                                                             │
│  ┌─────────────────────┐   vsock only   ┌───────────────────────────────┐   │
│  │   Parent Instance   │◄══════════════►│        Nitro Enclave          │   │
│  │   (UNTRUSTED)       │                │        (TRUSTED)              │   │
│  │                     │                │                               │   │
│  │ Pulls ciphertext    │                │ 1. Attests to your KMS key    │   │
│  │ from object storage │                │ 2. Receives decryption key    │   │
│  │                     │                │    re-encrypted for enclave   │   │
│  │ Proxies KMS calls   │                │    public key only            │   │
│  │ on behalf of        │                │ 3. Decrypts data in memory    │   │
│  │ enclave             │                │ 4. Runs analysis pipeline     │   │
│  │                     │                │ 5. Re-encrypts results        │   │
│  │ Writes encrypted    │                │ 6. Returns ciphertext         │   │
│  │ results to storage  │                │    via vsock                  │   │
│  │                     │                │                               │   │
│  │ Never sees          │                │  No network. No persistent    │   │
│  │ plaintext           │                │  storage. No SSH access.      │   │
│  └─────────────────────┘                │  No operator access.          │   │
│                                         └───────────────────────────────┘   │
│  Hypervisor enforces memory isolation at hardware level                     │
│  Platform operators cannot read enclave memory even with root access        │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ ciphertext only
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                            YOUR MACHINE                                     │
│                                                                             │
│  Client CLI retrieves and decrypts results locally                          │
│  using your own cloud credentials. Plaintext never existed outside          │
│  the enclave or your machine.                                               │
└─────────────────────────────────────────────────────────────────────────────┘

The trusted zone is minimal: the hardware, the attested enclave code, and your own encryption key. Everything else (the parent instance, the network, the platform operator’s application code, cloud provider staff) is untrusted by design. The architecture assumes those layers are compromised and still protects your data.

What This Means for Your Architecture

When you draw trust boundaries this way, several things change:

Networking becomes simpler. The enclave has no network access. You don’t need complex VPN configurations, network segmentation, or bastion hosts to protect data during processing. The enclave’s network isolation is enforced by hardware, not firewall rules.

Monitoring changes. You can observe metadata (job timing, status, resource usage) but not data content. This requires different operational patterns than traditional infrastructure where you can SSH in and inspect files.

Updates require coordination. When the enclave code changes, the attestation hash changes. If your key policy references a specific attestation hash, you need to update the policy before the new code can access your data. This is a feature (you control when to trust new code), but it’s also operational overhead.

Incident response is different. If the infrastructure outside the enclave is compromised, your data is still encrypted. You can tell your customers exactly what was and wasn’t exposed, backed by cryptographic proof rather than forensic analysis.

Decision 3: Which Confidential Computing Technology?

Not all confidential computing is equal. The three major approaches have different properties, and the right choice depends on your workload.

AWS Nitro Enclaves

A stripped-down VM carved from your EC2 instance. No network, no persistent storage, no interactive access. Communication via a single vsock channel. Attestation via the Nitro Hypervisor’s PCR measurements.

Strengths: Minimal attack surface. Full VM-level isolation. No dependency on CPU-specific features. Works with any code that runs in a Linux environment.

Limitations: No network inside the enclave (all data must be passed via vsock). Available only on AWS. Memory and CPU are carved from the parent instance, so you need to size the parent for the enclave’s needs plus your own.

Best for: Batch processing workloads with clear input/output boundaries. This maps directly to proteomics and genomics workflows: a DIA proteomics run produces a bounded set of raw instrument files (typically 1-5 GB each) that flow through a search pipeline and yield quantified protein or peptide results. The data arrives, gets processed, and leaves. No persistent enclave state is needed between steps. The same model applies to WGS variant calling pipelines, where FASTQ inputs are processed against a reference and results are written out before the enclave terminates.

Intel SGX

Application-level enclaves that protect specific code and data within a process. The CPU encrypts enclave memory, and only code running inside the enclave can access it.

Strengths: Granular protection at the application level. Smaller trusted computing base (TCB) than VM-level approaches.

Limitations: Limited enclave memory (historically capped at 128–256 MB enclave page cache; expanded in recent generations per Intel SGX documentation). Requires application code changes to use the SGX SDK. Side-channel attacks have been demonstrated against SGX enclave memory (Kocher et al., “Spectre Attacks,” IEEE S&P 2019; Van Bulck et al., “Foreshadow,” USENIX Security 2018). Performance overhead for enclave transitions.

Best for: Workloads where you need to protect specific secrets (keys, credentials) within a larger application. Less suited for processing large biotech datasets that don’t fit in enclave memory.

AMD SEV-SNP

Encrypts the entire VM’s memory using a key managed by the AMD Secure Processor. The hypervisor cannot read the VM’s memory.

Strengths: Transparent to application code. No code changes needed. Protects the entire VM memory space. Available on Azure (Confidential VMs) and GCP (Confidential Computing).

Limitations: The trust boundary is the entire VM, which has a larger attack surface than a minimal enclave. The VM has full network access, so data exfiltration through application-level bugs is still possible. Attestation model is less mature than Nitro’s.

Best for: Lift-and-shift scenarios where you want memory encryption without rewriting code. Workloads that need network access during processing.

Comparison

Dimension	Intel SGX	AMD SEV-SNP	Nitro Enclaves
Isolation scope	Application partition	Full VM	Full VM (isolated from parent)
Memory limit	128–512 MB enclave page cache	Entire VM	Allocated from parent (flexible)
Network access	Full	Full	None (vsock only)
Persistent storage	No	Full disk	None
Performance overhead	High for large datasets (memory paging)	2–5% (per AMD SEV-SNP performance benchmarks)	Near-zero compute, vsock I/O overhead
Attack surface	Smallest (app-level)	Medium (full VM)	Small (no network, no disk)
Ease of migration	Difficult (requires app partitioning)	Easy (lift-and-shift VMs)	Medium (containerize, architect for vsock)

Our Take

For biotech batch processing (which covers most genomics, proteomics, and clinical workloads), Nitro Enclaves’ restrictions are features. No network means no data exfiltration. No persistent storage means no data leakage between jobs. No interactive access means no operator snooping. The minimal attack surface is exactly what you want when the data is too sensitive for “trust me” security models.

If your workload requires interactive analysis (Jupyter notebooks, real-time dashboards, exploratory data science), Nitro Enclaves are the wrong tool. AMD SEV-SNP provides memory encryption without the network restriction, though with a larger trust boundary.

Decision 4: What Are Your Actual Threat Scenarios?

Security architecture without a threat model is just expensive plumbing. Before adopting confidential computing, be specific about what you’re protecting against.

Threats That Confidential Computing Addresses

Privileged insider access. Cloud provider employees, platform operators, or your own administrators with infrastructure access. Nitro Enclaves prevent memory inspection even with root access on the host.

Infrastructure compromise. An attacker gains access to your cloud account, object storage, or databases. With customer-owned keys and attestation, encrypted data remains encrypted. The attacker gets ciphertext they can’t decrypt.

Compelled disclosure. A court order or regulatory demand targeting the platform operator. If the operator doesn’t have the keys, they can’t comply with a decryption demand. They can produce ciphertext and metadata, but not plaintext.

Supply chain attacks. Compromised dependencies, malicious container images, or tampered build pipelines. Attestation detects code changes: if the enclave code is modified, the attestation hash changes, and the key management service refuses to release the key.

Threats That Confidential Computing Doesn’t Address

Application logic bugs and Iago attacks. Attestation proves that a specific, verified binary is running inside the enclave. It says nothing about whether that code is correct or resistant to manipulation. A class of attacks known as Iago attacks exploits this boundary directly: a compromised host manipulates the return values of system calls made by the enclave, causing the enclave application to behave against its own interests, without ever breaking hardware isolation. The enclave is running exactly the attested code. The attested code is being fed malicious inputs it doesn’t validate. For a genomics pipeline, this could mean corrupted input dimensions that produce silently wrong variant calls, with no cryptographic alarm raised.

Malicious or vulnerable code in the enclave itself. This is the logical complement to Iago attacks. Attestation proves what code is running, not whether that code is trustworthy. If vulnerable or malicious code is packaged into the enclave image, it will be attested and run with full access to decrypted data. The hardware did its job. The code didn’t. This shifts the security boundary in a fundamental way: confidential computing doesn’t eliminate the need to trust the enclave code. It makes that code the only thing you need to trust. Which means code provenance and verification become your new perimeter.

A rigorous verification pipeline before any enclave image is trusted should include formal code audit, thorough code review, static code analysis (SAST), Docker image security scanning, and independent PCR0 hash verification against the published enclave image.

Data at rest and in transit. Confidential computing protects data during processing. You still need encryption at rest and in transit. These are complementary, not alternatives.

Authorized misuse. A legitimate user with valid credentials who misuses their access. Confidential computing doesn’t solve authorization problems. If someone is authorized to submit jobs and receive results, they can process any data they have access to.

Side-channel attacks on hardware. Academic research has demonstrated side-channel attacks against some confidential computing technologies, notably Intel SGX (Spectre, Foreshadow, and subsequent variants). Nitro Enclaves have a smaller attack surface, but no hardware is immune to all possible physical attacks. The practical question is whether an attack requires physical hardware access (very difficult for cloud infrastructure) or can be mounted remotely.

Right-Sizing Your Security

Not every dataset needs zero-knowledge processing. The overhead (operational complexity, update coordination, debugging limitations) is justified when:

Unauthorized access triggers legal and contractual exposure (consult your compliance team on applicable regulations and data processing obligations)
The data has competitive value (proprietary sequences, drug candidates, unpublished research)
Your customers require cryptographic proof of protection (pharma sponsors, clinical partners)
You can’t afford a breach investigation that runs for months

For published reference genomes, synthetic test data, or de-identified population statistics, standard cloud security with proper access controls is sufficient and much simpler.

Decision 5: How Will You Handle the Operational Trade-offs?

Adopting zero-knowledge architecture means accepting specific constraints. Plan for them before you commit.

You Can’t Debug by Inspecting Data

When a job fails inside an enclave, you can see error codes, log messages, resource utilization, and timing information. You cannot inspect the input data, intermediate results, or output. This changes how you troubleshoot.

Practical approach: Build comprehensive logging into your pipeline code. Log data shapes (dimensions, row counts, file sizes) rather than data content. Create synthetic test datasets that exercise the same code paths as real data, so you can reproduce issues outside the enclave.

Updates Require Coordination

Changing the enclave code changes the attestation hash. Every customer whose key policy references the old hash needs to update their policy before the new code can process their data.

Practical approach: Plan a transition window where both old and new enclave versions run simultaneously. Notify customers in advance. Provide tooling that makes the policy update a single command, not a manual console operation.

Resource Sizing Is Inflexible

Enclave resources (CPU, memory) are allocated at launch and can’t grow. If your pipeline needs more memory than declared, the job fails.

Practical approach: Profile your workloads on representative data. In proteomics search pipelines (MSFragger, MaxQuant), peak memory during database search typically runs 2-3x the raw file size when searching against a full human proteome database: a 4 GB raw file will require 16-24 GB at peak, not 4 GB. WGS variant calling pipelines (BWA-MEM2, GATK HaplotypeCaller) against 30x coverage inputs commonly peak at 32-64 GB depending on the reference and ploidy. Size for your largest expected input with 25-50% headroom. Extreme outlier inputs may need a separate resource profile rather than sizing everything for the worst case.

Vendor Lock-in Is Real

Nitro Enclaves are AWS-specific. Intel SGX is Intel-specific. AMD SEV-SNP runs on AMD hardware. Your confidential computing choice ties you to a platform.

Practical approach: Keep your pipeline code portable. Use standard tools (Nextflow, Snakemake, standard bioinformatics packages). The enclave is the execution environment; the pipeline logic should run anywhere. If you need to switch platforms, the migration cost should be the enclave wrapper, not the analysis code.

Putting Your Confidential Computing Architecture Together

Key ownership is the one decision you cannot patch later. Technology choice, threat model scoping, and operational trade-offs can all evolve as your architecture matures. Trust boundaries can be redrawn. But if the wrong party holds the encryption keys from the start, no amount of hardware isolation recovers from it. Get that decision right, and the rest of the architecture follows.

AWS Nitro Enclaves for Biotech Data

2026-03-01T00:00:00+00:00

Part 2 in the “Building Zero-Knowledge Biotech Infrastructure” series

In Part 1, we explained why standard cloud security fails for biotech: your data sits in plaintext memory during processing, and the cloud provider’s staff can access it. Encryption at rest and in transit protects data everywhere except where it matters most.

This post covers the fix. AWS Nitro Enclaves create hardware-isolated virtual machines where your genomics pipeline, mass spec analysis, or clinical data processing runs in memory that the cloud provider cannot access. Not “won’t access per policy.” Cannot access, enforced by hardware.

We chose Nitro Enclaves after evaluating Intel SGX, AMD SEV-SNP, and the confidential computing offerings from Azure and GCP. Here’s why, and how the architecture works in practice for life sciences workloads.

What Makes Nitro Enclaves Different

Most confidential computing approaches encrypt VM memory (AMD SEV) or create application-level enclaves (Intel SGX). AWS Nitro Enclaves take a different path: they carve out a completely isolated virtual machine from your EC2 instance, with no network access, no persistent storage, and no interactive login.

That sounds restrictive. It is. And that’s the point.

A Nitro Enclave is a stripped-down VM that runs inside your EC2 instance. When you launch it, the Nitro Hypervisor allocates dedicated CPU cores and memory from the parent instance. Those resources become inaccessible to the parent. The hypervisor enforces this boundary in hardware. No software on the parent instance, no AWS operations engineer, no privileged process can read the enclave’s memory.

The enclave communicates with the parent through exactly one channel: a local socket connection called vsock. That’s it. No TCP/IP networking. No disk. No SSH. This minimal attack surface is what makes the security model credible.

The Architecture in Practice

┌─────────────────────────────────────────────────────────┐
|                    EC2 Instance                         |
|                                                         |
|  ┌──────────────────┐    vsock    ┌──────────────────┐  |
|  |   Parent VM      |<===========>|   Nitro Enclave  |  |
|  |                  |             |                  |  |
|  | - Network access |             | - No network     |  |
|  | - Disk access    |             | - No disk        |  |
|  | - SSH access     |             | - No SSH         |  |
|  | - Proxy to S3,   |             | - Your pipeline  |  |
|  |   KMS, etc.      |             |   runs here      |  |
|  └──────────────────┘             └──────────────────┘  |
|                                                         |
|  Nitro Hypervisor (hardware-enforced isolation)         |
└─────────────────────────────────────────────────────────┘

The parent VM handles all I/O: pulling encrypted data from S3, proxying requests to AWS KMS for key management, and writing encrypted results back to storage. The enclave handles all computation on decrypted data. The two communicate over vsock.

This separation means the parent never sees plaintext data. It handles only ciphertext. The enclave handles plaintext but has no way to exfiltrate it (no network, no disk). The only output path is vsock back to the parent, and your application controls what goes over that channel.

Cryptographic Attestation: Proving What Code Is Running

Hardware isolation is half the story. The other half is proving that the enclave is running exactly the code you expect.

Every Nitro Enclave contains a Nitro Security Module (NSM), a hardware component that generates cryptographic attestation documents. When your enclave starts, the NSM measures everything about the environment: the enclave image, the kernel, the application code. These measurements become PCR (Platform Configuration Register) values, essentially cryptographic hashes of the exact software stack running inside the enclave.

The attestation document is signed by the NSM using a key that chains back to the AWS Nitro Attestation Root CA. You can verify this chain independently. The document contains:

PCR0: Hash of the enclave image (your entire application package)
PCR1: Hash of the Linux kernel running inside the enclave
PCR2: Hash of the application binary
PCR3: Hash of the IAM role attached to the parent instance
A public key generated inside the enclave (the private key never leaves)
A timestamp and nonce for freshness

This is the foundation for zero-trust key management. Instead of trusting people or organizations, you trust code, and you verify that trust cryptographically.

Zero-Trust Key Management with KMS

Here’s where it gets practical for biotech workloads. AWS KMS supports condition keys that reference attestation values. You can create a KMS key policy that says: “Only allow decryption when the request comes from an enclave with this exact PCR0 value.”

The flow works like this:

Your genomics data sits encrypted in S3, protected by a KMS key
The parent instance pulls the encrypted data and sends it to the enclave via vsock
The enclave requests the decryption key from KMS, attaching its attestation document
KMS validates the attestation: Is the signature valid? Does PCR0 match the policy?
If valid, KMS decrypts the data key and re-encrypts it with the enclave’s public key (from the attestation document)
The enclave decrypts using its private key, which was generated inside the enclave and has never existed anywhere else
Your pipeline processes the data
Results are encrypted with an output key before being sent back through vsock

The critical detail: KMS re-encrypts the response using the enclave’s public key. The parent instance proxies the KMS request and response, but it only ever sees ciphertext. The plaintext data key exists only inside the enclave’s memory.

If someone modifies the enclave image (changes the analysis code, adds a data exfiltration step, swaps in a different application), the PCR0 hash changes. The KMS policy rejects the request. The data stays encrypted.

This is cryptographic proof, not a compliance checkbox. You can audit the enclave image, verify its hash matches the KMS policy, and know mathematically that only that specific code can access the decryption keys.

What This Means for Biotech Workloads

Genomics: Variant Calling in an Enclave

A typical secure genomics pipeline using Nitro Enclaves (building on the Nextflow/nf-core patterns many teams already use):

S3 (encrypted FASTQ/BAM)
    |
    v
Parent Instance (pulls encrypted files)
    |
    v [vsock - encrypted data]
Nitro Enclave:
    1. Gets decryption key from KMS (attested)
    2. Decrypts genomic data
    3. Runs alignment (BWA-MEM2) + variant calling (GATK)
    4. Encrypts VCF output with results key
    5. Sends encrypted results via vsock
    |
    v
Parent Instance (writes to S3)
    |
    v
S3 (encrypted VCF/results)

At no point does the parent instance see decrypted patient genomes. The alignment and variant calling happen entirely inside the enclave. The output VCF is encrypted before it leaves.

For organizations processing patient-derived whole genome sequencing data, this architecture satisfies the security requirements that keep legal teams from approving cloud workloads. The data is protected during the processing step that traditional encryption misses.

Proteomics: Secure Mass Spec Analysis

Mass spectrometry raw files from drug discovery contain proprietary biomarker signatures worth years of R&D investment. The same enclave architecture protects this IP:

Raw mzXML/mzML files encrypted at rest in S3
Decryption only inside an attested enclave
Peptide identification and quantification run in isolated memory
Results encrypted before leaving the enclave

The performance characteristics matter here. Nitro Enclaves run on dedicated CPU cores carved from the parent instance. Compute-bound tasks like spectral matching and database searching run at near-native speed. The overhead comes from data transfer through vsock, which adds latency for the initial data load but doesn’t affect processing speed once data is in enclave memory.

Multi-Party Collaboration Without Trust

The most compelling use case for biotech is collaborative analysis where neither party trusts the other with raw data.

Two pharmaceutical companies want to run a combined analysis on their patient cohorts. Neither wants to share raw data. With Nitro Enclaves:

Both companies agree on the analysis code and audit it
Both set KMS policies restricting their keys to enclaves running that exact code (verified by PCR0)
The enclave decrypts both datasets, runs the combined analysis
Each company receives encrypted aggregate results, encrypted with their own output key
Neither company’s raw data leaves the enclave. Neither company needs to trust the other, or the infrastructure operator

This pattern eliminates the “clean room” problem that blocks multi-site clinical collaborations and competitive pre-competitive research. The trust is in the code, verified by hardware, not in legal agreements between organizations.

How Nitro Enclaves Compare to Alternatives

We evaluated three confidential computing approaches before choosing Nitro Enclaves for biotech workloads:

Dimension	Intel SGX	AMD SEV-SNP	AWS Nitro Enclaves
Isolation scope	Application partition	Full VM	Full VM (isolated from parent)
Memory limit	128-512 MB enclave page cache	Entire VM	Allocated from parent (flexible)
Network access	Full	Full	None (vsock only)
Persistent storage	No	Full disk	None
Performance overhead	High for large datasets (memory paging)	2-5%	Near-zero compute, vsock I/O overhead
Attack surface	Smallest (app-level)	Medium (full VM)	Small (no network, no disk)
Ease of migration	Difficult (requires app partitioning)	Easy (lift-and-shift VMs)	Medium (containerize, architect for vsock)

Why SGX didn’t work for us: The enclave page cache (EPC) is limited to 128-512 MB. Genomics and proteomics datasets routinely exceed this. When data spills beyond the EPC, performance degrades dramatically due to memory paging, sometimes by 10-100x. Intel has improved this with SGX2, but the fundamental memory constraint makes it impractical for large scientific workloads.

Why SEV-SNP is worth considering: AMD’s approach encrypts the entire VM memory with minimal performance impact. It’s the easiest migration path: take your existing VM, enable SEV-SNP, and your memory is encrypted. The tradeoff is a larger attack surface. The VM still has networking, storage, and a full operating system. For organizations that need confidential computing with minimal re-architecture, it’s a strong option. Azure and GCP both offer SEV-SNP based confidential VMs.

Why we chose Nitro Enclaves: For our proof-of-concept, we needed a technology where the security model is real, not a configuration promise that degrades the moment someone misconfigures an IAM policy or a network ACL. Nitro Enclaves deliver that. The isolation isn’t enforced by software rules that an admin can override — it’s enforced by the Nitro Hypervisor at the hardware level. There is no network interface to misconfigure because none exists. There is no disk to accidentally leave unencrypted because the enclave has no storage. And the KMS attestation policy doesn’t check whether you claim to be running approved code — it verifies the cryptographic hash of the actual binary in memory. That distinction matters for a PoC. We wanted to demonstrate to biotech clients that confidential computing isn’t security theater layered on top of standard cloud infrastructure. It’s a fundamentally different trust model: the math proves what code is running, the hardware enforces what that code can access, and neither the cloud operator nor a compromised admin can circumvent it. The architectural work is real (you have to design for vsock communication), but for high-sensitivity biotech data, the security properties justify the engineering investment.

Practical Limitations You Should Know

Nitro Enclaves aren’t a drop-in solution. The constraints that make them secure also make them harder to work with:

No persistent storage. Everything in the enclave exists only in memory. If the enclave crashes, intermediate results are gone. Your application needs to stream results to the parent via vsock for durable storage. For long-running genomics pipelines, this means building checkpointing into your vsock communication protocol.

No network access. Every external service call (S3, KMS, databases, reference genome downloads) must go through the parent as a vsock proxy. AWS provides a vsock proxy for KMS, but other services require custom proxy code.

Memory constraints. The enclave shares memory with the parent instance. A r5.4xlarge (128 GiB) might allocate 96 GiB to the enclave, leaving 32 GiB for the parent proxy. For memory-intensive bioinformatics tools (BWA-MEM, STAR, large protein databases), you need to plan instance sizing carefully. There’s no swap. Out-of-memory kills the enclave process.

Debugging is painful. You cannot SSH into the enclave. No interactive console, no debugger attachment. You get stdout/stderr through nitro-cli console, but only in debug mode. And debug mode changes PCR values, which breaks attestation policies. The development workflow is: build outside the enclave, test inside with debug mode, then switch to production mode with updated KMS policies.

One enclave per instance. Each EC2 instance supports a single enclave. You can’t run multiple isolated workloads on the same instance.

These are real engineering challenges. We’ve built a vsock proxy layer that handles KMS, S3, and custom service endpoints through a unified API, along with a checkpointing system for long-running pipeline stages. That tooling took months to get right. Organizations should budget for the architectural work, especially the vsock communication layer and the proxy infrastructure for external services.

Getting Started

If your organization is evaluating confidential computing for genomics, proteomics, or clinical workloads, here’s a practical starting path:

Step 1: Identify your highest-sensitivity workload. Don’t boil the ocean. Pick one pipeline that processes data your legal team won’t approve for standard cloud deployment.

Step 2: Containerize it. Nitro Enclaves are built from Docker images. If your pipeline already runs in a container, you’re halfway there. If it doesn’t, containerization is the prerequisite.

Step 3: Prototype the vsock communication. Build a simple proof-of-concept that sends data to the enclave and receives results. The AWS Nitro Enclaves SDK and the vsock proxy are the starting points.

Step 4: Set up KMS attestation. Create a KMS key with a policy that requires enclave attestation. This is where the zero-trust model clicks into place. Once you see your KMS policy rejecting requests from non-attested callers, the security model becomes concrete.

These steps get you to a working prototype. But the engineering is only half the challenge. Before you commit to an architecture, you need to make harder decisions: who owns the encryption keys, where you draw trust boundaries, and how you’ll handle the operational overhead of attestation-based systems.

In Part 3, we cover the five architectural decisions that determine whether confidential computing actually protects your data or just adds complexity.

Why Biotech Can't Trust the Cloud

2026-02-03T00:00:00+00:00

Part 1 of 3 in the “Building Zero-Knowledge Biotech Infrastructure” series

Most biotech organizations can’t use cloud computing for their most valuable data. Not because the cloud isn’t technically capable (AWS could process 50TB of whole-genome sequencing data in days instead of months). Not because teams don’t want to (IT and bioinformatics groups know the operational benefits).

They can’t use it because legal and compliance teams won’t sign off and they’re right to refuse.

The standard cloud security model (encrypt data at rest, encrypt data in transit, trust the provider’s compliance certifications) doesn’t actually protect data during processing. When your variant calling pipeline runs, or your mass spec analysis executes, or your imaging segmentation processes, the data sits in plaintext memory on infrastructure the cloud provider controls. Their hypervisor can access it. Their operations staff can access it. Anyone who compromises those systems can access it.

For genomics, proteomics, clinical trials, and manufacturing data, that’s not acceptable. Patient genomes are permanent identifiers. Biomarker signatures represent years of R&D investment. Manufacturing processes are core IP. Clinical trial data carries regulatory liability.

This post explains why the current cloud trust model fails for biotech. In Part 2, we’ll show you how confidential computing changes the equation by making data processing cryptographically verifiable and hardware-isolated.

Why Biotech Data Is Different

Biotech data isn’t like other enterprise data. The sensitivity spans every “-omics” domain and imaging modality, but for different reasons. What unites them: you can’t afford a breach, you can’t undo exposure, and standard cloud security doesn’t address the actual threats.

Genomics: The Permanent Identifier

Genomic data is the ultimate personally identifiable information. Unlike passwords or credit cards, a genome can’t be changed or reissued after a breach.

A leaked genome identifies you and your relatives, reveals disease predispositions, and remains compromised permanently. There’s no remediation path. The data exposure is irreversible.

This isn’t theoretical. In October 2023, 23andMe disclosed that hackers accessed genetic data from 6.9 million users. The attackers used credential stuffing to breach individual accounts, then exploited the “DNA Relatives” feature to access data from millions of connected profiles. The stolen data (including ancestry information, birth years, and genetic heritage) appeared on hacking forums within weeks.

Proteomics: The Billion-Dollar Biomarker

Mass spectrometry data from proteomics studies often represents years of biomarker discovery work. A leaked proteomic signature for early cancer detection could:

Give competitors a multi-year head start
Invalidate pending patents worth hundreds of millions
Compromise ongoing clinical validation studies

Unlike genomics, proteomics data changes over time, which makes longitudinal studies even more valuable and their protection more critical.

Imaging: Regulatory and Technical Challenges

Pathology slides, radiology images, and microscopy data present unique challenges:

Patient re-identification: AI can identify individuals from medical images even without metadata
Diagnostic liability: Leaked images could be used to challenge clinical decisions
GxP compliance: Images used in drug submissions must maintain chain of custody

A single high-resolution pathology slide from a clinical trial can be 2-5 GB and contain enough information to identify the patient, the disease state, and potentially compromise the trial’s blinding.

Cell and Gene Therapy: Manufacturing Secrets

Cell therapy combines multiple sensitivity concerns:

Patient-derived materials: Autologous therapies use the patient’s own cells
Manufacturing processes: Differentiation protocols and expansion conditions are core IP
Batch records: Single-cell sequencing of manufacturing intermediates reveals process performance

When your product literally contains patient cells, the line between patient data and manufacturing data disappears entirely.

Clinical Trials: The Regulatory Gauntlet

For research institutions and pharmaceutical companies, clinical data carries the highest stakes:

21 CFR Part 11: FDA requires audit trails and access controls that most cloud deployments can’t prove
HIPAA Security Rule: Breach penalties up to $1.5 million per violation category
ICH E6 (GCP): Clinical trial data must be attributable, legible, contemporaneous, original, and accurate, with documented evidence
EU Clinical Trials Regulation: Requires specific technical measures for data protection

The Regulatory Landscape Is Getting More Complex

The EU AI Act includes provisions for high-risk AI systems in healthcare
China’s biosecurity laws restrict the export of human genetic data
State-level genetic privacy laws are multiplying faster than compliance teams can track
GDPR Article 9 treats genetic data as a “special category” requiring explicit consent and additional safeguards
Brazil’s LGPD and other emerging privacy regimes add more jurisdictional complexity

The Bottom Line: Every Domain Has Exposure

Domain	Data Type	Primary Risk	Regulatory Pressure
Genomics	WGS, WES, RNA-seq	Patient re-identification, permanent PII	HIPAA, GDPR, state genetic laws
Proteomics	Mass spec raw files, peptide IDs	IP theft, biomarker leakage	Trade secrets, patent risk
Metabolomics	LC-MS/MS profiles	Diagnostic IP, patient health status	HIPAA, IVD regulations
Imaging	Pathology, radiology, microscopy	Patient ID from images, diagnostic liability	HIPAA, GxP for submissions
Cell Therapy	Single-cell seq, batch records	Manufacturing IP, patient-derived data	21 CFR 1271, HIPAA
Clinical Trials	EDC data, adverse events	Trial integrity, patient safety	21 CFR 11, ICH E6, GDPR
Drug Discovery	Compound libraries, screening data	Competitive intelligence loss	Trade secrets
Biologics	Sequence data, expression systems	Manufacturing process IP	21 CFR 600s, biosimilar competition

Current Cloud Security Solutions for Biotech (And Why They Fail)

Organizations facing these challenges typically choose from a menu of imperfect options:

Option 1: On-Premises Everything

The traditional approach: build your own data center, hire your own system administrators, maintain your own hardware refresh cycles.

This can work, but the economics are brutal:

A single high-memory node for genome assembly costs $50,000+ and sits idle most of the time between analysis runs
Recruiting bioinformatics talent is hard; finding people who can also manage HPC infrastructure is harder
Hardware procurement cycles mean you’re perpetually behind on the latest capabilities
Disaster recovery requires duplicating your entire infrastructure

For well-funded pharmaceutical companies, this works. For academic labs, startups, and smaller biotechs, the capital requirements are prohibitive.

Option 2: VPNs and Private Cloud

Connect your on-premises network to a cloud VPC via VPN. Run your workloads on dedicated instances. Treat the cloud as an extension of your data center.

This doesn’t solve the trust problem. You’re still trusting the cloud provider completely:

A compromised hypervisor or malicious insider can access memory on your “private” instances
Your encryption keys are managed by the same provider who manages your compute. They can access both
“Private” subnets still run on shared physical infrastructure
You have no visibility into who actually accessed your data

The Business Associate Agreements (BAAs) that cloud providers offer for HIPAA workloads are legal protection, not technical protection. They define who pays the fine when something goes wrong. They don’t prevent things from going wrong.

Option 3: Encrypt Everything

The most common approach: encrypt data at rest in S3, encrypt data in transit with TLS, and call it a day.

This protects against a narrow set of threats: someone stealing hard drives from the data center, or intercepting network traffic. It does nothing for the actual processing phase.

When your genomics pipeline runs, the data must be decrypted. It exists in plaintext in memory on a virtual machine running on shared infrastructure. Any of the following can access it:

Cloud provider employees with privileged access
Attackers who compromise the hypervisor
Nation-state actors with lawful intercept capabilities
Anyone who can execute a cold boot attack or memory dump

Encryption at rest and in transit protects data except when you’re actually using it. That exception is where all the interesting attacks happen.

Encryption Comparison: Where Your Data Is Vulnerable

Security Approach	Data at Rest	Data in Transit	Data in Use (Processing)	Protection Level
Standard Cloud	✅ Encrypted (S3/EBS)	✅ Encrypted (TLS)	❌ Plaintext in memory	LOW (Vulnerable during processing)
Private VPC	✅ Encrypted	✅ Encrypted	❌ Plaintext in memory	LOW (Provider still has access)
Customer-Managed Keys	✅ Encrypted	✅ Encrypted	❌ Plaintext in memory	MEDIUM (Key and compute separated, but still vulnerable)
Confidential Computing (TEE)	✅ Encrypted	✅ Encrypted	✅ Encrypted in enclave	HIGH (Protected at all stages)

Option 4: Compliance Certifications

“But AWS is SOC 2 Type II certified! They’re HIPAA eligible! They have FedRAMP authorization!”

True. These certifications matter. They demonstrate that the cloud provider has security controls, follows processes, and submits to audits.

What they can’t do:

Prevent a malicious insider from accessing customer data
Stop a sophisticated attacker who compromises privileged credentials
Give you visibility into who actually accessed your data

Certifications are table stakes for running regulated workloads. They’re not a guarantee of security.

Why Cloud Security Trust Models Don’t Work for Biotech

Cloud marketing materials talk about security features. They don’t mention the trust model underneath.

When you use cloud computing, you’re trusting the provider’s security team configured things correctly. You’re trusting their employees won’t abuse privileged access. You’re trusting contractors were vetted appropriately. You’re trusting incident response will detect and disclose breaches. You’re trusting key management infrastructure won’t be compromised. You’re trusting legal teams won’t comply with overbroad government requests. You’re trusting executives won’t change policies later.

That’s enormous trust to place in an organization you’ve never met, whose financial incentives don’t always align with your security requirements.

Who can theoretically access your “encrypted” data on a major cloud provider? Cloud operations engineers with hypervisor access. Security team members investigating incidents on shared infrastructure. On-call personnel responding to alerts. Key management administrators who operate the HSMs backing your “customer-managed” keys. Government agencies with legal process (valid or otherwise). Auditors reviewing security controls. Third-party contractors on specialized projects. Anyone who compromises credentials for any of the above.

This isn’t about cloud providers being bad actors. They employ talented security professionals and invest billions in infrastructure protection. The problem is architectural: centralized computing requires centralized trust, and that centralized trust becomes a single point of failure. No amount of perimeter security can fully address it.

The Trust Hierarchy in Traditional Cloud Computing

┌─────────────────────────────────────────────────────────┐
│                  YOUR SENSITIVE DATA                    │
│         (genomics, proteomics, clinical trials)         │
└────────────────────┬────────────────────────────────────┘
                     │
                     ▼
         ┌───────────────────────┐
         │   Hypervisor Layer    │ ◄─── Cloud ops engineers
         │  (Full memory access) │ ◄─── Security team
         └───────────┬───────────┘ ◄─── On-call staff
                     │             ◄─── Contractors
                     ▼             ◄─── Government orders
         ┌───────────────────────┐
         │  Key Management (HSM) │ ◄─── Key admins
         │ (Decryption possible) │ ◄─── Auditors
         └───────────────────────┘

              SINGLE POINT OF FAILURE
         Every layer can access your data

Traditional cloud architecture creates a trust hierarchy where your sensitive biotech data sits at the top, but every layer below can access it. The hypervisor layer has full memory access during processing. Cloud operations engineers, security teams, on-call staff, and contractors all have potential access paths. Key management systems that handle your encryption keys are operated by the same administrators. Government orders can compel access at any layer.

This creates a single point of failure: the cloud provider’s access controls and employee vetting. Your data security depends entirely on trusting their processes.

With Confidential Computing (TEE):

┌─────────────────────────────────────────────────────────┐
│         ENCRYPTED ENCLAVE (Hardware-Isolated)           │
│         Your data + code in protected memory            │
│    ✓ Hypervisor CANNOT access                           │
│    ✓ Cloud provider CANNOT access                       │
│    ✓ Cryptographic attestation proves integrity         │
└─────────────────────────────────────────────────────────┘

              ZERO-TRUST ARCHITECTURE
         Hardware enforces isolation, not policies

Confidential computing eliminates the trust hierarchy. Your data and processing code run inside a hardware-isolated encrypted enclave. The hypervisor cannot access it. The cloud provider cannot access it. Cryptographic attestation proves the integrity of the execution environment. The hardware enforces isolation, not policies or promises.

The Cloud Provider Insider Threat in Biotech

In enterprise security discussions, “insider threat” usually means employees at your organization going rogue. But for cloud computing, the more relevant insider threat is employees at the cloud provider.

According to the 2024 Verizon Data Breach Investigations Report, insider threats and privilege misuse account for a significant portion of breaches. These aren’t typically malicious. They’re usually mistakes, like misconfigured permissions or misdirected emails. But they happen.

Now multiply that by the number of people with privileged access to cloud infrastructure. Major cloud providers employ thousands of operations staff with varying levels of access. Each one represents potential risk.

The Edward Snowden disclosures revealed that intelligence agencies specifically targeted cloud provider infrastructure for mass surveillance. The PRISM program collected data directly from the servers of major tech companies. Whether through legal compulsion or technical compromise, government access to cloud data is a documented reality.

For biotech specifically, this matters because:

Clinical trial data could be valuable for investment decisions or competitor intelligence
Drug discovery research (including compound libraries, screening data, and hit-to-lead optimization) represents huge R&D investments worth stealing
Biomarker signatures from proteomics or metabolomics studies could invalidate years of discovery work
Manufacturing processes for biologics and cell therapies are core IP that competitors would pay millions to access
Patient data can be used for blackmail, identity theft, or discrimination
Genetic data has implications for individuals and their relatives for generations
Proprietary algorithms for image analysis or variant interpretation represent significant competitive advantages

The “we’re not important enough to target” argument misses the point. Breaches don’t discriminate. You don’t need to be the intended target to have your data exposed when infrastructure is compromised.

What Secure Cloud Computing for Biotech Actually Requires

Given these challenges, what would a truly secure cloud model look like for biotech?

First: process data without anyone seeing it. Not “encrypted at rest and in transit” but encrypted during processing too. The compute infrastructure should be mathematically incapable of accessing plaintext data, even if every employee at the provider wanted to. This applies whether you’re running variant calling on whole-genome sequencing, peptide identification on mass spec raw files, image segmentation on pathology slides, or statistical analysis on clinical trial endpoints.

Second: prove mathematically that no one saw it. Trust but verify isn’t enough. You need cryptographic proof that the code processing your data is exactly the code you approved, running in an environment that prevents data exfiltration. Not a compliance certification. Not a contractual promise. Mathematical certainty. For regulated environments, this proof becomes audit evidence. For IP-sensitive work, it becomes competitive protection.

Third: scale like cloud, secure like on-prem.

The point of cloud computing is elasticity: spin up 1,000 cores for a day, pay for what you use, then spin them down. Security measures shouldn’t require giving that up. If your “secure” solution requires dedicated hardware with long procurement cycles, you’ve just reinvented on-premises computing with extra steps and higher costs. Proteomics core facilities need burst capacity for peak instrument output. Cell therapy manufacturers need to scale with production batches. Clinical trials need to process data from multiple sites simultaneously. Modern DevOps practices enable this elasticity, but only if the security model supports it.

Fourth: maintain regulatory compliance. Whatever solution exists must satisfy HIPAA, GDPR, 21 CFR Part 11, SOC 2, GxP, ICH E6. Not by adding more legal agreements, but by providing technical controls that exceed regulatory requirements. The goal isn’t just passing an audit. It’s having technical measures so strong that the audit becomes a formality.

Fifth: work with existing tools.

Scientists have workflows they know and trust:

Genomics: Nextflow/Snakemake pipelines, BWA, GATK, STAR
Proteomics: MaxQuant, Proteome Discoverer, custom R/Python scripts
Imaging: CellProfiler, QuPath, deep learning models
Statistics: R, SAS, specialized clinical trial analysis packages

A secure platform that requires rewriting everything in a custom framework is dead on arrival. The tools must come to the data, not the other way around.

Confidential Computing: The Technology That Changes Biotech Cloud Security

These requirements sound impossible. For most of computing history, they were. But a category of technology called confidential computing has emerged in the past few years that makes them achievable.

The basic idea: what if the hardware itself enforced security boundaries? What if there were a region of memory that even the operating system (even the hypervisor) couldn’t access? What if code running in that region could prove cryptographically that it hadn’t been tampered with?

Technologies like Intel SGX, AMD SEV, and AWS Nitro Enclaves provide exactly this. They’re called Trusted Execution Environments (TEEs), and they fundamentally change what’s possible for secure cloud computing.

In our next post, we’ll dive deep into how AWS Nitro Enclaves work, why we chose them over alternatives, and how we built a biotech data processing platform that uses cryptographic attestation to prove your data stays private (even from us).

Setting up NF Core with MicroK8s

2025-02-01T00:00:00+00:00

Introduction

This tutorial introduces a workflow system designed for scientific data processing. It showcases a leading solution that helps scientists streamline their computational needs through code-defined pipelines for data extraction and visualization. Following our previous blog posts, we’ll set up everything using a MicroK8s Kubernetes cluster to ensure workload scalability.

We recommend reading our previous blog posts to familiarize yourself with the terminology and techniques used for provisioning and architecting the system. These posts also cover setting up a Docker-based DevOps environment to keep your local system clean.

Nextflow Basics

Nextflow is a domain-specific language (DSL) and workflow manager designed to streamline data analysis pipelines in high-performance computing (HPC), cloud, and containerized environments. Built on Scala and Groovy, it enables scalable, reproducible, and portable execution of computational workflows.

Key Features of Nextflow:

Pipeline Modularity – Workflows are defined using a simple script format (.nf), making them easy to develop and extend.
Parallel Execution – Nextflow automatically schedules and parallelizes tasks, efficiently utilizing available compute resources.
Container Support – Native support for Docker, Singularity, and Conda, ensuring environment consistency.
Resumability & Fault Tolerance – If a pipeline fails, Nextflow can resume from the last completed step, reducing re-computation.
Cloud & HPC Integration – Seamlessly runs on local machines, SLURM, SGE, Kubernetes, AWS Batch, and Google Cloud Life Sciences.

Basic Nextflow Syntax:

A simple Nextflow script (example.nf) looks like this:

process HELLO {
    input:
    val name

    output:
    stdout

    script:
    """
    echo "Hello, $name!"
    """
}

workflow {
    names = ['Alice', 'Bob', 'Charlie']
    HELLO(names)
}

You can run the script by running

nextflow run example.nf

One key thing to note is that all processes in nextflow are parralizable by default, so it’s important to understand what’s happening under the hood before implementing production grade workflows
More information here https://www.nextflow.io/docs/latest/your-first-script.html

Why Nextflow for Scientific Workflows?

Nextflow is widely used in bioinformatics, genomics, and large-scale data analysis because it allows researchers to define workflows in code, scale workloads efficiently, and maintain reproducibility. Combined with NF Core, Nextflow ensures robust, versioned, and best-practice pipelines, making it an excellent choice for scientific computing on Kubernetes.

What is NF Core and Why Do We Care?

NF Core is a community-driven framework for building and sharing best-practice computational pipelines using Nextflow, a workflow manager optimized for scalable and reproducible data analysis. NF Core provides a standardized set of pipelines that are rigorously tested and maintained, ensuring robust execution across different computational environments, including local workstations, high-performance computing (HPC) clusters, and cloud-based infrastructure.

Why Use NF Core?

Reproducibility – NF Core pipelines enforce strict version control, ensuring results remain consistent across different computational environments.
Scalability – Built with Nextflow, NF Core pipelines support seamless execution across multi-threaded, cluster-based, and cloud-based environments.
Portability – The pipelines are containerized using Docker, Singularity, or Conda, reducing dependency conflicts.
Community-Driven Best Practices – Each pipeline adheres to strict guidelines, is thoroughly tested, and receives updates from the global research community.

NF Core vs. Galaxy with CVMFS Drivers

Both NF Core and Galaxy are designed to facilitate scientific workflows, but they serve different use cases:

Feature	NF Core (Nextflow)	Galaxy (CVMFS)
User Experience	Command-line, highly scriptable	Web-based GUI for ease of access
Pipeline Development	Requires coding (Nextflow DSL)	Prebuilt tools with workflow assembly
Scalability	Highly scalable for cloud and HPC	Scales well but requires CVMFS for distributed workflows
Portability	Works with containers (Docker/Singularity)	CVMFS provides shared software repositories
Community Support	Strong, pipeline-focused community	Broad bioinformatics user base

DevOps Environment

As in previous tutorials, we’ll provision a DevOps Docker image that includes Terraform and AWS CLI for EC2 provisioning. These tools will suffice for this tutorial. You can follow our previous blog post’s guide on setting up the environment. Here’s what the provisioning script should look like.

#!/bin/bash
# provision.sh

AWSCLI_VERSION=2.13.25
TERRAFORM_VERSION=1.6.1

function install_base_packages(){
    sudo apt-get update
    sudo apt-get install -y unzip wget curl openssh-client rsync
}

function install_terraform() {
    cd /tmp
    wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
    unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip
    sudo mv terraform /usr/local/bin/
    rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip
}

function install_aws_cli() {
    cd /tmp
    curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-$AWSCLI_VERSION.zip" -o "awscli.zip"
    unzip awscli.zip > /dev/null
    sudo ./aws/install
    cd /tmp
    rm -rf aws
}

function main()
{
    install_base_packages
        install_terraform
        install_aws_cli
}

main

Nextflow Configuration

We need to create a Persistent Volume (PV) and a Persistent Volume Claim for the Nextflow images to use. You may need to adjust the storage request size depending on your image’s claim size. Here’s what we’ll use in this tutorial:

Persistent Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nextflow-pv
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nextflow-storage
  hostPath:
    path: "/home/ubuntu/work"

Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nextflow-pvc
  namespace: nextflow
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: nextflow-storage

You will need to define a nextflow.config file that will configure how Nextflow runs pipelines in your instance. Here is the configuration we’ll use:

process {
    executor = 'k8s'
}

k8s {
    storageClaimName = 'nextflow-pvc'
    storageMountPath = '/home/ubuntu/work'
    namespace = 'nextflow'
}

The configuration specifies Kubernetes as the executor rather than the default local executor. It also defines the PVC to use, container mount paths, and target namespace.

Inside your setup.sh script for EC2 provisioning, you’ll find the following method to set up NextFlow

function install_nextflow(){
    sudo DEBIAN_FRONTEND=noninteractive apt install openjdk-11-jdk -y
    curl -s https://get.nextflow.io | bash
    chmod +x nextflow
    sudo mv nextflow /usr/local/bin/

    # K8s Config for root
    sudo mkdir -p /root/.kube
    sudo ln -s /var/snap/microk8s/current/credentials/client.config /root/.kube/config

    # Setup up PV and PVCs
    microk8s kubectl create namespace nextflow
    microk8s kubectl apply -f nextflow-pv.yaml
    microk8s kubectl apply -f nextflow-pvc.yaml -n nextflow

    # Install FUSE plugin - Not used right now - better practice?
    # kubectl create -f https://github.com/nextflow-io/k8s-fuse-plugin/raw/master/manifests/k8s-fuse-plugin.yml
}

Following NextFlow’s installation instructions, we set up the required namespaces, Persistent Volume, and Persistent Volume Claim for pipeline execution. While NextFlow recommends using FUSE to deploy and run workflows, we’ve included but commented out the FUSE plugin installation for MicroK8s, as we’re not currently using it.

We are using a very similar setup.sh script as before, the main method does the following

function main() {
    install_microk8s
    install_nfs
    setup_microK8s
    update_ip_in_microk8s_config
    update_ip_in_kubeconfig
    install_nextflow
    install_docker
}

All of these methods can be found in the previous blog posts.

Deploying EC2 Instance

We’ll again use Terraform to deploy our EC2 instance, with additional files being transferred to the instance after provisioning. The security rules remain similar to our previous blog post. Below is the complete Terraform script. We’ve selected an m5.4xlarge instance type for this tutorial since many nf-core workflows require substantial CPU resources.

provider "aws" {
  region = "us-east-1"
}

data "http" "myip" {
  url = "https://ipv4.icanhazip.com"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }
  owners = ["099720109477"]
}

resource "aws_security_group" "microk8s_sg" {
  name        = "microk8s-sg"
  description = "Security group for MicroK8s EC2 instance"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow SSH access from your public IP"
  }

  ingress {
    from_port   = 16443
    to_port     = 16443
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow K8s access from your public IP"
  }
}

resource "aws_instance" "microk8s_instance" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = var.instance_type
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [aws_security_group.microk8s_sg.id]
  key_name               = var.key_name

  root_block_device {
    volume_size = 500
    volume_type = "gp2"
  }

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file(var.private_key_path)
    host        = self.public_ip
  }

  provisioner "file" {
    source      = "${path.module}/setup.sh"
    destination = "/home/ubuntu/setup.sh"
  }

  provisioner "file" {
    source      = "${path.module}/nextflow-pv.yaml"
    destination = "/home/ubuntu/nextflow-pv.yaml"
  }

  provisioner "file" {
    source      = "${path.module}/nextflow-pvc.yaml"
    destination = "/home/ubuntu/nextflow-pvc.yaml"
  }

  provisioner "file" {
    source      = "${path.module}/nextflow.config"
    destination = "/home/ubuntu/nextflow.config"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo bash /home/ubuntu/setup.sh"
    ]
  }

  tags = {
    Name = "microk8s-dev-instance"
  }
}

We can now deploy the instance by running the following

#!/bin/bash

terraform init
terraform validate
terraform plan -out=qc-microk8s-dev-plan
terraform apply qc-microk8s-dev-plan

Running our First NextFlow Workflow

Once everything has been deployed, we can connect to our EC2 instance using our ./connect.sh script. Once connected, we test that everything worked correctly by running the following commands

NextFlow has installed correctly

sudo nextflow info

########################## OUTPUT ##########################
  Version: 24.10.3 build 5933
  Created: 16-12-2024 15:34 UTC 
  System: Linux 5.15.0-1072-aws
  Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 11.0.25+9-post-Ubuntu-1ubuntu120.04
  Encoding: UTF-8 (UTF-8)

MicroK8s is running

microk8s status

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none

If the above two are good, now we can attempt to run our first workflow. If you run the command in the directory of the nextflow.config file, you should see some fun stuff happening!

sudo nextflow run hello

########################## OUTPUT ##########################
 N E X T F L O W   ~  version 24.10.3

Launching `https://github.com/nextflow-io/hello` [astonishing_caravaggio] DSL2 - revision: afff16a9b4 [master]

executor >  k8s (4) # Important that this does NOT say `local`
[fd/f7ba24] sayHello (4) [100%] 4 of 4 ✔
Bonjour world!

Ciao world!

Hello world!

Hola world!

If you get the above output running the hello image AND you have the k8s executer running, then you are in great shape. If this is not working, ensure your config matches up with what is here and that you are running it in the same directory at the config file.

Running our First NF Core Workflow

As mentioned before, NF Core is community driven by scientists around the world. As a result, there are hundreds are pre-built pipelines that can be used. All of the pipelines can be found here

https://nf-co.re/pipelines/

Now we are not subject matter experts in the field, but we do know how to set up configurations, so just going over the basic pipelines, there is a demo pipeline we can attempt to use in order to ensure that our K8s setup is correct.

demo: Introduction

Once we have the samplesheet.csv file as our input to the pipeline. We need to update the nextflow.config file slightly

process {
    executor = 'k8s'
}

k8s {
    storageClaimName = 'nextflow-pvc'
    storageMountPath = '/home/ubuntu/work'
    namespace = 'nextflow'
    pod = [ // required as the pipeline performs a sym-link
        [volumeClaim: "nextflow-pvc", mountPath: "/home/ubuntu/work"],
        [hostPath: "/root/.nextflow/assets", mountPath: "/root/.nextflow/assets"]
    ]
}

You should also create an output directory

Now we can run the following

sudo nextflow run nf-core/demo --input samplesheet.csv --outdir output

########################## OUTPUT ##########################
 N E X T F L O W   ~  version 24.10.3

Launching `https://github.com/nf-core/demo` [nice_fourier] DSL2 - revision: 04060b4644 [master]

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/demo 1.0.1
------------------------------------------------------
Input/output options
  input      : samplesheet.csv
  outdir     : output

Core Nextflow options
  revision   : master
  runName    : nice_fourier
  launchDir  : /home/ubuntu
  workDir    : /home/ubuntu/work
  projectDir : /root/.nextflow/assets/nf-core/demo
  userName   : root
  profile    : standard
  configFiles: 

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------* The pipeline
  https://doi.org/10.5281/zenodo.12192442

* The nf-core framework
    https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
    https://github.com/nf-core/demo/blob/master/CITATIONS.md

executor >  k8s (6)
executor >  k8s (6)
executor >  k8s (6)
executor >  k8s (6)
executor >  k8s (6)
executor >  k8s (7)
executor >  k8s (7)
executor >  k8s (7)
[d6/384cf0] NFCORE_DEMO:DEMO:FASTQC (SAMPLE1_PE)     [100%] 3 of 3 ✔
[db/180458] NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE1_PE) [100%] 3 of 3 ✔
[1a/799111] NFCORE_DEMO:DEMO:MULTIQC                 [100%] 1 of 1 ✔
-[nf-core/demo] Pipeline completed successfully-

If you monitor your cluster, you should see that pods start spinning up and completed their jobs. You can then go to the output directory and verify that data and result html files are there.

Conclusion

In this tutorial, we demonstrated how to set up and execute Nextflow workflows on a MicroK8s Kubernetes cluster, leveraging persistent storage and containerized execution to ensure scalability and reproducibility. We covered the basics of Nextflow, including its advantages for scientific workflows, and explored NF Core, a community-driven repository of curated pipelines.

By provisioning our infrastructure with Terraform, configuring Nextflow on Kubernetes, and successfully running both Nextflow’s Hello World workflow and an NF Core demo pipeline, we validated our deployment. These workflows illustrate how computational pipelines can be streamlined, modularized, and efficiently managed in a cloud-native environment.

This setup provides a solid foundation for running complex data analysis workflows in genomics, bioinformatics, and other scientific domains. Whether you’re processing large datasets, integrating various bioinformatics tools, or scaling workloads across cloud and on-premise clusters, Nextflow and Kubernetes provide a powerful combination for computational reproducibility.

As a next step, you can explore customizing Nextflow workflows, integrating cloud storage (e.g., AWS S3, Google Cloud Storage), or optimizing resource allocation for large-scale datasets. If you encounter any challenges, feel free to reach out.

Use Case: The Galaxy Project

2025-01-15T00:00:00+00:00

Previous Section: Putting it All Together: CVMFS In Action

Galaxy Project

In the era of big data, researchers across scientific disciplines require robust, scalable, and user-friendly platforms to analyze complex datasets. The Galaxy Project is an open-source, web-based computational workbench designed to empower researchers by providing accessible, reproducible, and transparent data analysis. Originally developed for bioinformatics and genomics, Galaxy has since evolved into a general-purpose platform for a wide range of scientific workflows, spanning domains such as climate science, cheminformatics, and machine learning.

At its core, Galaxy simplifies complex computational tasks by offering an interactive graphical user interface (GUI), eliminating the need for researchers to have deep programming knowledge. It also supports workflow automation, allowing users to build and share reproducible pipelines effortlessly. Galaxy’s extensibility is powered by a rich ecosystem of tools, a thriving community, and integrations with cloud and high-performance computing (HPC) environments.

With deployments ranging from local installations to large-scale Kubernetes-based infrastructures, Galaxy ensures that researchers can harness computational power efficiently, regardless of their expertise level. Whether running on personal servers, institutional clusters, or public clouds, Galaxy facilitates seamless collaboration, data sharing, and robust computational analysis.

In this post, we will use the skills we’ve gained to set up CVMFS drivers and allow us to set up a cluster that runs the Galaxy Project and process some demo data.

Chart Resources

If you have not read the previous blog posts, it is strongly advised that you do in order to gain a strong understanding of the DevOps technologies we are using to deploy this project.

Terraform Files

We will continue to use our main.tf as the backbone to the operations of deploying an EC2 Instance and Provisioning it using Terraform. In this file, we will also add an elastic IP that will remain static so that if we need to take down the instance and bring it back up, the IP will still remain the same to point to the Galaxy project webpage. Another major difference is that we will be running our provisioning scripts after the EIP is assigned, this is due to the fact that we require the IP address in order to connect to the cluster locally via Lens.

provider "aws" {
  region = "us-east-1"
}

data "http" "myip" {
  url = "https://ipv4.icanhazip.com"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }
  owners = ["099720109477"]
}

resource "aws_security_group" "microk8s_sg" {
  name        = "microk8s-sg"
  description = "Security group for MicroK8s EC2 instance"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow SSH access from your public IP"
  }

  ingress {
    from_port   = 16443
    to_port     = 16443
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow K8s access from your public IP"
  }

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow port-forward access to Galaxy service"
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow HTTP access on port 80"
  }

   # Allow access to the NodePort (e.g., 32000)
  ingress {
    from_port   = var.web_node_port                         
    to_port     = var.web_node_port                         
    protocol    = "tcp"
    cidr_blocks = [var.your_ip, "${chomp(data.http.myip.response_body)}/32"]
    description = "Allow public access to NodePort service"
  }
}

# Step 1: Allocate the Elastic IP
resource "aws_eip" "microk8s_eip" {
  domain = "vpc"
}

resource "aws_instance" "microk8s_instance" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = var.instance_type
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [aws_security_group.microk8s_sg.id]
  key_name               = var.key_name

  root_block_device {
    volume_size = 500
    volume_type = "gp2"
  }

  tags = {
    Name = "microk8s-dev-instance"
  }
}

resource "aws_eip_association" "microk8s_eip_assoc" {
  instance_id   = aws_instance.microk8s_instance.id
  allocation_id = aws_eip.microk8s_eip.id
}

resource "null_resource" "post_eip_provisioning" {
  depends_on = [aws_eip_association.microk8s_eip_assoc] 

  provisioner "file" {
    source      = "${path.module}/setup.sh"
    destination = "/home/ubuntu/setup.sh"
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }

  provisioner "file" {
    source      = "${path.module}/cvmfs-values.yaml"
    destination = "/home/ubuntu/cvmfs-values.yaml"
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }

  provisioner "file" {
    source      = "${path.module}/galaxy-pvc.yaml"
    destination = "/home/ubuntu/galaxy-pvc.yaml"
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }

  provisioner "file" {
    source      = "${path.module}/galaxy-values.yaml"
    destination = "/home/ubuntu/galaxy-values.yaml"
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }

  provisioner "file" {
    source      = "${path.module}/galaxy-services.sh"
    destination = "/home/ubuntu/galaxy-services.sh"
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }

  provisioner "remote-exec" {
    inline = [
      "sudo bash /home/ubuntu/setup.sh",
      "sudo chmod +x /home/ubuntu/galaxy-services.sh",
    ]
    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.private_key_path)
      host        = aws_eip.microk8s_eip.public_ip
    }
  }
}

Routing (Optional)

To simplify access to our Galaxy Project instance, we will configure a domain name, galaxy.lablytics.com, for easy navigation.

To achieve this, you will need to create a hosted zone within your DNS provider. Alternatively, you can define the hosted zone directly in your infrastructure code for a more automated setup.

routes.tf

#Route 53 A Record for NodePort service using the Elastic IP
data "aws_route53_zone" "zone" {
  name = "lablytics.com"
}

resource "aws_route53_record" "nodeport_service" {
  zone_id = data.aws_route53_zone.zone.zone_id
  name    = var.domain_name
  type    = "A"
  ttl     = 300
  records = [aws_eip.microk8s_eip.public_ip]
}

resource "aws_route53_record" "www_nodeport_service" {
  zone_id = data.aws_route53_zone.zone.zone_id
  name    = "www.${var.domain_name}"
  type    = "A"
  ttl     = 300
  records = [aws_eip.microk8s_eip.public_ip]
}

Setting Up the Provisioning Process

To provision and configure the CVMFS drivers, we will use the setup.sh script, which follows the standard approach outlined in the official CVMFS driver tutorials. This script ensures that the necessary drivers are installed and ready for use. You can find our version of the setup script in our public repository here: [TODO: Insert link].

Provisioning Galaxy Services

We will create a galaxy-services.sh script that will be executed directly within the EC2 instance. This approach is necessary because performing Helm updates over an SSH connection initiated by Terraform can sometimes result in connectivity issues.

The script itself is straightforward. It installs CVMFS and deploys the Galaxy Project using predefined configuration values. By running this script within the EC2 instance, we ensure a more reliable provisioning process without encountering SSH-related limitations.

function install_cvmfs(){
    echo "Installing CVMFS on K8s..."
    git clone https://github.com/lablytics/cvmfs-csi /home/ubuntu/cvmfs-csi
    microk8s helm3 install cvmfs /home/ubuntu/cvmfs-csi/deployments/helm/cvmfs-csi -n kube-system -f /home/ubuntu/cvmfs-values.yaml
    echo "CVMFS installed successfully"
    sleep 120
}

function install_galaxy(){
    echo "Installing Galaxy on K8s..."
    microk8s kubectl create namespace galaxy
    microk8s kubectl apply -f /home/ubuntu/galaxy-pvc.yaml 
    echo "Sleeping for 15 seconds..."
    sleep 15
    echo "Waking up..."

    # Install Galaxy
    git clone https://github.com/galaxyproject/galaxy-helm.git /home/ubuntu/galaxy-helm
    cd /home/ubuntu/galaxy-helm/galaxy
    git checkout 2557baac75ee56c8c9801946c9ee8d633db7d56e # Checkout to the version that works with the current Galaxy version
    sudo microk8s helm3 dependency update
    microk8s helm3 install -n galaxy galaxy-project /home/ubuntu/galaxy-helm/galaxy -f /home/ubuntu/galaxy-values.yaml --timeout 1m30s
    echo "Galaxy installed successfully"
}

function main(){
    install_cvmfs
    echo "Sleeping for 30 seconds..."
    sleep 30
    echo "Waking up..."
    install_galaxy
}

main

Value Files

We will need to update our cvmfs-values.yaml file as well as create a galaxy-values.yaml configuration file for the Galaxy deployment. As you’ll notice in the cvmfs-values.yaml , there are specific galaxy repositories here that we will require access to.

# cvmfs-values.yaml
extraConfigMaps:
  cvmfs-csi-default-local:
    default.local: |
      CVMFS_HTTP_PROXY="DIRECT"
      CVMFS_QUOTA_LIMIT="4000"
      CVMFS_USE_GEOAPI="yes"
      CVMFS_AUTOFS_TIMEOUT=3600
      CVMFS_DEBUGLOG=/tmp/cvmfs.log

      {{- if .Values.cache.alien.enabled }}
      CVMFS_ALIEN_CACHE={{ .Values.cache.alien.location }}
      # When alien cache is used, CVMFS does not control the size of the cache.
      CVMFS_QUOTA_LIMIT=-1
      # Whether repositories should share a cache directory or each have their own.
      CVMFS_SHARED_CACHE=no
      {{- end -}}
  cvmfs-csi-config-d:
    data.galaxyproject.org.conf: |
      CVMFS_SERVER_URL="http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-mel0.gvl.org.au/cvmfs/@fqrn@;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@"
      CVMFS_PUBLIC_KEY="/etc/cvmfs/config.d/data.galaxyproject.org.pub"
    data.galaxyproject.org.pub: |
      -----BEGIN PUBLIC KEY-----
      MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5LHQuKWzcX5iBbCGsXGt
      6CRi9+a9cKZG4UlX/lJukEJ+3dSxVDWJs88PSdLk+E25494oU56hB8YeVq+W8AQE
      3LWx2K2ruRjEAI2o8sRgs/IbafjZ7cBuERzqj3Tn5qUIBFoKUMWMSIiWTQe2Sfnj
      GzfDoswr5TTk7aH/FIXUjLnLGGCOzPtUC244IhHARzu86bWYxQJUw0/kZl5wVGcH
      maSgr39h1xPst0Vx1keJ95AH0wqxPbCcyBGtF1L6HQlLidmoIDqcCQpLsGJJEoOs
      NVNhhcb66OJHah5ppI1N3cZehdaKyr1XcF9eedwLFTvuiwTn6qMmttT/tHX7rcxT
      owIDAQAB
      -----END PUBLIC KEY-----

automountHostPath: /cvmfs
kubeletDirectory: /var/snap/microk8s/common/var/lib/kubelet

automountStorageClass:
  create: true
  name: cvmfs

# galaxy-values.yaml
cvmfs:
  deploy: false

persistence:
  existingClaim: "galaxy-pvc"

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - host: ~
      paths:
        - path: "/"
        - path: "/training-material"

Putting it all together

Now we can create the EC2 instance by running the .create.sh script that we previously wrote. This will tell Terraform to initialize, create a plan, validate it and finally create our AWS resources required to deploy the Galaxy Project.

#!/bin/bash

terraform init
terraform validate
terraform plan -out=qc-microk8s-dev-plan
terraform apply qc-microk8s-dev-plan

Once our instance has been successfully created and provisioned, we can establish an SSH connection using the connect.sh script.

In this iteration, we’ve made a small but impactful change from the previous tutorial: the Kubernetes configuration (kubeconfig) is now copied to a location that Lens can automatically sync with. This is achieved by setting up a mount from our local system, where Lens expects the configuration, to the Docker DevOps container.

As a result, when we transfer the kubeconfig from the EC2 instance to the Docker container using scp, the mount ensures that our local machine also has direct access to the file. This streamlines the workflow, eliminating the need for additional manual steps.

#!/bin/bash

# Set your AWS region
AWS_REGION="us-east-1"  # Change this to the region where your EC2 instance is running

# Set your EC2 key pair path
KEY_PAIR_PATH="../devbox-key-pair.pem"  # Change this to the correct path to your key pair

# Set your EC2 tag name to find the instance
INSTANCE_TAG_NAME="microk8s-dev-instance"

# Get the public IP of the instance with the tag name 'microk8s-dev-instance'
INSTANCE_PUBLIC_IP=$(aws ec2 describe-instances \
    --region $AWS_REGION \
    --filters "Name=tag:Name,Values=$INSTANCE_TAG_NAME" "Name=instance-state-name,Values=running" \
    --query "Reservations[*].Instances[*].PublicIpAddress" \
    --output text)

# Check if an IP address was found
if [ -z "$INSTANCE_PUBLIC_IP" ]; then
  echo "Instance with tag name '$INSTANCE_TAG_NAME' is not running or doesn't exist."
  exit 1
fi

# Connect to the instance via SSH
echo "Connecting to EC2 instance with IP: $INSTANCE_PUBLIC_IP"

scp -i "$KEY_PAIR_PATH" ubuntu@$INSTANCE_PUBLIC_IP:/home/ubuntu/.kube/config-public /configs/microk8s-config
ssh -i "$KEY_PAIR_PATH" ubuntu@$INSTANCE_PUBLIC_IP

Once you are connected to the instance, you should be able to run the ./galaxy-services.sh script which installs everything you need. This does take a few minutes (around 5 or so), but once everything is running, you should see something like this

Lens

CLI

ubuntu@ip-172-31-85-199:~/galaxy-helm/galaxy$ kubectl get pods -n galaxy
NAME                                                              READY   STATUS    RESTARTS   AGE
galaxy-galaxy-project-postgres-0                                  1/1     Running   0          41m
galaxy-project-celery-5f95bf89bd-txsqd                            1/1     Running   0          41m
galaxy-project-celery-beat-7cf5f8857c-87k2q                       1/1     Running   0          41m
galaxy-project-job-0-64964cf69c-rtkkv                             1/1     Running   0          41m
galaxy-project-nginx-76cf7cc5bf-9bkvs                             1/1     Running   0          41m
galaxy-project-post-install-cvmfs-fix-job-jst2j                   1/1     Running   0          41m
galaxy-project-postgres-6f8c745954-8fbnk                          1/1     Running   0          41m
galaxy-project-rabbitmq-7bfcd857c9-99tx8                          1/1     Running   0          41m
galaxy-project-rabbitmq-messaging-topology-operator-65f67ccr5n7   1/1     Running   0          41m
galaxy-project-rabbitmq-server-server-0                           1/1     Running   0          41m
galaxy-project-tusd-65b6b64df6-kq8v8                              1/1     Running   0          41m
galaxy-project-web-6bc44df94d-qvqv8                               1/1     Running   0          41m
galaxy-project-workflow-7cd9dbd78c-lkmhp                          1/1     Running   0          41m

Going through the demo

In order to ensure everything is working correct, follow the tutorial provided by Galaxy

https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-short/tutorial.html

First, in order to access the web UI, we need to do one of the following.

Forward SSH Tunnel to the WebUI Pod
Go to the IP address of the EC2 Instance
Go to the assigned domain name of the EC2 Instance

Forward SSH Tunnel

Lens

In Lens, go to the galaxy-project-web-... pod, and open it. If you scroll down to Containers, then Port, you should see that the container is running on port 8080. There is a blue Forward… button that allows you to port forward to the pod instance and access it. Once you set that up, you should be able to view http://localhost:[ASSIGNED_PORT]/ and see the Galaxy UI.

CLI

You can run the following command to port forward

kubectl port-forward pod/<POD_NAME> <LOCAL_PORT>:8080 -n galaxy

If you are able to reach the UI, you should see something like

If you are able to get through the entire demo, then you are done and have everything set up!

Conclusion

By following this guide, we have successfully deployed the Galaxy Project on an EC2 instance using Terraform, Kubernetes, Helm, and CVMFS. This setup provides a scalable and reproducible environment for researchers and data scientists to leverage Galaxy’s powerful workflow automation and computational capabilities.

We began by provisioning our infrastructure with Terraform, ensuring that our EC2 instance was correctly configured with an Elastic IP for persistent access. We then deployed CVMFS drivers, enabling efficient data access across distributed environments, and installed Galaxy within a Kubernetes cluster. Through the use of Helm charts and predefined configuration files, we automated the deployment process, ensuring a seamless setup.

To simplify access, we configured a hosted domain name and explored different methods to connect to the Galaxy Web UI, including port forwarding via SSH, direct access through the instance’s public IP, and routing through a custom domain. Once everything was set up, we validated our deployment by running a sample workflow from the Galaxy training materials, confirming that the system was operational.

With this foundation in place, you now have a fully functional Galaxy instance running in a cloud-based, containerized environment. This setup not only enables high-performance data analysis but also ensures scalability, reproducibility, and ease of collaboration. Whether you’re using Galaxy for bioinformatics, cheminformatics, climate science, or machine learning, this deployment provides the flexibility and power needed for modern scientific computing.

Now that your Galaxy instance is up and running, you can start exploring its vast ecosystem of tools and workflows. If you encounter any issues, feel free to reach out.

Code Reference

https://github.com/lablytics/cvmfs-csi/tree/master/devops/cvmfs-full-setup-basic-galaxy

Putting it All Together: CVMFS In Action

2025-01-14T00:00:00+00:00

Previous Section: Kubernetes and Helm: The Beauty of Orchestration

We are finally at the place where we start to see all of our hard work pay off! This is the point where we continue to automate the creation of our resources as well as connect to our cluster pods and see CVMFS in action. It is assumed that ALL of the previous sections have been read.

Why all the Automation?

Manual setup of cloud infrastructure can be both time-consuming and error-prone. Automating these tasks not only ensures consistency but also accelerates the deployment process, making your infrastructure scalable and easily reproducible. By using a script to automate the deployment of resources, you minimize the risks associated with human error and improve efficiency.

The automation provided here serves two purposes:

Consistency: Every time you run the script, it ensures that the environment is set up in exactly the same way. This eliminates discrepancies that can arise from manual setup, where missed steps or subtle differences can lead to unexpected behavior.
Efficiency: Manual deployment requires time and detailed knowledge of each step involved, whereas an automated script performs all actions in sequence, reducing the overall setup time. This is particularly useful when deploying multiple environments (e.g., for staging, testing, and production), or when you need to make infrastructure adjustments and roll out updates.

If you’re curious to dive deeper into the underlying processes, feel free to execute each step manually and explore the commands used in the script. Understanding what each command does will give you a solid foundation in how the cloud infrastructure is built and managed.

Deploy our EC2 Instance With Terraform

In this script, we’ll automate the setup and deployment of an EC2 instance using Terraform through a simple Bash script. This script automates key Terraform steps: initialization, validation, planning, and deployment, making the process efficient and consistent.

Prerequisites

Before running this script, ensure:

You have a local PEM key pair (replace YOUR_KEY_PAIR.pem with your actual key).
Terraform is installed on your machine.
AWS credentials are configured.

Script Overview: `create.sh`

The script performs the following steps:

Set Key Permissions: Secures the PEM key for SSH access.
Initialize Terraform: Prepares Terraform by downloading providers and setting up configuration.
Validate Configuration: Checks for syntax and structural errors.
Plan Deployment: Previews resources to be created or modified.
Apply the Plan: Deploys resources as per the saved plan.

Script Example

Save this script as create.sh in your Terraform project directory:

# create.sh

chmod 400 ../YOUR_KEY_PAIR.pem  # Secure the PEM key

terraform init                   # Initialize Terraform
terraform validate               # Validate configuration
terraform plan -out=qc-microk8s-dev-plan  # Save the plan
terraform apply qc-microk8s-dev-plan      # Execute deployment

This script provides a reliable, automated way to deploy infrastructure using Terraform.

We can run the script by

./create.sh

It will take a few minutes to provision depending on the instance size you are using, but when its all done, you should see the following few messages.

aws_instance.microk8s_instance (remote-exec): Resolving deltas: 100% (6318/6318)
aws_instance.microk8s_instance (remote-exec): Resolving deltas: 100% (6318/6318), done.
aws_instance.microk8s_instance (remote-exec): NAME: cvmfs
aws_instance.microk8s_instance (remote-exec): LAST DEPLOYED: Mon Nov 11 21:17:34 2024
aws_instance.microk8s_instance (remote-exec): NAMESPACE: kube-system
aws_instance.microk8s_instance (remote-exec): STATUS: deployed
aws_instance.microk8s_instance (remote-exec): REVISION: 1
aws_instance.microk8s_instance (remote-exec): TEST SUITE: None
aws_instance.microk8s_instance (remote-exec): CVMFS installed successfully
aws_instance.microk8s_instance (remote-exec): Deploying CVMFS PVC
aws_instance.microk8s_instance (remote-exec): persistentvolumeclaim/cvmfs created
aws_instance.microk8s_instance (remote-exec): CVMFS PVC deployed successfully
aws_instance.microk8s_instance (remote-exec): Deploying CVMFS test pod
aws_instance.microk8s_instance (remote-exec): pod/cvmfs-demo created
aws_instance.microk8s_instance (remote-exec): CVMFS test pod deployed successfully
aws_instance.microk8s_instance: Creation complete after 2m53s [id=i-0545cfb13070491e5]

You should start to see things happening if everything is working correctly. If errors occur, please ensure that Terraform has been installed correctly and your AWS credentials are setup. We will also walk through how to get around of the issues later in this post.

Connecting to Your MicroK8s EC2 Instance

After deploying an EC2 instance to host our MicroK8s cluster, the next step is setting up a seamless connection to manage it. This is a simple Bash script designed to connect to the EC2 instance via SSH and download the Kubernetes configuration file for local access. With this configuration file, we can use tools like Lens to manage and monitor the cluster in a user-friendly environment.

Script Overview: `connect.sh`

The connect.sh script automates two main tasks:

Retrieve the EC2 Instance’s Public IP: Using AWS CLI commands, the script identifies the public IP address of the EC2 instance by filtering based on a specified tag name (e.g., microk8s-dev-instance). This is particularly useful in environments with multiple instances, as it allows you to pinpoint the right one.
Download Kubernetes Config for Local Access: Once the IP address is identified, the script uses scp to download the config-public file from the MicroK8s instance to your local environment. This configuration allows tools like Lens to connect to the Kubernetes cluster and view all pods and resources through an intuitive interface.
Connect via SSH: Finally, the script initiates an SSH connection to the instance, providing access to the EC2 server and any further on-instance configurations.

Script Details

Let’s walk through the script. Save it as connect.sh and customize it with your own key pair path, AWS region, and instance tag name.

#!/bin/bash

# Set your AWS region
AWS_REGION="us-east-1"  # Change this to the region where your EC2 instance is running

# Set your EC2 key pair path
KEY_PAIR_PATH="../devbox-key-pair.pem"  # Change this to the correct path to your key pair

# Set your EC2 tag name to find the instance
INSTANCE_TAG_NAME="microk8s-dev-instance"

# Get the public IP of the instance with the tag name 'microk8s-dev-instance'
INSTANCE_PUBLIC_IP=$(aws ec2 describe-instances \
    --region $AWS_REGION \
    --filters "Name=tag:Name,Values=$INSTANCE_TAG_NAME" "Name=instance-state-name,Values=running" \
    --query "Reservations[*].Instances[*].PublicIpAddress" \
    --output text)

# Check if an IP address was found
if [ -z "$INSTANCE_PUBLIC_IP" ]; then
  echo "Instance with tag name '$INSTANCE_TAG_NAME' is not running or doesn't exist."
  exit 1
fi

# Connect to the instance via SSH
echo "Connecting to EC2 instance with IP: $INSTANCE_PUBLIC_IP"

scp -i "$KEY_PAIR_PATH" ubuntu@$INSTANCE_PUBLIC_IP:/home/ubuntu/.kube/config-public .  # Download kube config
ssh -i "$KEY_PAIR_PATH" ubuntu@$INSTANCE_PUBLIC_IP  # Connect to the instance

Ensuring Everything is Running

Once we create everything with create.sh and connect to the instance using connect.sh, we should be able to see the following

Check to see if all the required pods are running

kubectl get pods -n kube-system

You should see the following

and

kubectl get pods

By downloading the Kubernetes config-public file to your local machine, you can use Lens or similar Kubernetes management tools to gain a graphical view of your cluster. This simplifies monitoring and managing your pods, services, and resources, offering a clearer and more accessible experience than using command-line tools alone.

We can also see the config-public in our current working directory in the docker container, we can copy and paste the config into Lens to connect to it. We will then be able to see the NFS and CVMFS drivers up and running.

Issues

If you DO NOT see the CVMFS drivers OR the Test Pod running, then there may have been an issue with the remote provisioning to get all the drivers up and running. This is a common occurrence so you can also do the following

Check to see if the repo is in the current directory by doing ls

Enter the following command to manually install the CVMFS Drivers

microk8s helm3 install cvmfs /home/ubuntu/cvmfs-csi/deployments/helm/cvmfs-csi -n kube-system -f cvmfs-values.yaml

You should see the following

You can check to see if the pods are running by doing

kubectl get pods -n kube-system

And you should see the following

Now deploy the PVC and the Test Pod by doing the following in the command prompt

microk8s kubectl apply -f cvmfs-pvc.yaml --validate=false
microk8s kubectl apply -f cvmfs-demo-pod.yaml --validate=false

and you should now see once the container is created

The Main Event

This is the moment we have all been waiting for, all of the tireless config creations and changes and head banging, we can finally see if everything is configured properly. If you followed every step in the guide and used all of the scripts we have provided, there is a 99% chance that this just all works (We are very sorry for the 1%, please do reach out if you require help)

Let’s SSH into the demo pod by running

kubectl exec -it cvmfs-demo -- /bin/bash

Do an ls to make sure you have the /cvmfs folder in your root

cd into the /cvmfs folder
Do another ls to see if anything is there (There should not be)
BUT now we can do ls -l atlas.cern.ch
- What is atlas.cern.ch, you can read more about the Atlas project here https://atlas.cern/Discover/Detector
AND BOOM, we see something

We can now go into the folder and have access to all the resources found in this repository

root@cvmfs-demo:/cvmfs# ls
atlas.cern.ch  cvmfs-config.cern.ch
root@cvmfs-demo:/cvmfs# cd atlas.cern.ch/
root@cvmfs-demo:/cvmfs/atlas.cern.ch# ls
repo
root@cvmfs-demo:/cvmfs/atlas.cern.ch# cd repo/
root@cvmfs-demo:/cvmfs/atlas.cern.ch/repo# ls
ATLASLocalRootBase  benchmarks  conditions  containers  dev  sw  test  tools  tutorials
root@cvmfs-demo:/cvmfs/atlas.cern.ch/repo#

What Just Happened?

How did our pod go from empty to housing a repository full of tools for the ATLAS project? That’s the power of CVMFS at work. The CVMFS drivers we installed earlier are responsible for managing these dynamic volumes. When we access the /cvmfs mount, it signals the drivers to retrieve the requested resources. The drivers, with the help of FUSE (Filesystem in Userspace), seamlessly pull in the data needed for the repository, giving us instant access to specialized tools and data.

So What’s Next?

Now that we’ve accessed the ATLAS project’s repository through CVMFS, you might be wondering how this setup could benefit our own projects. Imagine having an instantly accessible, version-controlled library of tools, datasets, and configurations that’s always up-to-date and readily available to anyone who needs it. This is the power of CVMFS. It’s not only useful for accessing existing repositories like ATLAS but also for hosting and sharing your own resources.

How Can We Use This for Our Own Projects?

CVMFS allows us to create and manage custom repositories where we can package our own tools, scripts, datasets, and even entire software environments. By hosting these resources on a CVMFS repository, we can ensure consistency across development and production environments and make collaboration smoother for teams working with large or complex data.

In the next part of our series, we’ll explore a live use case: The Galaxy Project. This project leverages CVMFS to manage and distribute bioinformatics tools, datasets, and workflows. We’ll dive into how Galaxy uses CVMFS to ensure that researchers have access to the latest tools and resources, and how you can apply similar principles to streamline and scale your own projects. Stay tuned!

Optional: Teardown of Instance

Here is a small script to tear down your entire EC2 instance once you are done practicing. You can use this in any terraform setup

#!/bin/bash

echo "Destroying Terraform-managed resources..."
terraform init 
terraform destroy -auto-approve 

# Optional: Remove all terraform state files
rm -f terraform.tfstate
rm -f terraform.tfstate.backup
rm -f terraform.tfstate.lock.info
rm -f .terraform.lock.hcl
rm -rf .terraform
echo "Terraform resources destroyed."

Repo for FULL Setup https://github.com/lablytics/cvmfs-csi/tree/master/devops

Next Section: Use Case: The Galaxy Project