HL7 Transport: SFTP Batch Files

SFTP (SSH File Transfer Protocol) is an asynchronous transport for batches of HL7 v2 messages. Unlike MLLP (synchronous, connection-based), SFTP allows systems to upload or download HL7 message batches on a scheduled or periodic basis, using file-based polling. This is ideal for high-volume bulk transfers, end-of-day batches, or scenarios where sender and receiver cannot maintain persistent connections.

What is SFTP

SFTP is a protocol for transferring files over SSH (Secure Shell). It provides file operations (upload, download, list, delete, rename) with full encryption and authentication. For HL7 purposes, SFTP is used to:

Upload batches: A source system writes one or more HL7 messages to a batch file and uploads it to a remote SFTP server.
Download batches: A receiving system polls an SFTP server periodically, downloads batch files, parses them, and processes the messages.
Acknowledgment files: The receiver can upload an acknowledgment file back to the sender's SFTP directory.

Unlike MLLP (message-by-message), SFTP is file-oriented and inherently batch-based. Multiple HL7 messages are wrapped in a single batch file with header and trailer segments (FHS, BHS, BTS, FTS).

SFTP runs over SSH (TCP port 22 by default) and is defined in IETF RFC 4251 (SSH) and RFC 4253 (SSH Transport).

How it works

SFTP batch transport involves three main actors: a batch file, a sender (uploader), and a receiver (downloader/poller).

Batch File Structure

A single SFTP batch file contains multiple HL7 messages wrapped in batch-level header and trailer segments:

FHS|<file metadata>
  BHS|<batch metadata>
    MSH|...<Message 1>...
    PID|...
    ...
    MSH|...<Message 2>...
    OBR|...
    ...
  BTS|<message count>
  BHS|<another batch>
    MSH|...<Message 3>...
    ...
  BTS|<message count>
FTS|<batch count>|<message count>

Segment definitions:

FHS (File Header Segment): Marks the start of a batch file. Contains file-level metadata (sender, receiver, timestamp).
BHS (Batch Header Segment): Marks the start of a message batch within the file. May be repeated if the file contains multiple batches.
[HL7 Messages]: One or more complete HL7 messages (starting with MSH), each message complete with all its segments.
BTS (Batch Trailer Segment): Marks the end of a batch. Includes the message count for that batch.
FTS (File Trailer Segment): Marks the end of the file. Includes batch count and total message count.

Wire-level Sequence (File-based, Asynchronous)

Sender System                            SFTP Server                    Receiver System
      |                                       |                              |
      +-- SSH connect (port 22) ------------>|                              |
      |<-- SSH handshake -------------------+                              |
      |                                      |                              |
      +-- SFTP open (/)  ------------------>|                              |
      |<-- OK ----------|                    |                              |
      |                                      |                              |
      +-- Write batch to temp file -------->|                              |
      |<-- OK [wrote 1500 bytes]  --------+  |                              |
      |                                      |                              |
      +-- Rename temp -> live name ---------->|                              |
      |<-- OK [file ready] ----------------+  |                              |
      |                                      |                              |
      +-- Close & disconnect -------->|      |                              |
      |                                      |                              |
      |                              [File on disk: adt_20260701_001.hl7]   |
      |                                      |                              |
      |         [Receiver polls every 60 seconds] <------- Scheduled task ---|
      |                                      |                              |
      |                                      +-- SSH connect (port 22) ----->|
      |                                      |<-- SSH handshake -----------+|
      |                                      |                              |
      |                                      +-- SFTP list (inbox) ------>|
      |                                      |<-- [adt_20260701_001.hl7] -+|
      |                                      |                              |
      |                                      +-- Download file ------------>|
      |                                      |<-- File data -----[1500 bytes]|
      |                                      |                              |
      |                                      +-- Rename to .processing ----->|
      |                                      |<-- OK -----[atomic rename] -+|
      |                                      |                              |
      |                                      +-- Close & disconnect ------>|
      |                                      |                              |
      |                  [Receiver parses & processes messages]             |
      |                                      |                              |
      |                                      +-- SSH connect (port 22) ----->|
      |                                      |<-- SSH handshake -----------+|
      |                                      |                              |
      |                                      +-- Delete .processing file --->|
      |                                      |<-- OK -------[or rename .done]|
      |                                      |                              |
      |                                      +-- Upload ACK batch file ----->|
      |                                      |<-- OK [adt_20260701_001.ack]+|
      |                                      |                              |
      |                                      +-- Close & disconnect ------>|
      |                                      |                              |
      +-- Download ACK (optional) --------->|                              |
      |<-- adt_20260701_001.ack -----------+                               |

File Naming and Polling

Typical SFTP batch workflow:

Sender writes file: Creates file with a unique name, e.g., adt_20260701_120000_MSG001.hl7.
Atomic rename: Renames file from a temporary name (e.g., .tmp, .processing) to the final name to signal readiness.
Receiver polls: Scans the SFTP inbox directory at regular intervals (e.g., every 60 seconds).
Receiver downloads: Fetches the file via SFTP.
Mark as processing: Renames the file to .processing to prevent duplicate downloads.
Parse and process: Extracts messages from the batch file and inserts them into the receiver's system.
Mark as complete: Renames the file to .done or deletes it.
Send acknowledgment: Uploads an ACK batch file to a designated SFTP directory for the sender to download.

When to use it

SFTP is the transport of choice for batch, asynchronous, or high-volume message exchanges:

End-of-day batch reports: Labs, radiology, or pharmacy systems upload daily results in a single batch file.
High-volume data migration: Transferring large numbers of historical records (admission histories, patient demographics) in bulk.
Offline or intermittent connectivity: Sender and receiver do not need persistent connections; sender uploads at any time, receiver polls periodically.
Firewall-friendly: SFTP uses SSH (port 22), which is standard and less restrictive than opening random ports for MLLP listeners.
Audit and compliance: File-based transfers leave an audit trail; files can be encrypted and signed.
Non-real-time systems: When acknowledgments can arrive hours or days after transmission (unlike MLLP's immediate ACK).

Do NOT use SFTP if you need:

Real-time message delivery: SFTP polling introduces minutes to hours of latency.
Synchronous ACK: The sender must wait; SFTP is inherently asynchronous.
Point-to-point messaging: SFTP is file-oriented, not connection-oriented; use MLLP instead.
Small, frequent messages: The overhead of SFTP file operations outweighs the benefit for individual messages.

Configuration parameters

Port

SFTP runs over SSH, which defaults to TCP port 22. Some organizations use alternate ports (e.g., 2222) to avoid conflicts or comply with firewall policies. Always verify the SFTP server's configured port.

Authentication

SFTP requires SSH authentication. Choose one or both:

Username/password: Simple but less secure; the password is transmitted over the encrypted SSH channel.
Public key authentication (SSH keys): More secure; the server validates the client's private key. Recommended for production.

Store private keys securely (e.g., in a key management system or encrypted vault, never in code or configuration files).

Directories

Define separate SFTP directories for different purposes:

Inbox (download): Where the sender places files for the receiver to download. E.g., /sftp/incoming/adt/.
Outbox (upload): Where the receiver places acknowledgment or response files for the sender to download. E.g., /sftp/outgoing/adt/.
Archive: Optionally, a directory for processed files. E.g., /sftp/archive/adt/.

Polling Interval

The interval (in seconds) at which the receiver checks the SFTP inbox for new files. Common values:

High-frequency: 10–30 seconds (for near-real-time batches).
Normal: 60 seconds (for batches updated hourly or more).
Low-frequency: 300–3600 seconds (for end-of-day or daily batches).

Set based on the expected message arrival pattern and acceptable latency.

File Naming Convention

Establish a naming scheme to avoid collisions and enable filtering. Example:

{MessageType}_{Date}_{Time}_{Sequence}.hl7

adt_20260701_120000_001.hl7
orm_20260701_120015_002.hl7
oru_20260701_120030_001.hl7

MessageType: ADT, ORM, ORU, etc.
Date: YYYYMMDD
Time: HHMMSS
Sequence: 001, 002, etc. (or random UUID)
Extension: .hl7 or .txt

Atomic Write Pattern

To prevent the receiver from downloading a partially-written file:

Write the file to a temporary filename (e.g., .tmp or .processing).
Once complete, atomically rename the file to the final name in a single filesystem operation.
The receiver only processes files with the final name (not .tmp or .processing).

This ensures the file is complete before the receiver attempts to download it.

Character Encoding

Use consistent encoding (typically UTF-8 or ISO-8859-1) across sender and receiver. Specify the encoding in the MSH segment's character set field (MSH-18) or document it explicitly.

Security

SFTP is built on SSH, which provides strong encryption and authentication by default. Additional security measures:

SSH Encryption

SFTP traffic is encrypted using AES-128, AES-192, AES-256, or other algorithms negotiated during the SSH handshake. Always use SSH (port 22) instead of unencrypted FTP.

Key Management

Sender: Maintains a private key for authentication to the SFTP server.
Receiver: Maintains a private key for authentication to the SFTP server.
SFTP Server: Maintains the public keys of all authorized clients.

Rotate keys periodically (e.g., annually) and retire old keys.

File Permissions and Ownership

Configure SFTP directory permissions to restrict access:

Sender inbox: Read/write by sender only; read-only by receiver.
Receiver outbox: Write-only by receiver; read-only by sender.
Archive: Read-only by authorized administrators.

Use filesystem ACLs or SELinux policies to enforce these restrictions.

Message Encryption

Optionally, encrypt the batch file payload (not just the SSH transport) before uploading:

GPG encryption: Encrypt the .hl7 file with the receiver's public key before uploading. The receiver decrypts with their private key.
Digital signatures: Sign the batch file with the sender's private key; the receiver verifies the signature to confirm authenticity.

Audit Logging

Enable SFTP server logging to record all file operations:

[2026-07-01 12:00:15] sshd: User 'sender_app' (IP 192.168.1.100) authenticated with key
[2026-07-01 12:00:16] sftp: PUT /sftp/incoming/adt/adt_20260701_120000_001.hl7 (1500 bytes)
[2026-07-01 12:00:16] sftp: SETSTAT /sftp/incoming/adt/adt_20260701_120000_001.hl7 (rename from .tmp)
[2026-07-01 12:00:45] sshd: User 'receiver_app' (IP 192.168.1.101) authenticated with key
[2026-07-01 12:00:46] sftp: GET /sftp/incoming/adt/adt_20260701_120000_001.hl7 (1500 bytes)

Pros & cons

Advantages

Asynchronous: No need for persistent connections or real-time message delivery; sender and receiver operate on their own schedule.
High-volume efficient: Batches multiple messages in a single file transfer, reducing overhead per message.
Firewall-friendly: Uses standard SSH port (22), which is almost universally allowed.
Secure by design: SSH provides encryption and key-based authentication.
Audit trail: Files remain on disk; they can be archived, versioned, and logged for compliance.
Scalable: No connection-pool limits; thousands of files can be queued without resource exhaustion.
Flexibility: Batch size, content, and schedule are entirely under sender control.

Disadvantages

High latency: Polling introduces delays (minutes to hours between upload and processing).
No real-time ACK: The sender must poll a separate directory for acknowledgments; there is no synchronous response.
File-based complexity: Requires atomic write patterns, cleanup logic, and careful file naming to avoid collisions and duplicates.
Manual administration: Unlike MLLP (auto-connect), SFTP requires manual directory setup, permission management, and monitoring.
Polling overhead: Frequent polling of empty directories wastes bandwidth and CPU; infrequent polling increases latency.
Retry logic: If the receiver crashes mid-processing, recovery requires manual intervention or complex state-tracking (.processing files).
Not real-time capable: Large batches or slow networks can cause significant delays; unsuitable for time-critical workflows.
Batch semantics: All-or-nothing processing; if one message in a batch is invalid, the entire batch may be rejected.

Examples

Wire-level capture and sample SFTP handshake

SFTP file upload (SSH session, simplified):

SSH Stream (binary, shown conceptually):

1. Client initiates SSH connection:
   → SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5

2. Server responds:
   ← SSH-2.0-OpenSSH_7.4

3. Key exchange & authentication (encrypted)
   → [Key exchange algorithm negotiation]
   → [Diffie-Hellman key exchange]
   → [Client authentication: public key for user 'sender_app']
   ← [Authentication successful]

4. SFTP subsystem request:
   → SSH channel: Request SFTP subsystem

5. SFTP file write (binary):
   → Open: /sftp/incoming/adt/adt_20260701_120000_001.hl7.tmp
   → Write: [1500 bytes of HL7 batch data]
   → Close

6. Atomic rename (SFTP):
   → Rename: /sftp/incoming/adt/adt_20260701_120000_001.hl7.tmp
                    →
             /sftp/incoming/adt/adt_20260701_120000_001.hl7

7. Close SFTP & SSH:
   → [Session closed]

Annotated batch file frame

A complete HL7 batch file in SFTP:

FHS|^~&|LABSYS|HOSPITAL|REGISTRY|STATE|20260701120000||BATCH|FILE001|P|2.5.1
BHS|1|LABRESULTS|HOSPITAL|REGISTRY|STATE|20260701120000|FILE001|20260701
MSH|^~&|LABSYS|HOSPITAL|REGISTRY|STATE|20260701120000||ORU^R01^ORU_R01|MSG00001|P|2.5.1
PID|||LAB001^^^LAB||Doe^John||19700101|M|||123 Main St^^Chicago^IL^60601
OBR|1|LAB001|LAB001-1|1001-1^Glucose^LN|||20260701|
OBX|1|NM|1001-1^Glucose^LN||95|mg/dL|70-100|N|||F
BTS|1|1
MSH|^~&|LABSYS|HOSPITAL|REGISTRY|STATE|20260701120100||ORU^R01^ORU_R01|MSG00002|P|2.5.1
PID|||LAB002^^^LAB||Smith^Jane||19750315|F|||456 Oak Ave^^Denver^CO^80202
OBR|1|LAB002|LAB002-1|2001-5^Potassium^LN|||20260701|
OBX|1|NM|2001-5^Potassium^LN||4.2|mmol/L|3.5-5.0|N|||F
BTS|1|1
FTS|1|2

Annotations:
  FHS-1 = Field separator and encoding characters (^~&)
  FHS-3 = Sending application (LABSYS)
  FHS-4 = Sending facility (HOSPITAL)
  FHS-5 = Receiving application (REGISTRY)
  FHS-6 = Receiving facility (STATE)
  FHS-7 = Timestamp (20260701120000 = July 1, 2026, 12:00:00)
  FHS-9 = Message type (BATCH)
  FHS-10 = Batch control ID (FILE001)
  
  BHS-1 = Batch sequence number (1)
  BHS-2 = Batch message type (LABRESULTS)
  BHS-3 = Sending application
  ...
  
  [Two HL7 messages, each starting with MSH and including their segments]
  
  BTS-1 = Batch trailer; message count (1 = 1 message in this batch)
  BTS-2 = Batch message count (same as BTS-1 for single batches)
  
  FTS-1 = Number of batches in file (1)
  FTS-2 = Total message count across all batches (2)

Sample SFTP configuration (polling pattern)

A configuration for an SFTP batch receiver polling every 60 seconds:

Transport:
  Type: SFTP
  Protocol: HL7v2
  BatchMode: true

Connection:
  Host: sftp.hospital-lab.local
  Port: 22
  AuthMethod: PublicKey
  PrivateKeyFile: /etc/vorro/sftp-receiver-key.pem
  PrivateKeyPassphrase: ${VORRO_SFTP_KEY_PASSWORD}

SFTP:
  InboxDirectory: /sftp/incoming/adt/
  OutboxDirectory: /sftp/outgoing/adt/
  ArchiveDirectory: /sftp/archive/adt/

Polling:
  Enabled: true
  IntervalSeconds: 60
  MaxFilesPerPoll: 10
  FileNamePattern: "adt_[0-9]{8}_[0-9]{6}_[0-9]{3}.hl7"

FileHandling:
  AtomicWrite: true
  TempExtension: .tmp
  ProcessingExtension: .processing
  ArchiveProcessedFiles: true
  DeleteAfterDays: 30

Batch:
  ExpectFHS: true
  ExpectBHS: true
  ExpectFTS: true
  ValidateMessageCounts: true
  RejectIfAnyMessageInvalid: false  # Process valid messages even if one fails

Encoding:
  CharacterSet: UTF-8
  SegmentDelimiter: r
  FieldDelimiter: |

ErrorHandling:
  PartialDownloadAction: Rename  # Rename to .failed, not .processing
  ParseErrorAction: LogAndContinue  # Log error; process remaining messages
  DeleteOnSuccess: false
  MoveToArchive: true

Failure-mode example with error logs

Scenario: Receiver downloads a batch file, finds one invalid message, and handles the partial failure.

SFTP Server logs:

[2026-07-01 12:00:15] sshd[1234]: Accepted publickey for receiver_app from 192.168.1.101
[2026-07-01 12:00:16] sftp-server[1235]: session opened for user receiver_app
[2026-07-01 12:00:17] sftp-server[1235]: open "/sftp/incoming/adt/adt_20260701_120000_001.hl7" (read)
[2026-07-01 12:00:18] sftp-server[1235]: read 2048 bytes
[2026-07-01 12:00:18] sftp-server[1235]: close "/sftp/incoming/adt/adt_20260701_120000_001.hl7"
[2026-07-01 12:00:19] sftp-server[1235]: rename "/sftp/incoming/adt/adt_20260701_120000_001.hl7"
                                                 → "/sftp/incoming/adt/adt_20260701_120000_001.hl7.processing"
[2026-07-01 12:00:20] sshd[1234]: session closed

Receiver application logs:

[2026-07-01 12:00:17] SFTP: Connected to sftp.hospital-lab.local:22
[2026-07-01 12:00:18] SFTP: << [adt_20260701_120000_001.hl7] 2048 bytes
[2026-07-01 12:00:18] BATCH: Parsing FHS segment
[2026-07-01 12:00:18] BATCH: Batch file control ID = FILE001
[2026-07-01 12:00:19] BATCH: Parsing BHS segment
[2026-07-01 12:00:19] BATCH: Batch contains 2 messages
[2026-07-01 12:00:20] PARSE: Message 1 (MSG00001)
[2026-07-01 12:00:20] VALIDATE: PID segment present ✓
[2026-07-01 12:00:20] VALIDATE: OBR segment present ✓
[2026-07-01 12:00:21] INSERT: Inserted result for patient LAB001 into database
[2026-07-01 12:00:22] PARSE: Message 2 (MSG00002)
[2026-07-01 12:00:22] VALIDATE: PID segment present ✓
[2026-07-01 12:00:22] VALIDATE: OBR segment present ✗ (MISSING)
[2026-07-01 12:00:22] ERROR: Message 2 (MSG00002) invalid: missing required OBR segment
[2026-07-01 12:00:23] ALERT: Batch FILE001: 1/2 messages processed, 1 failed
[2026-07-01 12:00:24] ARCHIVE: Moved to /sftp/archive/adt/adt_20260701_120000_001.hl7 (1 pass, 1 fail)
[2026-07-01 12:00:25] NOTIFY: Sent alert to integration-team@hospital.local
[2026-07-01 12:00:25] SFTP: Connecting to upload ACK batch file
[2026-07-01 12:00:26] SFTP: >> [adt_20260701_120000_001.ack] (partial ACK with errors)
[2026-07-01 12:00:27] SFTP: Disconnected

Receiver's ACK batch file (partial success):

FHS|^~&|REGISTRY|STATE|LABSYS|HOSPITAL|20260701120027||BATCH^ACK|FILE001.ACK|P|2.5.1
BHS|1|LABRESULTS^ACK|REGISTRY|STATE|LABSYS|HOSPITAL|20260701120027|FILE001.ACK|
MSA|AA|MSG00001||A|2.5.1
MSA|AE|MSG00002|203^Required segment missing|E|2.5.1
BTS|2|2
FTS|1|2

Vorro support for SFTP

Vorro provides full SFTP batch transport support:

SFTP listener (receiver): Periodically polls SFTP directories, downloads batch files, parses FHS/BHS/BTS/FTS structures, and inserts messages into the data warehouse.
SFTP sender (uploader): Batches outbound messages, wraps them in FHS/BHS/BTS/FTS segments, applies atomic write patterns, and uploads to remote SFTP servers.
SSH key management: Supports public-key authentication with encrypted key storage and rotation.
Atomic file handling: Implements .tmp → final name renaming and .processing markers to prevent duplicate processing.
Configurable polling: Tune polling intervals, batch size, and file naming patterns per integration.
Partial batch processing: Optionally process valid messages even if some messages in a batch fail validation.
ACK batch generation: Automatically generates FHS/BHS/MSA/FTS acknowledgment batches and uploads them back to the sender.
Archive and audit: Logs all SFTP operations and optionally archives processed files for compliance.

Sources

← Back to HL7 v2 Guide