Zero-downtime maintenance for Apache ActiveMQ® is a discipline, not a configuration parameter. It requires understanding which operations require a restart and which do not, how to sequence HA failover to make restarts transparent, how to configure both the broker and Kubernetes to honor the graceful shutdown window, and how to validate that the maintenance operation achieved its goal without introducing new problems.
This guide covers the complete Apache ActiveMQ® zero-downtime maintenance procedure set: the maintenance type decision matrix, graceful shutdown for Apache ActiveMQ® and Apache Artemis™, the HA rolling restart pattern, Kubernetes drain procedures, hot-reloadable configuration, KahaDB maintenance operations, and the pre/post maintenance checklists.
Maintenance Type Decision Matrix
Not every maintenance operation requires a broker restart. Understanding which operations can be performed online saves maintenance windows and reduces operational risk.
| Operation | Restart Required? | Method | Risk Level |
|---|---|---|---|
| Log level change | No | log4j2 monitorInterval hot-reload | Low |
| Apache Artemis™ address settings change | No | broker.xml hot-reload (Apache Artemis™ detects file changes) | Low |
| Security role/policy update (Apache ActiveMQ®) | No | reload=true on authorizationPlugin | Low |
| memoryUsage limit change | No (via JMX) | JMX setBrokerService attributes | Low |
| KahaDB journal compaction | No | JMX gc() on KahaDB MBean | Low |
| Destination purge | No | JMX purge() on destination MBean | Medium |
| JVM flags change | Yes | Restart via HA rolling procedure | Medium |
| Transport connector change | Yes | Restart via HA rolling procedure | Medium |
| Persistence adapter change | Yes | Restart with full backup prerequisite | High |
| OS patching / node maintenance | Varies | K8s drain or HA failover | Medium |
| ActiveMQ® version upgrade | Yes | Upgrade procedure (see Post #19) | High |
The Shutdown Spectrum: SIGTERM vs. SIGKILL vs. Graceful
Every Apache ActiveMQ® broker shutdown falls somewhere on a spectrum from clean to catastrophic:
The Correct Approach: Broker Shutdown CLI or SIGTERM
| # Classic: clean shutdown via the activemq control script /opt/activemq/bin/activemq stop # Sends SIGTERM to the broker process # Broker: flushes KahaDB journal, closes network connectors, # closes client connections, checkpoints index # Artemis: clean shutdown via the artemis CLI /opt/artemis-instance/bin/artemis-service stop # Or if running directly: /opt/artemis-instance/bin/artemis stop # Verify the broker stopped cleanly (check exit code and log) grep “stopped” /opt/activemq/data/activemq.log | tail -5 # Expected: INFO | Apache ActiveMQ stopped |
On clean shutdown:
- KahaDB journal is flushed: no recovery replay needed on next startup
- Network connector bridges are closed gracefully: remote brokers detect the disconnect immediately
- Client connections receive an IOException with a meaningful disconnect message: failover transport can reconnect immediately to a backup broker
- JVM exits normally: heap dump is not triggered
The Catastrophic Approach: SIGKILL
| # DO NOT USE IN PRODUCTION kill -9 $(pgrep -f activemq) # Force-kills the JVM with no cleanup # KahaDB journal left in incomplete state → replay required on restart # In-flight journal writes may be partially written → potential corruption # Client connections receive abrupt TCP RST → failover transport # must wait for TCP timeout before detecting disconnect (up to 30s) |
SIGKILL bypasses all cleanup paths. Use it only as a last resort when the broker process is unresponsive to SIGTERM and has been so for more than 60 seconds. When SIGKILL is used, always check the KahaDB directory for corruption on next startup and ensure checksumJournalFiles=true is enabled.
Apache Artemis™ Graceful Shutdown
When graceful-shutdown-enabled=true and the broker is shut down, it will first prevent any additional clients from connecting and then wait for any existing connections to be terminated by the client before completing the shutdown process.
| <!– broker.xml — Artemis graceful shutdown configuration –> <configuration> <core> <!– Enable graceful shutdown: stop accepting new connections, wait for existing clients to close before completing shutdown –> <graceful-shutdown-enabled>true</graceful-shutdown-enabled> <!– Maximum wait time for clients to disconnect (milliseconds) –> <!– After this timeout, broker forces shutdown regardless –> <!– Default: Long.MAX_VALUE (wait forever) — set a reasonable bound –> <graceful-shutdown-timeout>30000</graceful-shutdown-timeout> </core> </configuration> |
What graceful shutdown enables: with graceful-shutdown-enabled=true, the Apache Artemis™ broker’s shutdown sequence is:
- Stop the acceptors: no new connections accepted
- Wait for existing client connections to close naturally (up to graceful-shutdown-timeout)
- If clients are using the failover transport, they receive a redirect signal and reconnect to the backup broker before the primary closes
- After all clients disconnect (or timeout is reached), complete shutdown
This means clients using the failover transport experience the broker shutdown as a transparent connection switch to the backup, not as an error. The failover transport reconnects within the graceful-shutdown-timeout window, well before the broker fully closes.
The allow-failback interaction: if allow-failback=false, the backup server will remain passive if this broker is shutdown gracefully. For a planned maintenance restart where you want the backup to activate, either set allow-failback=true or trigger failover explicitly via the management API before initiating the graceful shutdown.
The HA Rolling Restart: Zero Client Disruption
For HA deployments (Apache ActiveMQ® Master/Slave or Apache Artemis™ live-backup replication), the rolling restart achieves genuine zero client disruption. Clients using the failover transport are redirected to the backup before the primary goes down, experience a 2–5 second reconnection pause, and resume processing without any application-level errors.
Apache ActiveMQ® HA Rolling Restart Procedure
| # ============================================================ # Classic HA Rolling Restart # Assumptions: # – DC1-PRIMARY (broker1) is the active master # – DC1-BACKUP (broker2) is the passive slave # – Clients use: failover:(tcp://broker1:61616,tcp://broker2:61616) # ?priorityBackup=true&randomize=false # ============================================================ # STEP 1: Verify backup is synchronized before proceeding # Check broker2 log for sync completion ssh broker2 “grep ‘slave’ /opt/activemq/data/activemq.log | tail -3” # Expected: INFO | Slave started # STEP 2: Pre-maintenance backup of broker1 ssh broker1 “/opt/activemq/bin/activemq stop” sleep 5 ssh broker1 “cp -a /opt/activemq/data/kahadb /opt/activemq/data/kahadb_backup_$(date +%Y%m%d)” ssh broker1 “/opt/activemq/bin/activemq start” sleep 15 # Wait for broker1 to resume master role before proceeding # STEP 3: Stop broker1 cleanly — broker2 automatically becomes master ssh broker1 “/opt/activemq/bin/activemq stop” # Monitor broker2 log for promotion to master ssh broker2 “grep ‘Master’ /opt/activemq/data/activemq.log | tail -3” # Expected: INFO | Starting as Master # Clients reconnect to broker2 automatically via failover transport # STEP 4: Perform maintenance on broker1 (upgrade, config change, OS patch) # … (apply maintenance operations) … # STEP 5: Start broker1 — it rejoins as slave ssh broker1 “/opt/activemq/bin/activemq start” # Verify broker1 comes up as slave (not stealing master from broker2) ssh broker1 “grep -E ‘slave|Slave’ /opt/activemq/data/activemq.log | tail -3” # STEP 6 (optional): Fail back to broker1 as master # Only if clients should prefer broker1 (priorityBackup=true on client) # Wait for broker1 to sync fully with broker2, then: ssh broker2 “/opt/activemq/bin/activemq stop” # broker1 automatically promotes to master # broker2 restarts as slave ssh broker2 “/opt/activemq/bin/activemq start” echo “Rolling restart complete. Total client disruption: ~2-5 seconds during broker2→broker1 failover.” |
Apache Artemis™ HA Rolling Restart Procedure
| # ============================================================ # Artemis Live-Backup Rolling Restart # Assumptions: # – artemis-primary (DC1) is the live broker # – artemis-backup (DC1) is the passive backup # ============================================================ # STEP 1: Verify backup is synchronized ssh artemis-primary “grep ‘AMQ221024’ /opt/artemis/log/artemis.log | tail -1” # AMQ221024: Replication synchronized with backup node # STEP 2: Enable graceful-shutdown-enabled=true on the primary (if not already set) # (edit broker.xml or verify it is already configured) # STEP 3: Trigger graceful shutdown on the primary # The backup activates; clients reconnect to backup via failover transport ssh artemis-primary “/opt/artemis-instance/bin/artemis-service stop” # Monitor backup log for promotion ssh artemis-backup “grep ‘AMQ221109’ /opt/artemis-backup/log/artemis.log | tail -1” # AMQ221109: Apache ActiveMQ Artemis is now live # STEP 4: Perform maintenance on primary # … (apply maintenance) … # STEP 5: Restart primary — it reconnects as backup ssh artemis-primary “/opt/artemis-instance/bin/artemis-service start” # Primary starts as backup, synchronizes with the now-live backup ssh artemis-primary “grep ‘AMQ221024\|slave\|backup’ /opt/artemis/log/artemis.log | tail -3” # STEP 6: Optional failback — primary reasserts live role # Requires allow-failback=true in broker.xml ha-policy # When primary re-synchronizes, it automatically takes over if allow-failback=true |
Planning a Maintenance Window on a Production Apache ActiveMQ® Deployment?
Kubernetes: Zero-Downtime Broker Pod Maintenance
For Apache ActiveMQ® on Kubernetes (covered in our Apache ActiveMQ® on Kubernetes post), zero-downtime maintenance requires coordinating Kubernetes eviction controls with broker-level graceful shutdown.
PodDisruptionBudget: Protect Against Involuntary Eviction
| # Prevents Kubernetes from evicting the broker pod during node drains # unless minAvailable is satisfied by other pods apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: activemq-pdb namespace: messaging spec: minAvailable: 1 selector: matchLabels: app: activemq |
With minAvailable: 1, kubectl drain on a node hosting the Apache ActiveMQ® pod will block until the pod is rescheduled and Ready on another node. For a single-replica broker, this means node drain requires the pod to successfully restart elsewhere before the drain completes.
terminationGracePeriodSeconds: Critical for KahaDB Flush
The most common Kubernetes maintenance error for Apache ActiveMQ®: the default terminationGracePeriodSeconds is 30 seconds. If broker shutdown (including KahaDB journal flush and journal replay protection) takes longer than 30 seconds, Kubernetes sends SIGKILL before the broker can complete clean shutdown.
| # StatefulSet pod spec — align terminationGracePeriodSeconds with broker shutdown time spec: template: spec: # Must be > time required for clean broker shutdown # For large KahaDB: 120-300 seconds is appropriate terminationGracePeriodSeconds: 120 containers: – name: activemq # … # preStop hook: drain the broker before Kubernetes terminates the container lifecycle: preStop: exec: command: – /bin/sh – -c – | # Classic: initiate graceful stop before SIGTERM /opt/activemq/bin/activemq stop # Wait for the process to exit sleep 5 # If still running after 5s, proceed — SIGTERM will handle it |
Why preStop matters: Kubernetes sends SIGTERM to the container and simultaneously starts the terminationGracePeriodSeconds countdown. The preStop hook runs before SIGTERM, giving the broker time to initiate its own clean shutdown sequence (flush journal, close connections) before the countdown begins. Without preStop, SIGTERM arrives while the broker is in the middle of processing, relying entirely on the broker’s SIGTERM handler to clean up within the grace period.
StatefulSet Rolling Update
For configuration changes that require a pod restart, use Kubernetes’s native rolling update:
| # StatefulSet update strategy — controls how pod replacements happen spec: updateStrategy: type: RollingUpdate rollingUpdate: # partition=1: update only pods with ordinal >= 1 # Use partition to update pods incrementally (canary approach) partition: 0 # Trigger rolling update with new image or config change kubectl set image statefulset/activemq \ activemq=apache/activemq-classic:5.18.4 \ -n messaging # Watch the rolling update kubectl rollout status statefulset/activemq -n messaging # For multi-replica HA StatefulSet: use partition to update one pod at a time # Update pod with ordinal >= 1 first (the backup), validate, then update pod 0 (primary) kubectl patch statefulset activemq -n messaging \ -p ‘{“spec”:{“updateStrategy”:{“rollingUpdate”:{“partition”:1}}}}’ kubectl set image statefulset/activemq activemq=apache/activemq-classic:5.18.4 -n messaging # Wait for pod 1 (backup) to update and become Ready kubectl rollout status statefulset/activemq -n messaging # Then update pod 0 (primary) kubectl patch statefulset activemq -n messaging \ -p ‘{“spec”:{“updateStrategy”:{“rollingUpdate”:{“partition”:0}}}}’ |
Hot-Reloadable Configuration: What You Can Change Without Restarting
Log Levels: Instant Hot-Reload
The most frequently used hot-reload capability. As covered in our Log Analysis & Diagnostics post, both Apache ActiveMQ® and Apache Artemis™ use Log4j2 with monitorInterval:
| # conf/log4j2.properties (Classic) or etc/log4j2.properties (Artemis) monitorInterval = 30 # Reload config every 30 seconds # To change a log level without restart: # 1. Edit this file: change logger.transport.level = INFO to DEBUG # 2. Save the file # 3. Within 30 seconds, the change takes effect # 4. After debugging, revert to INFO and save — reverts within 30 seconds |
Apache Artemis™ Address Settings: File-Based Hot-Reload
Apache Artemis™ monitors broker.xml for changes to the address-settings section at runtime:
| <!– broker.xml — changes to address-settings are hot-reloaded –> <!– Artemis detects file modification and applies changes within seconds –> <address-settings> <!– Increase max size for a queue under pressure — no restart required –> <address-setting match=”orders.#”> <max-size-bytes>524288000</max-size-bytes> <!– Changed from 200MB to 500MB –> <address-full-policy>BLOCK</address-full-policy> </address-setting> </address-settings> |
Note: hot-reload applies to address-settings changes. Changes to acceptors, connectors, ha-policy, or persistence elements are NOT applied dynamically. They require a broker restart.
Apache ActiveMQ® Security Reload
| <!– activemq.xml — security plugin with hot-reload enabled –> <plugins> <jaasAuthenticationPlugin configuration=”activemq”/> <authorizationPlugin> <!– reload=true: security policies are re-read from file on each access check –> <!– Allows adding/removing permissions without broker restart –> <map> <authorizationMap> <authorizationEntries> <authorizationEntry queue=”>” read=”admins,producers,consumers” write=”admins,producers” admin=”admins”/> </authorizationEntries> </authorizationMap> </map> </authorizationPlugin> </plugins> |
JMX Runtime Configuration Changes (Apache ActiveMQ®)
Several Apache ActiveMQ® broker parameters can be changed via JMX without a restart:
| # Via JConsole or jmxterm — change memoryUsage limit at runtime # MBean: org.apache.activemq:type=Broker,brokerName=localhost # Operation: invoke setBrokerService with updated systemUsage # Via activemq CLI /opt/activemq/bin/activemq bstat # Shows current broker statistics — use to verify JMX changes took effect # Programmatic JMX update (from monitoring tool or script) # org.apache.activemq:type=Broker,brokerName=* # Attribute: MemoryPercentUsage (read) # Operation: resetStatistics() to reset counters without restart |
KahaDB Online Maintenance Operations
Journal Compaction (Online, No Restart Required)
KahaDB accumulates journal files as messages are written and acknowledged. Acknowledged messages leave “holes” in journal files. Compaction removes these holes, reclaiming disk space and improving read performance. This operation can be performed while the broker is running.
| # Via JMX: trigger KahaDB compaction (Classic) # MBean: org.apache.activemq:type=Broker,brokerName=localhost, # service=PersistenceAdapter # Operation: gc() # Via activemq CLI (if enabled) /opt/activemq/bin/activemq bstat | grep -i store # Check current store percentage before and after compaction # Expected: StorePercentUsage drops after compaction completes # Note: compaction is a background operation — monitor store usage over 5-10 minutes |
When to run compaction: when StorePercentUsage is elevated, but queue depths suggest the store should be smaller. If StorePercentUsage is 80% but total pending messages represent only 20% of configured capacity, journal files are accumulating unreclaimed space.
Index Rebuild (Requires Restart)
If the KahaDB index (db.data) is suspected to be corrupted or inconsistent, it must be rebuilt offline:
| # Requires broker shutdown (covered in Backup & DR post) # Step 1: Stop broker /opt/activemq/bin/activemq stop # Step 2: Back up existing index cp /opt/activemq/data/kahadb/db.data /opt/activemq/data/kahadb/db.data.bak cp /opt/activemq/data/kahadb/db.redo /opt/activemq/data/kahadb/db.redo.bak # Step 3: Delete index files (journal files preserved) rm /opt/activemq/data/kahadb/db.data rm /opt/activemq/data/kahadb/db.redo # Step 4: Restart — broker replays journal to rebuild index /opt/activemq/bin/activemq start # Monitor: grep “Recovery replayed” /opt/activemq/data/activemq.log |
We covered the complete KahaDB corruption recovery procedure in our Backup & DR post.
Destination Purge via JMX (Online)
For queues that have accumulated messages that should be deleted (e.g., poison messages, stale test data):
| # Via JConsole or activemq-cli: # Classic MBean: org.apache.activemq:type=Broker,brokerName=*, # destinationType=Queue,destinationName=orders.test # Operation: purge() # Artemis CLI: /opt/artemis-instance/bin/artemis queue purge \ –name orders.test \ –user admin –password admin # WARNING: purge() deletes ALL messages in the queue immediately # Verify the queue name exactly before running # Consider moveMatchingMessagesTo() to move to DLQ for inspection instead |
Network of Brokers: Maintenance Without Disrupting the Mesh
When performing maintenance on a node in a Network of Brokers topology, the sequence matters to prevent message loss or routing disruption:
| # NoB Maintenance Sequence for DC1 Hub Broker # STEP 1: Stop producers sending to DC1 hub # (or reduce rate to allow the drain) # STEP 2: Wait for DC1 hub queues to drain to zero # Monitor via JMX: QueueSize on all destination MBeans watch -n 5 “grep ‘QueueSize’ <(curl -s http://dc1:8161/api/jolokia/read/…)” # STEP 3: Once queues empty, remove DC1 from NoB mesh gracefully # On spoke brokers: stop the networkConnector to DC1 # (or let it disconnect — spokes will buffer and reconnect on DC1 restart) # STEP 4: Perform maintenance on DC1 # STEP 5: Restart DC1 hub # Spoke brokers reconnect automatically (networkConnector retry logic) # DC1 hub re-establishes connections to all spokes # STEP 6: Verify mesh connectivity grep “network bridge” /opt/activemq/data/activemq.log | tail -10 # Expected: INFO | Network connection between … established |
We covered NoB configuration and topology in our Network of Brokers Configuration post, and multi-DC maintenance sequencing in our Multi-Datacenter Deployment Patterns post.
Real-Time Broker State Visibility During Maintenance Operations
Pre and Post-Maintenance Checklists
Pre-Maintenance Checklist
□ 1. BACKUP: Complete backup of broker data and configuration
– KahaDB directory (Classic) or journal directory (Artemis)
– activemq.xml / broker.xml, bin/env / artemis.profile
– Verify backup integrity (restore test in staging, or md5sum check)
□ 2. HA VALIDATION (for HA rolling restart)
– Verify backup/slave is synchronized with primary/master
– Confirm backup broker is reachable and healthy
– Classic: grep “Slave” /opt/activemq/data/activemq.log
– Artemis: grep “AMQ221024” /opt/artemis/log/artemis.log
□ 3. QUEUE DRAIN STATUS
– Document current queue depths for all critical destinations
– Confirm consumer counts are at expected levels
– Note any queues with unusual depth (potential slow consumers)
□ 4. CLIENT FAILOVER VERIFICATION
– Confirm client applications use failover transport (Classic)
or static connector list with reconnect-attempts=-1 (Artemis)
– Verify failover was tested in staging within last 90 days
□ 5. MONITORING ACTIVE
– Confirm Prometheus / MeshIQ Console is collecting metrics
– Alert thresholds are active (not silenced)
– Log monitoring is active for ERROR and WARN patterns
□ 6. ROLLBACK PLAN DOCUMENTED
– Identify the rollback trigger condition
– Document the exact rollback steps
– Identify who has authority to call rollback
– Estimate rollback time (should be < 10 minutes)
□ 7. NOTIFICATION SENT
– Application teams notified of maintenance window
– On-call engineer confirmed available during maintenance
Post-Maintenance Validation
| #!/bin/bash # Post-maintenance validation script LOG=”/opt/activemq/data/activemq.log” # Adjust for Artemis echo “=== 1. Version Confirmation ===” grep -m1 “Apache ActiveMQ.*started” “$LOG” echo “=== 2. No Errors in First 5 Minutes Post-Start ===” START_TIME=$(grep -m1 “started” “$LOG” | cut -d’ ‘ -f1,2) # Check for ERRORs after start timestamp grep “ERROR” “$LOG” | tail -20 echo “=== 3. All Connectors Started ===” grep “Connector.*started” “$LOG” echo “=== 4. HA Synchronization ===” grep -E “Master|Slave|AMQ221024|replication” “$LOG” | tail -5 echo “=== 5. Queue Depths Returned to Pre-Maintenance Baseline ===” # Compare via JMX / Console with documented pre-maintenance depths echo “=== 6. Client Reconnection Confirmed ===” grep “TotalConnectionsCount” “$LOG” | tail -3 # Compare to pre-maintenance connection count echo “=== 7. No PFC Events Post-Maintenance ===” grep “Stopping producer” “$LOG” | grep “$(date +%Y-%m-%d)” # Should be empty if maintenance resolved the capacity concern echo “=== 8. GC Log Check ===” grep “Pause Full” /opt/activemq/data/gc.log | tail -5 # Any full GC in first 10 minutes post-restart is a warning signal echo “Validation complete. Review output for any anomalies.” |
Maintenance Is a First-Class Operation
Zero-downtime maintenance is not achieved by being lucky or moving fast. It is achieved by having the right procedures, validated in staging, with the right tooling (HA for transparent failover, graceful shutdown for orderly connection drain, Kubernetes PDB for pod protection), and the right checklists (pre-maintenance backup, post-maintenance validation).
Every procedure in this guide has an inverse: the rollback. Knowing the rollback before starting the maintenance operation is the difference between a 10-minute recovery and a multi-hour incident.
Get your Apache ActiveMQ® maintenance procedures reviewed by our team → Request Maintenance Support
Frequently Asked Questions
Q: How do I restart Apache ActiveMQ® without losing messages?
For persistent messages: use the proper stop command (not SIGKILL). Persistent messages survive a clean restart via journal recovery. For zero client disruption: use HA rolling restart. For Kubernetes: configure terminationGracePeriodSeconds > broker shutdown time and use a preStop lifecycle hook.
Q: Can I reload Apache ActiveMQ® configuration without restarting?
Yes for: log4j2 levels (both Apache ActiveMQ® and Apache Artemis™ via monitorInterval), Apache Artemis™ address settings (broker.xml file change), Apache ActiveMQ® security plugins (reload=true), JMX-accessible attributes (memoryUsage). No for: transport acceptors, persistence adapter, JVM flags, HA policy. These require restart.
Q: How do I drain an Apache ActiveMQ® broker before maintenance?
Stop or redirect producers, monitor queue depths via JMX until all reach zero, verify no active consumers remain, then shut down. For NoB, disconnect the bridge connector first to prevent new messages from remote brokers during the drain window.
Q: What is graceful shutdown in Apache Artemis™?
graceful-shutdown-enabled=true in broker.xml causes Apache Artemis™ to stop accepting new connections and wait for existing clients to close naturally (up to graceful-shutdown-timeout) before completing shutdown. This gives failover transport clients time to reconnect to the backup broker before the primary fully closes.
Q: How do I perform maintenance on a Kubernetes ActiveMQ® deployment?
Use PodDisruptionBudget with minAvailable:1 to prevent eviction during node drains. Set terminationGracePeriodSeconds ≥ 120 seconds. Add a preStop lifecycle hook calling the broker stop CLI. For HA StatefulSets, use rolling update with partition to update the backup pod first, then the primary.