DIY Exercise 13-1: Create a batch job to process records

Time estimate: 2 hours

Objectives
In this exercise, you create a batch processing system. You will:
• Control batch step processing using batch filters.
• Exchange data between batch steps using variables.
• Trigger batch processing on a schedule.
• Use watermarks to avoid duplicate processing.
• Improve performance by using batch aggregators.
• Improve performance and share data with other Mule applications using VM queues.

Scenario
The Finance department needs to audit certain transactions and needs a Mule application to
consistently retrieve data from a database and write these transactions as CSV files to a server.

To meet compliance standards, a CSV file can have no more than 50 records and the Mule
application must be deployed to a private server where the Mule application will share the same
Mule domain with other financial compliance Mule applications. You do not, however, need to
create a new Mule domain project yourself; another developer will be responsible for deploying
your project into an existing Mule domain.

Create a project that retrieves new transactions from the database using
batch
Create a new Mule application that retrieves data from the flights_transactions table in the
database using the following information:
• Host: mudb.learn.mulesoft.com
• Port: 3306
• User: mule
• Password: mule
• Database: training
• Table: flights_transactions
Schedule the main flow to automatically run every 5 seconds. Retrieve new database records
based on the value of the primary key field transactionID. Use an ObjectStore to save the
maximum transactionID processed for any batch session.
Hint: For test development, limit the query to only retrieve 10 records at a time.
Add a flow to mock the financial compliance application logic
Add a new flow to the Mule application with a VM Listener on the VM queue named validate.
Add a Transform Message component to this flow and add DataWeave code to simulate the
transactionID validation logic. It expects one record and returns the value true or false, where
true indicates that the record needs to be audited. In this simple mock flow, return a Boolean
value true if the transactionID is divisible by 4, and false otherwise:
%dw 2.0
output application/java
---
if ( mod(payload.transactionID as Number,4) == 0 )
true
else
false

Send each transaction record to a VM queue for conditional testing

Publish each transaction record to the validate VM queue and wait to consume the Boolean
response. Save the result in a variable to filter the current record in the next batch step.

Add batch filters to only process transactions that need auditing

Configure target variables and the accept expression in each batch step to keep track of your
records throughout each batch step.
Hint: For test development, you can create a flow that listens on the validate queue path and
arbitrarily return true or false for each record processed.

Write out transactions as CSV files

In a second batch step, configure an accept expression to only process this second batch step if
the previous VM queue response was true. Inside this batch step, transform the database
results to a CSV file and save this CSV file to this Mule application's file system. Use a property
placeholder for the file location so the file location can be modified by Ops staff at deployment
time. Add a batch aggregator so no more than 50 records at a time are written to each CSV file.

Log the batch processing summary

In the On Complete phase of the Batch Job, log the batch processing summary.

Test your solution

Debug your solution. Step through several polling steps and verify some queries return true
from the VM queue and are processed by the second batch step, but other database queries
return false and skip the second batch step. Also verify the output CSV files contain at most 50
records each.

Verify your solution

Import the solution /files/module13/audit-mod13-file-write-solution.jar deployable archive file (in
the MUFundamentals4.x DIY Files zip that you can download from the Course Resources) and
compare your solution.

Going further: Handle errors

Add logic to the first batch step to throw errors.
• Call a flow at the beginning of the first batch step and add event processors to this flow
that would sometimes throw an error, but not for every record.
• Experiment with what happens when you handle the error in the flow versus if you don't
handle the error in the flow.
• Add a third batch step with an ONLY_FAILURES accept policy to report failed messages
to a dead letter VM queue for failed batch steps.
• In the Batch Job scope's general configuration, change the max failed records to 1 and
observe the behavior of subsequent batch records after a record throws an error.
Change this value to 2 and observe any changes in behavior.
• In the Batch Job scope's general configurations, change the scheduling strategies
options to ROUND_ROBIN and observe the behavior, and compare it with the default
ORDERED_SEQUENTIAL option's behavior.
• Look at the logs for the On Complete phase to see how many times the same error is
reported for each record of the batch job.
• In the first batch step's referenced flow, add a Choice router and a sleep timer that
sleeps for a minute. Add logic to the Choice router to only call the sleep timer if the
transactionID ends with 6. Observe if later records in the batch job can skip ahead while
some records are paused by the sleep timer.
Note: For info about handling batch errors, see: https://blogs.mulesoft.com/dev/mule-
dev/handle-errors-batch-job/

Going further: Refactor the validation logic to another Mule application in a

new shared Mule domain
Create a Mule domain project named finance, then change the Mule application's Mule domain
from default to finance. Move the VM connector global element to the finance Mule domain
project.
Move the validation flow to a new Mule application and configure this Mule application to also
use the finance Mule domain.
Deploy the Mule domain and both Mule applications to a customer-hosted Mule runtime. Verify
batch jobs are still processed correctly.

Going further: Deploy both Mule applications to CloudHub

Instead of using customer-hosted Mule runtimes, configure both Mule applications to use an
external online JMS server that is accessible over the public internet. Deploy both Mule
applications to CloudHub and verify batch jobs are still processed correctly.