How To Send Syslog Messages From Azure Linux Vms To An Azure EventHub Part 1

6/8/22, 2:01 PM How to send Syslog messages from Azure Linux Vms to an Azure EventHub Part 1
Adrian Corona
Sign in to LinkedIn with Google
Ferdinand Escolar
ferdinand.escolar@itsmeph.com
Continue as Ferdinand
To create your account, Google will share your name,

email address, and profile picture with LinkedIn. See
LinkedIn's privacy policy and terms of service.
How to send Syslog messages from Azure

Linux Vms to an Azure EventHub Part 1
Adrian Corona
Head of Security and Compliance GTM at Google + Follow
Published Nov 20, 2018
A very big part of my job consists on helping our customers create a solid governance
and security framework as they move to the cloud, a very common request is that of
logging VMs data, specifically Syslog. While we have Log Analytics, a great tool for
log aggregation and intelligence, I also found many customers are using other SIEM
solutions like Splunk, and they all want to know how can they keep the investment they
already made on it as well as continue with a centralized logging location.
https://www.linkedin.com/pulse/how-send-syslog-messages-from-azure-linux-vms-eventhub-adrian-corona 1/13
One way to collect OS logs is deploying the universal log forwarder inside the VMs
Adrian Corona
themselves, while this works great it means managing yet
Sign in to another
LinkedIn agent,
with like if we didn't
Google
have enough agents to deploy and manage already yuck!

Ferdinand Escolar
Since I want to enjoy my life, instead of pushing individual agents, I'm going to use a
really cool feature: Azure VM Extensions, specifically Linux Diagnostic Extension.
This extension allows you to send VM Metrics, Diagnostics and Syslog data to Blob
Storage or Event Hubs, very important to note that this
LinkedIn's process
privacy policycan
and apply
terms ofto windows
service.
machines as well, you'd have to tweak the process a bit but let me know in your
comments if you think it would make a good write up. The best of this agent is that it
can be pushed from the control plane, so no need to manage individual vm agent
installations and free up some time to write more blogs.
In Part 1 I will show you how to configure the VM agent, the destination storage
account and event hubs, part 2 will be configuring Splunk ingestion so stay tuned!
Prerequisites:
Create Storage Account
I'm not going into the full details on how to create a storage account, lots of docs about
that you can just run this command and modify parameters:
az storage account create -g ResourceGroupName -n storageaccountname --sku Sta
Create EventHub
We need an eventhub to dump all our events we will create 2 hubs, one for syslog and
one for metrics, you can tweak this further but for now this will do.
Adrian
#Create Corona
Namespace
Sign in toResourceGroupName
az eventhubs namespace create --resource-group LinkedIn with Google --name myname
#Create 2 Eventhubs, one for Syslog and One for metrics Data
Ferdinand
az eventhubs eventhub create --resource-group Escolar
ResourceGroupName --namespace-na
az eventhubs eventhub create --resource-group ResourceGroupName --namespace-na

Agent Installation
This is the version 3.0 of the agent, if you are running version 2.3 I strongly suggest you
update to the new version as none of this will work, you must uninstall the old extension
(under the old publisher name), then install version 3 of the extension, you can use the
following az cli commands
#list vm extensions
az vm extension list --output table -g Resource_Group --vm-name VMNAME
#delete vm extension
azure vm extension delete --name LinuxDiagnostic --vm-name VMNAME -g Resource_
Once that's done, let's talk about settings, after installing the agent you will see a number
of new settings in the portal (see below screenshot).In fact, syslog settings are there as
well, however, to configure EventHub logging you MUST follow this process.
Adrian Corona
Ferdinand Escolar

There are a bunch of different things you can do with the extension, for the latest info on
Adrian Corona
it you should check out their Github repo but basically
Sign in tothe extension
LinkedIn settings are divided
with Google
in two separate JSON files Private and Public.

Ferdinand Escolar
Private (Protected) Settings:

This data is considered sensitive and contains:

Storage Account information (Account name and keys) - This is pretty self
explanatory, make sure you retrieve a valid SAS Token, you can get one in many ways,
you can generate a SAS token from the Azure portal as show below:
Sinks configuration: This setting will tell the agent what type of destination storage
will be used by the metrics defined below either Storage account or EventHub (NOTE:
Data will always be stored as table storage as well, you can't turn this option off).
In the the sinks configuration section you need to add a different SAS tokens than the
Adrian Corona
one created for the storage account, this one is specific
Sign in tofor EventHub,
LinkedIn with Google
Ferdinand Escolar
You will need the following data: ferdinand.escolar@itsmeph.com

You can retrieve a new SAS Token in multiple ways here's an example using az cli,
make sure to install jq (apt-get install jq)
#first run this whole function as-is
get_sas_token() {
local EVENTHUB_URI=$1
local SHARED_ACCESS_KEY_NAME=$2
local SHARED_ACCESS_KEY=$3
local EXPIRY=${EXPIRY:=$((60 * 60 * 24))} # Default token expiry is 1 day,

local ENCODED_URI=$(echo -n $EVENTHUB_URI | jq -s -R -r @uri)
local TTL=$(($(date +%s) + $EXPIRY))
local UTF8_SIGNATURE=$(printf "%s\n%s" $ENCODED_URI $TTL | iconv -t utf8)
local HASH=$(echo -n "$UTF8_SIGNATURE" | openssl sha256 -hmac $SHARED_ACCE

local ENCODED_HASH=$(echo -n $HASH | jq -s -R -r @uri)
echo -n "SharedAccessSignature sr=$ENCODED_URI&sig=$ENCODED_HASH&se=$TTL&s

}
#then run the following using your parameters
get_sas_token 'mynamespace.servicebus.windows.net/syslog' 'RootManageSharedAcc
#repeat this step for the metrics hub
get_sas_token 'mynamespace.servicebus.windows.net/metrics' 'RootManageSharedAc
Adrian Corona
Each time you run this, you'll get an output just like the one below, save each token for
the next part Ferdinand Escolar
SharedAccessSignature sr=mynamespace.servicebus.windows.net%2Fsyslog&sig=CzswM
With this you are now ready to create your file, go ahead and create a new file, paste the
following code and replace the variables with your info.
"storageAccountName": "STORAGEACCOUNTNAME",
"storageAccountSasToken": "SAS Token (starting with sv=)",
"sinksConfig": {
"sink": [
"name": "SyslogJsonBlob",
"type": "JsonBlob"
},
"name": "SyslogEventHub",
"type": "EventHub",
"sasURL": "https://mynamespace.servicebus.windows.net/syslog?sr=myname
},
"name": "LinuxCpuJsonBlob",
"type": "JsonBlob"
},
"name": "LinuxCpuEventHub",
"type": "EventHub",
"sasURL": "https://mynamespace.servicebus.windows.net/metrics?sr=mynam
}
Once done, save and name it protected_settings.json. As you can tell this is a very
simple file, it will add Syslog and Metrics to both storage account and EventHub, the
"name" parameter is just a friendly name, you can add whatever you like, type is either
Adrian Corona
JsonBlob or EventHub. Sign in to LinkedIn with Google
Ferdinand Escolar
Public Settings:
This file contains the definitions of the metrics that will
LinkedIn's be policy
privacy collected as well
and terms as the
of service.
destination sink (defined in the previous section) that those particular metrics will go to,
it is also used to define the interval in which data will be sent collected, in the case of
syslog you can specify the facility and the verbosity as follows:
"syslogEvents": {
"sinks": "",
"syslogEventConfiguration": {
"facilityName1": "minSeverity",
"facilityName2": "minSeverity",
...
}
Create a new file, paste the following code and replace variables with your info.
"StorageAccount": "storageaccountname",
"ladCfg": {
"diagnosticMonitorConfiguration": {
"eventVolume": "Medium",
"metrics": {
"metricAggregation": [
"scheduledTransferPeriod": "PT1H"
},
"scheduledTransferPeriod": "PT1M"
],
"resourceId": "VMRESOURCEID (/subscriptions/xxx/resourceGroups/xx/prov

},
"performanceCounters": {
"sinks": "LinuxCpuEventHub",
"performanceCounterConfiguration": [
Adrian Corona
"annotation": [
{
"displayName": "Disk read guest OS",
"locale": "en-us"
}
Ferdinand Escolar
],
"class": "disk",
"condition": "IsAggregate=TRUE",
"counter": "readbytespersecond",
"counterSpecifier": "/builtin/disk/readbytespersecond",
"type": "builtin",
"unit": "BytesPerSecond"
},
{
"annotation": [
"displayName": "Disk writes",
"locale": "en-us"
],
"class": "disk",
"condition": "IsAggregate=TRUE",
"counter": "writespersecond",
"counterSpecifier": "/builtin/disk/writespersecond",
"type": "builtin",
"unit": "CountPerSecond"
},
"syslogEvents": {
"sinks": "SyslogEventHub",
"syslogEventConfiguration": {
"LOG_AUTH": "LOG_DEBUG",
"LOG_AUTHPRIV": "LOG_DEBUG",
"LOG_CRON": "LOG_DEBUG",
"LOG_DAEMON": "LOG_DEBUG",
"LOG_FTP": "LOG_DEBUG",
"LOG_KERN": "LOG_DEBUG",
"LOG_LOCAL0": "LOG_DEBUG",
"LOG_LPR": "LOG_DEBUG",
"LOG_MAIL": "LOG_DEBUG",
"LOG_NEWS": "LOG_DEBUG",
"LOG_SYSLOG": "LOG_DEBUG",
"LOG_USER": "LOG_DEBUG",
"LOG_UUCP": "LOG_DEBUG"
},
"sampleRateInSeconds": 15
Adrian Corona
Save the file as public_settings.json in the same location as the previous file.
Ferdinand Escolar
Pushing the Settings To create your account, Google will share your name,
Change directory to the folder that has the two files you created and use az cli to push
the settings to your vm, use the following command:
az vm extension set --publisher Microsoft.Azure.Diagnostics --name LinuxDiagno
Testing
I followed the instructions here to create a quick .Net console app to test that the
messages were indeed being pushed to the hub and voila!
Adrian Corona
Ferdinand Escolar

Metrics
Success!!
From here you can use a number of tools to consume this info, stay tuned as in Part 2 of
this post I will be pushing those events into Splunk using a HTTP Collector.
Happy clouding!
Adrian Corona
35 ·
4 Comments
Ferdinand Escolar
Like Comment ferdinand.escolar@itsmeph.com Share
☁️Charbel N.
Continue as Ferdinand 2y
Hello Adrian Corona, great article. Thank You!

Quick question please, is Part 2 published (Splunk ingestion)?

I can see the data in Event Hub, but they are not showing on Splunk side.
I am using Azure Monitor TA-Azure_Monitor v1.3.3 add-on, and I have configured the data in‐
put on Splunk under the Azure Monitor Diagnostic Logs.
Any idea?
Thank You!
-Charbel
Like
Reply
1 Like
Rama Krishna
2y
Hi Charbel, Adrian,
We see events coming to EventHub, but not sure they are VM syslogs or metric data. I
dont have splunk in my solution, is there any way we can ensure we can see syslogs ar‐
rived at evenhub?
Like
Reply
☁️Charbel N.
2y
Thank you Adrian, much appreciated!
Yes, I came across the Azure Function for Splunk on GitHub. Finally, I was able to send
the Syslog to Splunk using HTTP Event Collector (HEC).
Since I have 3 event hubs in the same Event

Namsepace: cpueventhub, loggingeventhub, metriceventhub. I am looking on how I can
ingest all of them to Splunk?
The function by default will consume the following:
EhActivityLogsExt - consumes Azure Monitor Activity Logs
EhDiagnosticLogsExt - consumes Azure Monitor Diagnostic Logs
EhLadTelemetryExt - consumes telemetry from Azure Linux VMs
EhMetricsExt - consumes Azure Monitor Metrics
EhWadTelemetryExt - consumes telemetry from Azure Windows VMs
FaultProcessor - consumes queue messages from faulted transmissions
During the deployment of that Function, we need to specify the name of each Event
Hub. Since, we are using Linux VM, the EhLadTelemetryExt is what we need, I was able to
addCorona
Adrian one Event Hub only for EhLadTelemetryExt. I still have to add the remaining 2.

Many Thanks!
-Charbel
Ferdinand Escolar
Like
Reply
2 Likes
To view or add a comment, sign in
More articles by this author
How to send alerts

from OMS Log…
Sep 12, 2017
Others also viewed

How to perform SYN flooding Attack with Metasploit
Shubham Yadav
·
1y
© 2022 About
Accessibility User Agreement
Privacy Policy Cookie Policy
Copyright Policy Brand Policy
Guest Controls Community Guidelines
Language

How To Send Syslog Messages From Azure Linux Vms To An Azure EventHub Part 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How To Send Syslog Messages From Azure Linux Vms To An Azure EventHub Part 1

Uploaded by

Copyright:

Available Formats

6/8/22, 2:01 PM How to send Syslog messages from Azure Linux Vms to an Azure EventHub Part 1

To create your account, Google will share your name,