Developed by Excellians Alexander Slotte and Eric Schiller, this project demonstrates the use of Microsoft Azure to analyze real-time data streams and report on trends. For this prototype we used two types of data, digital data from a real-time Twitter feed, and physical data streamed from a Raspberry Pi.

Check out the source code for the project on GitHub https://github.com/excellalabs/azure-stream-analysis and a live demo of it here.

Sentiment Analysis of a Twitter Feed

This part analyzes Twitter data based on keywords or users utilizing serverless functionality such as Azure Logic Apps, EventHub, and Stream Analytics. The egressed and analyzed data can be visualized in a Power BI dashboard.

Streaming of Raspberry Pi IoT Data

This part demonstrates the ability to stream IoT sensor data from a Raspberry PI to the cloud, utilizing an Azure EventHub, an Azure Stream Analytics job, an Azure Service Bus and an Azure Logic App. The sensor data can be viewed in a Power BI dashboard, but there is also built in functionality to demonstrate how easy it is to set up your own burglar alarm. Any motion detected by the Raspberry PI’s sensors will put a message in a service bus queue that will be picked up by an Azure Logic App. The App then sends a notification e-mail to a specified address. The sample demonstrates the LAG functionality in particular, but also how to use reference data to enrich the stream.

Potential use-cases

There are endless potential use-cases in which this tech-stack can be useful. To mention a few, we could potentially stream:

• Social media and RSS feeds to determine sentiment and overall success of a newly released feature or product. We might message the marketing department to kick off an initiative to drive more traffic depending on what we observe.
• Data from live traffic cameras to determine current conditions and measure improvements after road work.
• Alibaba is using this tech to improve its personalization for millions of products across its e-commerce platform by analyzing real-time search behavior.
• Observing financial transactions to determine potentially fraudulent activities in real time and take action to reduce impact and increase responsiveness.
• IoT sensors of various kinds in the field that could monitor and predict air quality and provide that data to runners and bikers so that they can take the healthiest routes.
• Microsoft built an anomaly detection engine that detects real-time, malicious activity in the cloud, such as compromised accounts, insider threats, and ransomware.

Read More

If you’d like to find more detailed information regarding implementation details and the design of this project, check out these two blog posts:

• https://www.excella.com/insights/getting-started-with-azure-stream-analytics
• https://www.excella.com/insights/near-real-time-data-streaming-with-azure-stream-analytics