StreamSets Data Collector

Splunk indexes and correlates log and machine data, providing a rich set of search, analysis and visualization capabilities. In this blog post, I’ll explain how to efficiently send high volumes of data to Splunk’s HTTP Event Collector via the ... [More] StreamSets Data Collector Jython Evaluator. I’ll present a Jython script with which you’ll be able to build The post Ingest Data into Splunk with StreamSets Data Collector appeared first on StreamSets. [Less]

Today we hear a lot about streaming data, fast data, and data in motion. But the truth is that we have always needed ways to move our data.  Historically, the industry has been pretty inventive about getting this done. From the early days of data ... [More] warehousing and extract, transform, and load (ETL) to now, we The post Data in Motion Evolution: Where We’ve Been…Where We Need to Go appeared first on StreamSets. [Less]

When you're building a pipeline with StreamSets Data Collector (SDC), you can often implement the data transformations you require using a combination of ‘off-the-shelf' processors. Sometimes, though, you need to write some code. The script ... [More] evaluators included with SDC allow you to manipulate records in Groovy, JavaScript and Jython (an implementation of Python integrated with The post Calling External Java Code from Script Evaluators appeared first on StreamSets. [Less]

When you’re building a pipeline with StreamSets Data Collector (SDC), you can often implement the data transformations you require using a combination of ‘off-the-shelf’ processors. Sometimes, though, you need to write some code. The script ... [More] evaluators included with SDC allow you to manipulate records in Groovy, JavaScript and Jython (an implementation of Python integrated with The post Calling External Java Code from Script Evaluators appeared first on StreamSets. [Less]

As I explained in my recent tutorial, Creating a Custom Origin for StreamSets Data Collector, it's straightforward to extend StreamSets Data Collector (SDC) to ingest data from pretty much any source. Yogesh Choudhary, a software engineer at ... [More] consulting and services company Clairvoyant, just posted his own walkthrough of building a custom origin for Amazon Simple Queue Service The post Building an Amazon SQS Custom Origin for StreamSets Data Collector appeared first on StreamSets. [Less]

As I explained in my recent tutorial, Creating a Custom Origin for StreamSets Data Collector, it’s straightforward to extend StreamSets Data Collector (SDC) to ingest data from pretty much any source. Yogesh Choudhary, a software engineer at ... [More] consulting and services company Clairvoyant, just posted his own walkthrough of building a custom origin for Amazon Simple Queue Service The post Building an Amazon SQS Custom Origin for StreamSets Data Collector appeared first on StreamSets. [Less]

I'm frequently asked, ‘How does StreamSets Data Collector (SDC) integrate with Spark Streaming? How about on Databricks?'. In this blog entry, I'll explain how to use SDC to ingest data into a Spark Streaming app running on Databricks, but the ... [More] principles apply to Spark apps running anywhere. Databricks is a cloud-based data platform powered by The post Continuous Data Integration with StreamSets Data Collector and Spark Streaming on Databricks appeared first on StreamSets. [Less]

I’m frequently asked, ‘How does StreamSets Data Collector (SDC) integrate with Spark Streaming? How about on Databricks?’. In this blog entry, I’ll explain how to use SDC to ingest data into a Spark Streaming app running on Databricks, but the ... [More] principles apply to Spark apps running anywhere. Databricks is a cloud-based data platform powered by The post Continuous Data Integration with StreamSets Data Collector and Spark Streaming on Databricks appeared first on StreamSets. [Less]

Since writing tutorials for creating custom destinations and processors for StreamSets Data Collector (SDC), I've been looking for a good use case for a custom origin tutorial. It's been trickier than I expected, partly because the list of out of the ... [More] box origins is so extensive, and partly because the HTTP Client origin can access most web The post Creating a Custom Origin for StreamSets Data Collector appeared first on StreamSets. [Less]

Since writing tutorials for creating custom destinations and processors for StreamSets Data Collector (SDC), I’ve been looking for a good use case for a custom origin tutorial. It’s been trickier than I expected, partly because the list of out of the ... [More] box origins is so extensive, and partly because the HTTP Client origin can access most web The post Creating a Custom Origin for StreamSets Data Collector appeared first on StreamSets. [Less]