Apache Spark

Learn how the Arrow Flight service provided by IBM Cloud Pak for Data can be used to read and write data sets from within a Spark Java application that is deployed in IBM Analytics Engine. Arrow Flight provides a common interface for Spark applications to interact with a variety of different data sources.

Explore IBM watsonx.data Developer Edition, a free lakehouse for developers to query data, test AI, run Spark and Presto, and build prototypes fast.

In this tutorial, you'll complete an e-commerce use case with IBM watsonx.data and Apache Spark within the medallion architecture.

Learn how to get the Spark query execution plan using the explain API to debug and analyze your Apache Spark application.

Learn some performance optimization tips to keep in mind when developing your Spark applications.

Use Tekton Pipelines to automate the deployment of a Spark job on Kubernetes

Quickly set up a Spark engine locally so that you can try it out on your own system and dive into the lakehouse world.

Learn from one team's experience when migrating enterprise clients' legacy apps to Apache Spark.

This tutorial covers Big Data via PySpark (a Python package for spark programming). We explain SparkContext by using map and filter methods with Lambda functions in Python. We also create RDD from object and external files, transformations and actions on RDD and pair RDD, SparkSession, and PySpark DataFrame from RDD, and external files. In addition, we use sql queries with DataFrames (by using Spark SQL module). And finally, machine learning with PySpark MLlib library.

Learn some best practices in using Apache Spark Structured Streaming.