Creating Livy Spark Input Data Source
Livy is an open source REST interface for interacting with Apache Stark. It supports executing snippets of code or programs such as Scala, Python, Java, and R in a Spark context that runs locally or in Apache Hadoop YARN.
The Livy Spark connector allows you to run these codes and fetch the data in Panopticon Streams.
- In the New Data Source page, select Input > LivySpark in the Connector drop-down list.
- Enter the following information:
Property Description Host
Livy Spark host address.
Port
User Id that will be used to connect to Livy Spark.
Password
Password that will be used to connect to Livy Spark.
Kind
Currently, the supported kind of connection to be used is pyspark (Interactive Python Spark session).
Request Timeout
Length of time to wait for the server response. Default is 30.
Polling Count
The number of polling done to the Livy Spark server to check if the status of the app is successful. Default limit is 150.
Polling Frequency (in seconds)
Frequency of the polling. Default is 2.
Script
The script to use.
- Click
. The new data source is added in the Data Sources list.
(c) 2013-2024 Altair Engineering Inc. All Rights Reserved.