How to run a pyspark application in windows 8 command prompt

Published on Author Code Father
How to run a pyspark application in windows 8 command prompt?How to integrate with with IPython on Windows 10 and 7.

First, check your environment variables for Python and Spark. Here are mine: SPARK_HOME: C:\spark-1.6.0-bin-hadoop2.6\ I use Enthought Canopy, so Python is already integrated in my system path. Next, launch Python or IPython and use the following code. If you get an error, check what you get for ‘spark_home’. Otherwise, it should run just fine.

import os

import sys

spark_home = os.environ.get('SPARK_HOME', None)

if not spark_home:

raise ValueError('SPARK_HOME environment variable is not set')

sys.path.insert(0, os.path.join(spark_home, 'python'))

sys.path.insert(0, os.path.join(spark_home, 'C:/spark-1.6.0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip')) ## may need to adjust on your system depending on which Spark version you're using and where you installed it.

execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))

pySpark on IPython

Comments

comments