Airflow with Google BigQuery and Slack
Airflow is an open source tool for creating, scheduling, and monitoring data processing pipelines. Airflow can be installed via
conda install -c conda-forge airflow or
pip install airflow. Before running airflow, we need to initiate the database
Then we can start the airflow webserver, which a python flask app providing the UI of airflow. The default port of the webserver is 8080:
airflow webserver -p 8080. When we open 0.0.0.0:8080 in the browser, we can see a bunch of airflow examples/DAGs.